Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..................X....XXX..X.......X.....XX.XX.XXX.....X..X] duration: 30.002856251s trace: https://jaeger.dev.storj.io/trace/319e22bd076acae8 FAIL (2025-12-26 09:28:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DSZRWELUQHV26EIJISAMEPAP5OXXLYJ7NUD3K4XHIYQTPTIPNR3Q) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: JSBCDJT2NRKVKR6X3PABNSWRRHQRI2Q4GZ7QUGKIKJ2TUID4ZPUQ) context deadline exceeded ======LOGS====== oad=437071]: Uploading segments... Dec 26 09:28:08.418955570 (497h59m8.290038605s) [upload=437071]: Got next segment. Inline: false Dec 26 09:28:08.419031294 (497h59m8.290114329s) [upload=437071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:28:08.419036804 (497h59m8.290119839s) [upload=437071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:28:08.663968175 (497h59m8.53505126s) [seg_pos={0 0}, upload=437071]: Begin upload segment... Dec 26 09:28:08.664026606 (497h59m8.535109642s) [seg_pos={0 0}, upload=437071]: Done begin upload segment. Dec 26 09:28:08.664068326 (497h59m8.535151371s) [noise=true , address=145.79.227.52:4447 , piece_id=QPOPFXJLXURCKFPXWSE7HQPXEZRV3XZ5R74BP55C3J6E67U7MK6Q, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.664145823 (497h59m8.535228868s) [noise=true , address=145.79.227.11:4731 , piece_id=XJPXNJQHVIYLEPV22DZYULXUVMWHEN5WSGJLRLN6BPC2XZB4MOCQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.664921371 (497h59m8.536004416s) [noise=true , address=145.79.227.4:4186 , piece_id=7O3BVVNANVOCSC4OZY5Q3YMXGXYE4OUEBCPVT4PYMMXVCHPRYZHA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.665214528 (497h59m8.536297573s) [noise=true , address=45.91.116.10:4081 , piece_id=DSZRWELUQHV26EIJISAMEPAP5OXXLYJ7NUD3K4XHIYQTPTIPNR3Q, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.665614420 (497h59m8.536697455s) [noise=true , address=45.91.116.8:4046 , piece_id=JSBCDJT2NRKVKR6X3PABNSWRRHQRI2Q4GZ7QUGKIKJ2TUID4ZPUQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.665993302 (497h59m8.537076337s) [noise=true , address=45.91.116.12:4226 , piece_id=O372B7PCAKW4AUAHRYBHBVGVLE2W734JCS6UNXEZJQHLC3K6DDZQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666040852 (497h59m8.537123887s) [noise=true , address=145.79.227.42:4122 , piece_id=6K4GMP2A4VAOO3Y6LPALBPXBVWLA54AR4SM5MA4YTETBBY2B752A, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666405606 (497h59m8.537488651s) [noise=true , address=145.79.227.50:4374 , piece_id=CVOZ4UIWRNBZAIN5RKEKZ32ICGZFD34TVPLDYMVJW435V3RZC6CA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666702762 (497h59m8.537785798s) [noise=true , address=45.91.116.13:4156 , piece_id=VXT2YAMO6RCYHGFV24VFUNSWUZPBWDD3UWXDCTYLAOPDLAVCAVEA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.666995319 (497h59m8.538078364s) [noise=true , address=45.91.116.7:4009 , piece_id=HGZWM43ET6JM3IIT7MOAJBSOREJNTPUYUPDWGF6O4IECZGTZGPRQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.667420760 (497h59m8.538503795s) [noise=true , address=145.79.227.38:4979 , piece_id=BVH2BXNSN2G3MNESWFVFKI52KS72DDB6TWTJPDTFNIDODIEJXGSA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.667791636 (497h59m8.538874671s) [noise=true , address=77.90.31.14:4662 , piece_id=HOHKERASZN2YDRGHKPBYUF3WQH55QIALQSLFNS5UAE52QXGY6J4A, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.667930320 (497h59m8.539013365s) [noise=true , address=145.79.227.46:4603 , piece_id=YFLTIC3TDVZL5VSEVPTZA4OJCPNODT3ZQ3QVLEK2T7GBHM7WAKTQ, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.668574898 (497h59m8.539657933s) [noise=true , address=145.79.227.45:4231 , piece_id=WOXIAN5R7RTWOUIFD7GWRPRQHNUA6S6BUAYPH4STJ4LV2VAEGXFA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.668945613 (497h59m8.540028658s) [noise=true , address=145.79.227.39:4014 , piece_id=Y5AZV5RYKECLQWNIBC7ETIMM274GE2AV3M5I3MNJA5PTWCCHA5HA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:08.669338902 (497h59m8.540421937s) [noise=true , address=45.91.116.14:4189 , piece_id=XFLQI3GPFZHLMLEFBIMXMA2SC6P7VWVNIKYEWLHOKDUZQW5KFHMA, seg_pos={0 0}, upload=437071]: Uploading piece... Dec 26 09:28:10.290945294 (497h59m10.16202835s) [upload=437071]: Next returned nil segment Dec 26 09:28:10.290958509 (497h59m10.162041554s) [upload=437071]: Waiting for error group managing segments... Dec 26 09:28:10.678285896 (497h59m10.549368931s) [noise=true , address=145.79.227.50:4374 , piece_id=CVOZ4UIWRNBZAIN5RKEKZ32ICGZFD34TVPLDYMVJW435V3RZC6CA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.681122548 (497h59m10.552205583s) [noise=true , address=145.79.227.42:4122 , piece_id=6K4GMP2A4VAOO3Y6LPALBPXBVWLA54AR4SM5MA4YTETBBY2B752A, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.683302087 (497h59m10.554385132s) [noise=true , address=145.79.227.4:4186 , piece_id=7O3BVVNANVOCSC4OZY5Q3YMXGXYE4OUEBCPVT4PYMMXVCHPRYZHA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.683990328 (497h59m10.555073363s) [noise=true , address=145.79.227.52:4447 , piece_id=QPOPFXJLXURCKFPXWSE7HQPXEZRV3XZ5R74BP55C3J6E67U7MK6Q, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.693692112 (497h59m10.564775157s) [noise=true , address=145.79.227.39:4014 , piece_id=Y5AZV5RYKECLQWNIBC7ETIMM274GE2AV3M5I3MNJA5PTWCCHA5HA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.696002942 (497h59m10.567085987s) [noise=true , address=145.79.227.45:4231 , piece_id=WOXIAN5R7RTWOUIFD7GWRPRQHNUA6S6BUAYPH4STJ4LV2VAEGXFA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.697317195 (497h59m10.56840023s) [noise=true , address=145.79.227.11:4731 , piece_id=XJPXNJQHVIYLEPV22DZYULXUVMWHEN5WSGJLRLN6BPC2XZB4MOCQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.705818693 (497h59m10.576901738s) [noise=true , address=145.79.227.38:4979 , piece_id=BVH2BXNSN2G3MNESWFVFKI52KS72DDB6TWTJPDTFNIDODIEJXGSA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.713470103 (497h59m10.584553148s) [noise=true , address=145.79.227.46:4603 , piece_id=YFLTIC3TDVZL5VSEVPTZA4OJCPNODT3ZQ3QVLEK2T7GBHM7WAKTQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.955672247 (497h59m10.826755292s) [noise=true , address=45.91.116.14:4189 , piece_id=XFLQI3GPFZHLMLEFBIMXMA2SC6P7VWVNIKYEWLHOKDUZQW5KFHMA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:10.992053222 (497h59m10.863136257s) [noise=true , address=45.91.116.12:4226 , piece_id=O372B7PCAKW4AUAHRYBHBVGVLE2W734JCS6UNXEZJQHLC3K6DDZQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:11.024551975 (497h59m10.89563502s) [noise=true , address=45.91.116.8:4046 , piece_id=JSBCDJT2NRKVKR6X3PABNSWRRHQRI2Q4GZ7QUGKIKJ2TUID4ZPUQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:11.031430964 (497h59m10.902514009s) [noise=true , address=45.91.116.10:4081 , piece_id=DSZRWELUQHV26EIJISAMEPAP5OXXLYJ7NUD3K4XHIYQTPTIPNR3Q, seg_pos={0 0}, upload=437071]: Done uploading piece. err: <nil> Dec 26 09:28:11.031476941 (497h59m10.902559987s) [seg_pos={0 0}, upload=437071]: Segment reached optimal threshold of 13 pieces. Dec 26 09:28:11.031753057 (497h59m10.902836102s) [noise=true , address=77.90.31.14:4662 , piece_id=HOHKERASZN2YDRGHKPBYUF3WQH55QIALQSLFNS5UAE52QXGY6J4A, seg_pos={0 0}, upload=437071]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:38144->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:38144->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38144->77.90.31.14:4662: use of closed network connection Dec 26 09:28:11.031803594 (497h59m10.902886659s) [noise=true , address=77.90.31.14:4662 , piece_id=HOHKERASZN2YDRGHKPBYUF3WQH55QIALQSLFNS5UAE52QXGY6J4A, seg_pos={0 0}, upload=437071]: Long tail cancellation due to optimal threshold Dec 26 09:28:11.032208154 (497h59m10.903291189s) [noise=true , address=45.91.116.13:4156 , piece_id=VXT2YAMO6RCYHGFV24VFUNSWUZPBWDD3UWXDCTYLAOPDLAVCAVEA, seg_pos={0 0}, upload=437071]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 09:28:11.032217091 (497h59m10.903300126s) [noise=true , address=45.91.116.13:4156 , piece_id=VXT2YAMO6RCYHGFV24VFUNSWUZPBWDD3UWXDCTYLAOPDLAVCAVEA, seg_pos={0 0}, upload=437071]: Long tail cancellation due to optimal threshold Dec 26 09:28:11.032346407 (497h59m10.903429452s) [noise=true , address=45.91.116.7:4009 , piece_id=HGZWM43ET6JM3IIT7MOAJBSOREJNTPUYUPDWGF6O4IECZGTZGPRQ, seg_pos={0 0}, upload=437071]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:40496->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:40496->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40496->45.91.116.7:4009: use of closed network connection Dec 26 09:28:11.032354913 (497h59m10.903437958s) [noise=true , address=45.91.116.7:4009 , piece_id=HGZWM43ET6JM3IIT7MOAJBSOREJNTPUYUPDWGF6O4IECZGTZGPRQ, seg_pos={0 0}, upload=437071]: Long tail cancellation due to optimal threshold Dec 26 09:28:11.032372537 (497h59m10.903455582s) [seg_pos={0 0}, upload=437071]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:28:11.032470764 (497h59m10.903553799s) [upload=437071]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:28:11.032477387 (497h59m10.903560432s) [upload=437071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:28:11.319582514 (497h59m11.190665559s) [upload=437071]: Done uploading segments... Dec 26 09:28:11.319601119 (497h59m11.190684154s) [upload=437071]: Upload finished. err: <nil> Duration: 30.002883935s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...................X....XXX..X.......X.....XX.XX.XXX.....X..] duration: 6.062604128s trace: https://jaeger.dev.storj.io/trace/5a6672a524f8bd5c FAIL (2025-12-26 09:27:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 26 09:25:49.895347339 (497h56m49.766430374s) [upload=437042]: Got next segment. Inline: false Dec 26 09:25:49.895382436 (497h56m49.766465471s) [upload=437042]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:25:49.895388638 (497h56m49.766471673s) [upload=437042]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:25:50.140874455 (497h56m50.0119575s) [seg_pos={0 0}, upload=437042]: Begin upload segment... Dec 26 09:25:50.140938667 (497h56m50.012021703s) [seg_pos={0 0}, upload=437042]: Done begin upload segment. Dec 26 09:25:50.140969937 (497h56m50.012052982s) [noise=true , address=45.91.116.5:4297 , piece_id=BATJXBJ3LBJUNYL6KLTI6LJA3XLIV6CGLUDNKXYMQUOO55X3SX5A, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.141055701 (497h56m50.012138736s) [noise=true , address=145.79.227.12:4767 , piece_id=M4PDDCTAWGEB3QVC4DAS2ARX4KEE2LF4CKX2WSXRHQDYHTME4WZQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.141560251 (497h56m50.012643286s) [noise=true , address=145.79.227.45:4229 , piece_id=CL357SM3G2AA4OL5L2OLYGPFJS7DIAPKHYL22GXN53SEORT5N64A, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.141987484 (497h56m50.01307054s) [noise=true , address=145.79.227.29:4942 , piece_id=KJKCQQLMXX7Q4YTAQ3LT2QKWTD34O7VGIX5Z7XQLVMATQWQY5MTQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142370263 (497h56m50.013453298s) [noise=true , address=145.79.227.35:4482 , piece_id=RY6VOEVKAAN42A5GKDPH57Y4ZFRRHWZWJONB3ZLMQX7ZELHGXAVQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142568982 (497h56m50.013652017s) [noise=true , address=145.79.227.5:4222 , piece_id=E5E25UYXESEQSVMSHNH4XDFTH56IQZLBINAT45U5TCEO6LQ2Q5ZA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142667370 (497h56m50.013750405s) [noise=true , address=145.79.227.49:4338 , piece_id=AWMUDKGCIUXBHU7MADZ4NPX3I6IFRROMW5VWPZX7R3XQQVRSBYVQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142934909 (497h56m50.014017954s) [noise=true , address=145.79.227.40:4050 , piece_id=SBQ3LW76DXC7B5BCQGU23DPB5WOTRZD5V6A2VRKEJ4S2KZRD3DXQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143000143 (497h56m50.014083188s) [noise=true , address=45.91.116.11:4117 , piece_id=IJF4JGZSJ3W5AM6LG4HOH323MKXCO4FTBUOXRE4IU4L6DMNEMOPQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143229550 (497h56m50.014312585s) [noise=true , address=145.79.227.51:4410 , piece_id=IYYH3QBMSPGBAFYZSTSO67EVAPHFQUZC7LQHPSYNY5GMP2XIJ7YQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143514592 (497h56m50.014597638s) [noise=true , address=45.91.116.12:4226 , piece_id=K5DEANUOCTI56TR5C7LGRDAOEXTBCKGAYR3DY3LRJQGRBVLYOJRA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143525974 (497h56m50.014609009s) [noise=true , address=145.79.227.41:4087 , piece_id=TTSD747HZZBIQ4N75ANKAIBCUA3BLKNOOKKRG5EQJMSEQDCIBAZQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143931587 (497h56m50.015014622s) [noise=true , address=45.91.116.6:4974 , piece_id=OOAM2E2TEN72VJK34KA6DVBZVB5QY5FHSKM3AK7BA4SYMKLYQ2GQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.144360102 (497h56m50.015443137s) [noise=true , address=145.79.227.36:4518 , piece_id=N5FUZ6ZZTDG5RVDLLUN4IL7M3SCOKBYSYUCTML4IT5JR2MWXAKBA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.144691474 (497h56m50.015774509s) [noise=true , address=145.79.227.52:4445 , piece_id=YUZW73ADUNPWDG2H2NPGUOP34TCV7WRR53X7LEGZMQFCEGKMSTUA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.146265421 (497h56m50.017348466s) [noise=true , address=45.91.116.10:4083 , piece_id=4IODL6VNPNPT3P3JM5KZEBC6XTAKTDCYH6RADIEP7VUDZGK4FUNQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:51.856038844 (497h56m51.727121879s) [upload=437042]: Next returned nil segment Dec 26 09:25:51.856051348 (497h56m51.727134384s) [upload=437042]: Waiting for error group managing segments... Dec 26 09:25:52.140557808 (497h56m52.011640843s) [noise=true , address=145.79.227.51:4410 , piece_id=IYYH3QBMSPGBAFYZSTSO67EVAPHFQUZC7LQHPSYNY5GMP2XIJ7YQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.153967352 (497h56m52.025050387s) [noise=true , address=145.79.227.41:4087 , piece_id=TTSD747HZZBIQ4N75ANKAIBCUA3BLKNOOKKRG5EQJMSEQDCIBAZQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.158364354 (497h56m52.029447399s) [noise=true , address=145.79.227.45:4229 , piece_id=CL357SM3G2AA4OL5L2OLYGPFJS7DIAPKHYL22GXN53SEORT5N64A, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.162378237 (497h56m52.033461283s) [noise=true , address=145.79.227.49:4338 , piece_id=AWMUDKGCIUXBHU7MADZ4NPX3I6IFRROMW5VWPZX7R3XQQVRSBYVQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.164871355 (497h56m52.03595439s) [noise=true , address=145.79.227.12:4767 , piece_id=M4PDDCTAWGEB3QVC4DAS2ARX4KEE2LF4CKX2WSXRHQDYHTME4WZQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.171507481 (497h56m52.042590526s) [noise=true , address=145.79.227.52:4445 , piece_id=YUZW73ADUNPWDG2H2NPGUOP34TCV7WRR53X7LEGZMQFCEGKMSTUA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.172026720 (497h56m52.043109755s) [noise=true , address=145.79.227.35:4482 , piece_id=RY6VOEVKAAN42A5GKDPH57Y4ZFRRHWZWJONB3ZLMQX7ZELHGXAVQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.174049311 (497h56m52.045132356s) [noise=true , address=145.79.227.36:4518 , piece_id=N5FUZ6ZZTDG5RVDLLUN4IL7M3SCOKBYSYUCTML4IT5JR2MWXAKBA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.178618181 (497h56m52.049701226s) [noise=true , address=145.79.227.29:4942 , piece_id=KJKCQQLMXX7Q4YTAQ3LT2QKWTD34O7VGIX5Z7XQLVMATQWQY5MTQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.190615577 (497h56m52.061698612s) [noise=true , address=145.79.227.5:4222 , piece_id=E5E25UYXESEQSVMSHNH4XDFTH56IQZLBINAT45U5TCEO6LQ2Q5ZA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.399223531 (497h56m52.270306567s) [noise=true , address=45.91.116.12:4226 , piece_id=K5DEANUOCTI56TR5C7LGRDAOEXTBCKGAYR3DY3LRJQGRBVLYOJRA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.460637806 (497h56m52.331720841s) [noise=true , address=45.91.116.11:4117 , piece_id=IJF4JGZSJ3W5AM6LG4HOH323MKXCO4FTBUOXRE4IU4L6DMNEMOPQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.482636275 (497h56m52.35371931s) [noise=true , address=45.91.116.5:4297 , piece_id=BATJXBJ3LBJUNYL6KLTI6LJA3XLIV6CGLUDNKXYMQUOO55X3SX5A, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.482655711 (497h56m52.353738746s) [seg_pos={0 0}, upload=437042]: Segment reached optimal threshold of 13 pieces. Dec 26 09:25:52.482830083 (497h56m52.353913128s) [noise=true , address=145.79.227.40:4050 , piece_id=SBQ3LW76DXC7B5BCQGU23DPB5WOTRZD5V6A2VRKEJ4S2KZRD3DXQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 09:25:52.482836605 (497h56m52.35391964s) [noise=true , address=145.79.227.40:4050 , piece_id=SBQ3LW76DXC7B5BCQGU23DPB5WOTRZD5V6A2VRKEJ4S2KZRD3DXQ, seg_pos={0 0}, upload=437042]: Long tail cancellation due to optimal threshold Dec 26 09:25:52.483038169 (497h56m52.354121204s) [noise=true , address=45.91.116.10:4083 , piece_id=4IODL6VNPNPT3P3JM5KZEBC6XTAKTDCYH6RADIEP7VUDZGK4FUNQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:52288->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:52288->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52288->45.91.116.10:4083: use of closed network connection Dec 26 09:25:52.483044140 (497h56m52.354127176s) [noise=true , address=45.91.116.10:4083 , piece_id=4IODL6VNPNPT3P3JM5KZEBC6XTAKTDCYH6RADIEP7VUDZGK4FUNQ, seg_pos={0 0}, upload=437042]: Long tail cancellation due to optimal threshold Dec 26 09:25:52.483447479 (497h56m52.354530524s) [noise=true , address=45.91.116.6:4974 , piece_id=OOAM2E2TEN72VJK34KA6DVBZVB5QY5FHSKM3AK7BA4SYMKLYQ2GQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:49186->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:49186->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49186->45.91.116.6:4974: use of closed network connection Dec 26 09:25:52.483468148 (497h56m52.354551203s) [noise=true , address=45.91.116.6:4974 , piece_id=OOAM2E2TEN72VJK34KA6DVBZVB5QY5FHSKM3AK7BA4SYMKLYQ2GQ, seg_pos={0 0}, upload=437042]: Long tail cancellation due to optimal threshold Dec 26 09:25:52.483549814 (497h56m52.354632869s) [seg_pos={0 0}, upload=437042]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:25:52.483819818 (497h56m52.354902863s) [upload=437042]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:25:52.483828595 (497h56m52.35491164s) [upload=437042]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:25:52.771102113 (497h56m52.642185158s) [upload=437042]: Done uploading segments... Dec 26 09:25:52.771131258 (497h56m52.642214303s) [upload=437042]: Upload finished. err: <nil> Duration: 6.06263213s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....................X....XXX..X.......X.....XX.XX.XXX.....X.] duration: 5.746081271s trace: https://jaeger.dev.storj.io/trace/5a6672a524f8bd5c FAIL (2025-12-26 09:26:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 26 09:25:49.895347339 (497h56m49.766430374s) [upload=437042]: Got next segment. Inline: false Dec 26 09:25:49.895382436 (497h56m49.766465471s) [upload=437042]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:25:49.895388638 (497h56m49.766471673s) [upload=437042]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:25:50.140874455 (497h56m50.0119575s) [seg_pos={0 0}, upload=437042]: Begin upload segment... Dec 26 09:25:50.140938667 (497h56m50.012021703s) [seg_pos={0 0}, upload=437042]: Done begin upload segment. Dec 26 09:25:50.140969937 (497h56m50.012052982s) [noise=true , address=45.91.116.5:4297 , piece_id=BATJXBJ3LBJUNYL6KLTI6LJA3XLIV6CGLUDNKXYMQUOO55X3SX5A, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.141055701 (497h56m50.012138736s) [noise=true , address=145.79.227.12:4767 , piece_id=M4PDDCTAWGEB3QVC4DAS2ARX4KEE2LF4CKX2WSXRHQDYHTME4WZQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.141560251 (497h56m50.012643286s) [noise=true , address=145.79.227.45:4229 , piece_id=CL357SM3G2AA4OL5L2OLYGPFJS7DIAPKHYL22GXN53SEORT5N64A, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.141987484 (497h56m50.01307054s) [noise=true , address=145.79.227.29:4942 , piece_id=KJKCQQLMXX7Q4YTAQ3LT2QKWTD34O7VGIX5Z7XQLVMATQWQY5MTQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142370263 (497h56m50.013453298s) [noise=true , address=145.79.227.35:4482 , piece_id=RY6VOEVKAAN42A5GKDPH57Y4ZFRRHWZWJONB3ZLMQX7ZELHGXAVQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142568982 (497h56m50.013652017s) [noise=true , address=145.79.227.5:4222 , piece_id=E5E25UYXESEQSVMSHNH4XDFTH56IQZLBINAT45U5TCEO6LQ2Q5ZA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142667370 (497h56m50.013750405s) [noise=true , address=145.79.227.49:4338 , piece_id=AWMUDKGCIUXBHU7MADZ4NPX3I6IFRROMW5VWPZX7R3XQQVRSBYVQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142934909 (497h56m50.014017954s) [noise=true , address=145.79.227.40:4050 , piece_id=SBQ3LW76DXC7B5BCQGU23DPB5WOTRZD5V6A2VRKEJ4S2KZRD3DXQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143000143 (497h56m50.014083188s) [noise=true , address=45.91.116.11:4117 , piece_id=IJF4JGZSJ3W5AM6LG4HOH323MKXCO4FTBUOXRE4IU4L6DMNEMOPQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143229550 (497h56m50.014312585s) [noise=true , address=145.79.227.51:4410 , piece_id=IYYH3QBMSPGBAFYZSTSO67EVAPHFQUZC7LQHPSYNY5GMP2XIJ7YQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143514592 (497h56m50.014597638s) [noise=true , address=45.91.116.12:4226 , piece_id=K5DEANUOCTI56TR5C7LGRDAOEXTBCKGAYR3DY3LRJQGRBVLYOJRA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143525974 (497h56m50.014609009s) [noise=true , address=145.79.227.41:4087 , piece_id=TTSD747HZZBIQ4N75ANKAIBCUA3BLKNOOKKRG5EQJMSEQDCIBAZQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143931587 (497h56m50.015014622s) [noise=true , address=45.91.116.6:4974 , piece_id=OOAM2E2TEN72VJK34KA6DVBZVB5QY5FHSKM3AK7BA4SYMKLYQ2GQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.144360102 (497h56m50.015443137s) [noise=true , address=145.79.227.36:4518 , piece_id=N5FUZ6ZZTDG5RVDLLUN4IL7M3SCOKBYSYUCTML4IT5JR2MWXAKBA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.144691474 (497h56m50.015774509s) [noise=true , address=145.79.227.52:4445 , piece_id=YUZW73ADUNPWDG2H2NPGUOP34TCV7WRR53X7LEGZMQFCEGKMSTUA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.146265421 (497h56m50.017348466s) [noise=true , address=45.91.116.10:4083 , piece_id=4IODL6VNPNPT3P3JM5KZEBC6XTAKTDCYH6RADIEP7VUDZGK4FUNQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:51.856038844 (497h56m51.727121879s) [upload=437042]: Next returned nil segment Dec 26 09:25:51.856051348 (497h56m51.727134384s) [upload=437042]: Waiting for error group managing segments... Dec 26 09:25:52.140557808 (497h56m52.011640843s) [noise=true , address=145.79.227.51:4410 , piece_id=IYYH3QBMSPGBAFYZSTSO67EVAPHFQUZC7LQHPSYNY5GMP2XIJ7YQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.153967352 (497h56m52.025050387s) [noise=true , address=145.79.227.41:4087 , piece_id=TTSD747HZZBIQ4N75ANKAIBCUA3BLKNOOKKRG5EQJMSEQDCIBAZQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.158364354 (497h56m52.029447399s) [noise=true , address=145.79.227.45:4229 , piece_id=CL357SM3G2AA4OL5L2OLYGPFJS7DIAPKHYL22GXN53SEORT5N64A, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.162378237 (497h56m52.033461283s) [noise=true , address=145.79.227.49:4338 , piece_id=AWMUDKGCIUXBHU7MADZ4NPX3I6IFRROMW5VWPZX7R3XQQVRSBYVQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.164871355 (497h56m52.03595439s) [noise=true , address=145.79.227.12:4767 , piece_id=M4PDDCTAWGEB3QVC4DAS2ARX4KEE2LF4CKX2WSXRHQDYHTME4WZQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.171507481 (497h56m52.042590526s) [noise=true , address=145.79.227.52:4445 , piece_id=YUZW73ADUNPWDG2H2NPGUOP34TCV7WRR53X7LEGZMQFCEGKMSTUA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.172026720 (497h56m52.043109755s) [noise=true , address=145.79.227.35:4482 , piece_id=RY6VOEVKAAN42A5GKDPH57Y4ZFRRHWZWJONB3ZLMQX7ZELHGXAVQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.174049311 (497h56m52.045132356s) [noise=true , address=145.79.227.36:4518 , piece_id=N5FUZ6ZZTDG5RVDLLUN4IL7M3SCOKBYSYUCTML4IT5JR2MWXAKBA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.178618181 (497h56m52.049701226s) [noise=true , address=145.79.227.29:4942 , piece_id=KJKCQQLMXX7Q4YTAQ3LT2QKWTD34O7VGIX5Z7XQLVMATQWQY5MTQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.190615577 (497h56m52.061698612s) [noise=true , address=145.79.227.5:4222 , piece_id=E5E25UYXESEQSVMSHNH4XDFTH56IQZLBINAT45U5TCEO6LQ2Q5ZA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.399223531 (497h56m52.270306567s) [noise=true , address=45.91.116.12:4226 , piece_id=K5DEANUOCTI56TR5C7LGRDAOEXTBCKGAYR3DY3LRJQGRBVLYOJRA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.460637806 (497h56m52.331720841s) [noise=true , address=45.91.116.11:4117 , piece_id=IJF4JGZSJ3W5AM6LG4HOH323MKXCO4FTBUOXRE4IU4L6DMNEMOPQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.482636275 (497h56m52.35371931s) [noise=true , address=45.91.116.5:4297 , piece_id=BATJXBJ3LBJUNYL6KLTI6LJA3XLIV6CGLUDNKXYMQUOO55X3SX5A, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.482655711 (497h56m52.353738746s) [seg_pos={0 0}, upload=437042]: Segment reached optimal threshold of 13 pieces. Dec 26 09:25:52.482830083 (497h56m52.353913128s) [noise=true , address=145.79.227.40:4050 , piece_id=SBQ3LW76DXC7B5BCQGU23DPB5WOTRZD5V6A2VRKEJ4S2KZRD3DXQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 09:25:52.482836605 (497h56m52.35391964s) [noise=true , address=145.79.227.40:4050 , piece_id=SBQ3LW76DXC7B5BCQGU23DPB5WOTRZD5V6A2VRKEJ4S2KZRD3DXQ, seg_pos={0 0}, upload=437042]: Long tail cancellation due to optimal threshold Dec 26 09:25:52.483038169 (497h56m52.354121204s) [noise=true , address=45.91.116.10:4083 , piece_id=4IODL6VNPNPT3P3JM5KZEBC6XTAKTDCYH6RADIEP7VUDZGK4FUNQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:52288->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:52288->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52288->45.91.116.10:4083: use of closed network connection Dec 26 09:25:52.483044140 (497h56m52.354127176s) [noise=true , address=45.91.116.10:4083 , piece_id=4IODL6VNPNPT3P3JM5KZEBC6XTAKTDCYH6RADIEP7VUDZGK4FUNQ, seg_pos={0 0}, upload=437042]: Long tail cancellation due to optimal threshold Dec 26 09:25:52.483447479 (497h56m52.354530524s) [noise=true , address=45.91.116.6:4974 , piece_id=OOAM2E2TEN72VJK34KA6DVBZVB5QY5FHSKM3AK7BA4SYMKLYQ2GQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:49186->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:49186->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49186->45.91.116.6:4974: use of closed network connection Dec 26 09:25:52.483468148 (497h56m52.354551203s) [noise=true , address=45.91.116.6:4974 , piece_id=OOAM2E2TEN72VJK34KA6DVBZVB5QY5FHSKM3AK7BA4SYMKLYQ2GQ, seg_pos={0 0}, upload=437042]: Long tail cancellation due to optimal threshold Dec 26 09:25:52.483549814 (497h56m52.354632869s) [seg_pos={0 0}, upload=437042]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:25:52.483819818 (497h56m52.354902863s) [upload=437042]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:25:52.483828595 (497h56m52.35491164s) [upload=437042]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:25:52.771102113 (497h56m52.642185158s) [upload=437042]: Done uploading segments... Dec 26 09:25:52.771131258 (497h56m52.642214303s) [upload=437042]: Upload finished. err: <nil> Duration: 5.74609694s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X....................X....XXX..X.......X.....XX.XX.XXX.....X] duration: 30.003748748s trace: https://jaeger.dev.storj.io/trace/5a6672a524f8bd5c FAIL (2025-12-26 09:26:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 26 09:25:49.895347339 (497h56m49.766430374s) [upload=437042]: Got next segment. Inline: false Dec 26 09:25:49.895382436 (497h56m49.766465471s) [upload=437042]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:25:49.895388638 (497h56m49.766471673s) [upload=437042]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:25:50.140874455 (497h56m50.0119575s) [seg_pos={0 0}, upload=437042]: Begin upload segment... Dec 26 09:25:50.140938667 (497h56m50.012021703s) [seg_pos={0 0}, upload=437042]: Done begin upload segment. Dec 26 09:25:50.140969937 (497h56m50.012052982s) [noise=true , address=45.91.116.5:4297 , piece_id=BATJXBJ3LBJUNYL6KLTI6LJA3XLIV6CGLUDNKXYMQUOO55X3SX5A, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.141055701 (497h56m50.012138736s) [noise=true , address=145.79.227.12:4767 , piece_id=M4PDDCTAWGEB3QVC4DAS2ARX4KEE2LF4CKX2WSXRHQDYHTME4WZQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.141560251 (497h56m50.012643286s) [noise=true , address=145.79.227.45:4229 , piece_id=CL357SM3G2AA4OL5L2OLYGPFJS7DIAPKHYL22GXN53SEORT5N64A, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.141987484 (497h56m50.01307054s) [noise=true , address=145.79.227.29:4942 , piece_id=KJKCQQLMXX7Q4YTAQ3LT2QKWTD34O7VGIX5Z7XQLVMATQWQY5MTQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142370263 (497h56m50.013453298s) [noise=true , address=145.79.227.35:4482 , piece_id=RY6VOEVKAAN42A5GKDPH57Y4ZFRRHWZWJONB3ZLMQX7ZELHGXAVQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142568982 (497h56m50.013652017s) [noise=true , address=145.79.227.5:4222 , piece_id=E5E25UYXESEQSVMSHNH4XDFTH56IQZLBINAT45U5TCEO6LQ2Q5ZA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142667370 (497h56m50.013750405s) [noise=true , address=145.79.227.49:4338 , piece_id=AWMUDKGCIUXBHU7MADZ4NPX3I6IFRROMW5VWPZX7R3XQQVRSBYVQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.142934909 (497h56m50.014017954s) [noise=true , address=145.79.227.40:4050 , piece_id=SBQ3LW76DXC7B5BCQGU23DPB5WOTRZD5V6A2VRKEJ4S2KZRD3DXQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143000143 (497h56m50.014083188s) [noise=true , address=45.91.116.11:4117 , piece_id=IJF4JGZSJ3W5AM6LG4HOH323MKXCO4FTBUOXRE4IU4L6DMNEMOPQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143229550 (497h56m50.014312585s) [noise=true , address=145.79.227.51:4410 , piece_id=IYYH3QBMSPGBAFYZSTSO67EVAPHFQUZC7LQHPSYNY5GMP2XIJ7YQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143514592 (497h56m50.014597638s) [noise=true , address=45.91.116.12:4226 , piece_id=K5DEANUOCTI56TR5C7LGRDAOEXTBCKGAYR3DY3LRJQGRBVLYOJRA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143525974 (497h56m50.014609009s) [noise=true , address=145.79.227.41:4087 , piece_id=TTSD747HZZBIQ4N75ANKAIBCUA3BLKNOOKKRG5EQJMSEQDCIBAZQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.143931587 (497h56m50.015014622s) [noise=true , address=45.91.116.6:4974 , piece_id=OOAM2E2TEN72VJK34KA6DVBZVB5QY5FHSKM3AK7BA4SYMKLYQ2GQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.144360102 (497h56m50.015443137s) [noise=true , address=145.79.227.36:4518 , piece_id=N5FUZ6ZZTDG5RVDLLUN4IL7M3SCOKBYSYUCTML4IT5JR2MWXAKBA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.144691474 (497h56m50.015774509s) [noise=true , address=145.79.227.52:4445 , piece_id=YUZW73ADUNPWDG2H2NPGUOP34TCV7WRR53X7LEGZMQFCEGKMSTUA, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:50.146265421 (497h56m50.017348466s) [noise=true , address=45.91.116.10:4083 , piece_id=4IODL6VNPNPT3P3JM5KZEBC6XTAKTDCYH6RADIEP7VUDZGK4FUNQ, seg_pos={0 0}, upload=437042]: Uploading piece... Dec 26 09:25:51.856038844 (497h56m51.727121879s) [upload=437042]: Next returned nil segment Dec 26 09:25:51.856051348 (497h56m51.727134384s) [upload=437042]: Waiting for error group managing segments... Dec 26 09:25:52.140557808 (497h56m52.011640843s) [noise=true , address=145.79.227.51:4410 , piece_id=IYYH3QBMSPGBAFYZSTSO67EVAPHFQUZC7LQHPSYNY5GMP2XIJ7YQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.153967352 (497h56m52.025050387s) [noise=true , address=145.79.227.41:4087 , piece_id=TTSD747HZZBIQ4N75ANKAIBCUA3BLKNOOKKRG5EQJMSEQDCIBAZQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.158364354 (497h56m52.029447399s) [noise=true , address=145.79.227.45:4229 , piece_id=CL357SM3G2AA4OL5L2OLYGPFJS7DIAPKHYL22GXN53SEORT5N64A, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.162378237 (497h56m52.033461283s) [noise=true , address=145.79.227.49:4338 , piece_id=AWMUDKGCIUXBHU7MADZ4NPX3I6IFRROMW5VWPZX7R3XQQVRSBYVQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.164871355 (497h56m52.03595439s) [noise=true , address=145.79.227.12:4767 , piece_id=M4PDDCTAWGEB3QVC4DAS2ARX4KEE2LF4CKX2WSXRHQDYHTME4WZQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.171507481 (497h56m52.042590526s) [noise=true , address=145.79.227.52:4445 , piece_id=YUZW73ADUNPWDG2H2NPGUOP34TCV7WRR53X7LEGZMQFCEGKMSTUA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.172026720 (497h56m52.043109755s) [noise=true , address=145.79.227.35:4482 , piece_id=RY6VOEVKAAN42A5GKDPH57Y4ZFRRHWZWJONB3ZLMQX7ZELHGXAVQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.174049311 (497h56m52.045132356s) [noise=true , address=145.79.227.36:4518 , piece_id=N5FUZ6ZZTDG5RVDLLUN4IL7M3SCOKBYSYUCTML4IT5JR2MWXAKBA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.178618181 (497h56m52.049701226s) [noise=true , address=145.79.227.29:4942 , piece_id=KJKCQQLMXX7Q4YTAQ3LT2QKWTD34O7VGIX5Z7XQLVMATQWQY5MTQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.190615577 (497h56m52.061698612s) [noise=true , address=145.79.227.5:4222 , piece_id=E5E25UYXESEQSVMSHNH4XDFTH56IQZLBINAT45U5TCEO6LQ2Q5ZA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.399223531 (497h56m52.270306567s) [noise=true , address=45.91.116.12:4226 , piece_id=K5DEANUOCTI56TR5C7LGRDAOEXTBCKGAYR3DY3LRJQGRBVLYOJRA, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.460637806 (497h56m52.331720841s) [noise=true , address=45.91.116.11:4117 , piece_id=IJF4JGZSJ3W5AM6LG4HOH323MKXCO4FTBUOXRE4IU4L6DMNEMOPQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.482636275 (497h56m52.35371931s) [noise=true , address=45.91.116.5:4297 , piece_id=BATJXBJ3LBJUNYL6KLTI6LJA3XLIV6CGLUDNKXYMQUOO55X3SX5A, seg_pos={0 0}, upload=437042]: Done uploading piece. err: <nil> Dec 26 09:25:52.482655711 (497h56m52.353738746s) [seg_pos={0 0}, upload=437042]: Segment reached optimal threshold of 13 pieces. Dec 26 09:25:52.482830083 (497h56m52.353913128s) [noise=true , address=145.79.227.40:4050 , piece_id=SBQ3LW76DXC7B5BCQGU23DPB5WOTRZD5V6A2VRKEJ4S2KZRD3DXQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 09:25:52.482836605 (497h56m52.35391964s) [noise=true , address=145.79.227.40:4050 , piece_id=SBQ3LW76DXC7B5BCQGU23DPB5WOTRZD5V6A2VRKEJ4S2KZRD3DXQ, seg_pos={0 0}, upload=437042]: Long tail cancellation due to optimal threshold Dec 26 09:25:52.483038169 (497h56m52.354121204s) [noise=true , address=45.91.116.10:4083 , piece_id=4IODL6VNPNPT3P3JM5KZEBC6XTAKTDCYH6RADIEP7VUDZGK4FUNQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:52288->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:52288->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52288->45.91.116.10:4083: use of closed network connection Dec 26 09:25:52.483044140 (497h56m52.354127176s) [noise=true , address=45.91.116.10:4083 , piece_id=4IODL6VNPNPT3P3JM5KZEBC6XTAKTDCYH6RADIEP7VUDZGK4FUNQ, seg_pos={0 0}, upload=437042]: Long tail cancellation due to optimal threshold Dec 26 09:25:52.483447479 (497h56m52.354530524s) [noise=true , address=45.91.116.6:4974 , piece_id=OOAM2E2TEN72VJK34KA6DVBZVB5QY5FHSKM3AK7BA4SYMKLYQ2GQ, seg_pos={0 0}, upload=437042]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:49186->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:49186->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49186->45.91.116.6:4974: use of closed network connection Dec 26 09:25:52.483468148 (497h56m52.354551203s) [noise=true , address=45.91.116.6:4974 , piece_id=OOAM2E2TEN72VJK34KA6DVBZVB5QY5FHSKM3AK7BA4SYMKLYQ2GQ, seg_pos={0 0}, upload=437042]: Long tail cancellation due to optimal threshold Dec 26 09:25:52.483549814 (497h56m52.354632869s) [seg_pos={0 0}, upload=437042]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:25:52.483819818 (497h56m52.354902863s) [upload=437042]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:25:52.483828595 (497h56m52.35491164s) [upload=437042]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:25:52.771102113 (497h56m52.642185158s) [upload=437042]: Done uploading segments... Dec 26 09:25:52.771131258 (497h56m52.642214303s) [upload=437042]: Upload finished. err: <nil> Duration: 30.003771734s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX....................X....XXX..X.......X.....XX.XX.XXX.....] duration: 5.959248379s trace: https://jaeger.dev.storj.io/trace/389ac4bc3ff91e5c FAIL (2025-12-26 09:25:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding segments... Dec 26 09:22:39.145892195 (497h53m39.01697523s) [upload=436993]: Got next segment. Inline: false Dec 26 09:22:39.145927702 (497h53m39.017010737s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:22:39.145931580 (497h53m39.017014625s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:22:39.388524600 (497h53m39.259607645s) [seg_pos={0 0}, upload=436993]: Begin upload segment... Dec 26 09:22:39.388594132 (497h53m39.259677167s) [seg_pos={0 0}, upload=436993]: Done begin upload segment. Dec 26 09:22:39.388632365 (497h53m39.25971541s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.388653324 (497h53m39.259736359s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389447877 (497h53m39.260530922s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389850545 (497h53m39.26093358s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390368431 (497h53m39.261451476s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390681477 (497h53m39.261764512s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390976849 (497h53m39.262059884s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.392704890 (497h53m39.263787935s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.394668378 (497h53m39.265751413s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.396705798 (497h53m39.267788833s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.399751377 (497h53m39.270834412s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.400608700 (497h53m39.271691745s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.403933580 (497h53m39.275016626s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406414265 (497h53m39.2774973s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406579328 (497h53m39.277662374s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.408705978 (497h53m39.279789023s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:41.096910661 (497h53m40.967993696s) [upload=436993]: Next returned nil segment Dec 26 09:22:41.096923887 (497h53m40.968006922s) [upload=436993]: Waiting for error group managing segments... Dec 26 09:22:41.381072225 (497h53m41.25215526s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.383949273 (497h53m41.255032308s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.396690755 (497h53m41.26777379s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.411478754 (497h53m41.282561799s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421077892 (497h53m41.292160938s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421372643 (497h53m41.292455688s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.434903529 (497h53m41.305986574s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.435526675 (497h53m41.30660972s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.438982786 (497h53m41.310065821s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.625051454 (497h53m41.496134489s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.646488653 (497h53m41.517571688s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.687221335 (497h53m41.55830438s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690272836 (497h53m41.561355871s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690292143 (497h53m41.561375188s) [seg_pos={0 0}, upload=436993]: Segment reached optimal threshold of 13 pieces. Dec 26 09:22:41.690533532 (497h53m41.561616577s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection Dec 26 09:22:41.690545716 (497h53m41.561628751s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690586363 (497h53m41.561669398s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection Dec 26 09:22:41.690602573 (497h53m41.561685618s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690748803 (497h53m41.561831848s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 09:22:41.690758992 (497h53m41.561842037s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690901523 (497h53m41.561984568s) [seg_pos={0 0}, upload=436993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:22:41.691053442 (497h53m41.562136487s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:22:41.691060506 (497h53m41.562143551s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:22:41.976817686 (497h53m41.847900731s) [upload=436993]: Done uploading segments... Dec 26 09:22:41.976839939 (497h53m41.847922984s) [upload=436993]: Upload finished. err: <nil> Duration: 5.959276532s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXX....................X....XXX..X.......X.....XX.XX.XXX....] duration: 6.590385262s trace: https://jaeger.dev.storj.io/trace/389ac4bc3ff91e5c FAIL (2025-12-26 09:24:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding segments... Dec 26 09:22:39.145892195 (497h53m39.01697523s) [upload=436993]: Got next segment. Inline: false Dec 26 09:22:39.145927702 (497h53m39.017010737s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:22:39.145931580 (497h53m39.017014625s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:22:39.388524600 (497h53m39.259607645s) [seg_pos={0 0}, upload=436993]: Begin upload segment... Dec 26 09:22:39.388594132 (497h53m39.259677167s) [seg_pos={0 0}, upload=436993]: Done begin upload segment. Dec 26 09:22:39.388632365 (497h53m39.25971541s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.388653324 (497h53m39.259736359s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389447877 (497h53m39.260530922s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389850545 (497h53m39.26093358s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390368431 (497h53m39.261451476s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390681477 (497h53m39.261764512s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390976849 (497h53m39.262059884s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.392704890 (497h53m39.263787935s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.394668378 (497h53m39.265751413s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.396705798 (497h53m39.267788833s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.399751377 (497h53m39.270834412s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.400608700 (497h53m39.271691745s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.403933580 (497h53m39.275016626s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406414265 (497h53m39.2774973s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406579328 (497h53m39.277662374s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.408705978 (497h53m39.279789023s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:41.096910661 (497h53m40.967993696s) [upload=436993]: Next returned nil segment Dec 26 09:22:41.096923887 (497h53m40.968006922s) [upload=436993]: Waiting for error group managing segments... Dec 26 09:22:41.381072225 (497h53m41.25215526s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.383949273 (497h53m41.255032308s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.396690755 (497h53m41.26777379s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.411478754 (497h53m41.282561799s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421077892 (497h53m41.292160938s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421372643 (497h53m41.292455688s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.434903529 (497h53m41.305986574s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.435526675 (497h53m41.30660972s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.438982786 (497h53m41.310065821s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.625051454 (497h53m41.496134489s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.646488653 (497h53m41.517571688s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.687221335 (497h53m41.55830438s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690272836 (497h53m41.561355871s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690292143 (497h53m41.561375188s) [seg_pos={0 0}, upload=436993]: Segment reached optimal threshold of 13 pieces. Dec 26 09:22:41.690533532 (497h53m41.561616577s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection Dec 26 09:22:41.690545716 (497h53m41.561628751s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690586363 (497h53m41.561669398s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection Dec 26 09:22:41.690602573 (497h53m41.561685618s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690748803 (497h53m41.561831848s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 09:22:41.690758992 (497h53m41.561842037s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690901523 (497h53m41.561984568s) [seg_pos={0 0}, upload=436993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:22:41.691053442 (497h53m41.562136487s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:22:41.691060506 (497h53m41.562143551s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:22:41.976817686 (497h53m41.847900731s) [upload=436993]: Done uploading segments... Dec 26 09:22:41.976839939 (497h53m41.847922984s) [upload=436993]: Upload finished. err: <nil> Duration: 6.590414629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.XXX....................X....XXX..X.......X.....XX.XX.XXX...] duration: 6.136146383s trace: https://jaeger.dev.storj.io/trace/389ac4bc3ff91e5c FAIL (2025-12-26 09:24:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding segments... Dec 26 09:22:39.145892195 (497h53m39.01697523s) [upload=436993]: Got next segment. Inline: false Dec 26 09:22:39.145927702 (497h53m39.017010737s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:22:39.145931580 (497h53m39.017014625s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:22:39.388524600 (497h53m39.259607645s) [seg_pos={0 0}, upload=436993]: Begin upload segment... Dec 26 09:22:39.388594132 (497h53m39.259677167s) [seg_pos={0 0}, upload=436993]: Done begin upload segment. Dec 26 09:22:39.388632365 (497h53m39.25971541s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.388653324 (497h53m39.259736359s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389447877 (497h53m39.260530922s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389850545 (497h53m39.26093358s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390368431 (497h53m39.261451476s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390681477 (497h53m39.261764512s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390976849 (497h53m39.262059884s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.392704890 (497h53m39.263787935s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.394668378 (497h53m39.265751413s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.396705798 (497h53m39.267788833s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.399751377 (497h53m39.270834412s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.400608700 (497h53m39.271691745s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.403933580 (497h53m39.275016626s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406414265 (497h53m39.2774973s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406579328 (497h53m39.277662374s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.408705978 (497h53m39.279789023s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:41.096910661 (497h53m40.967993696s) [upload=436993]: Next returned nil segment Dec 26 09:22:41.096923887 (497h53m40.968006922s) [upload=436993]: Waiting for error group managing segments... Dec 26 09:22:41.381072225 (497h53m41.25215526s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.383949273 (497h53m41.255032308s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.396690755 (497h53m41.26777379s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.411478754 (497h53m41.282561799s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421077892 (497h53m41.292160938s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421372643 (497h53m41.292455688s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.434903529 (497h53m41.305986574s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.435526675 (497h53m41.30660972s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.438982786 (497h53m41.310065821s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.625051454 (497h53m41.496134489s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.646488653 (497h53m41.517571688s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.687221335 (497h53m41.55830438s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690272836 (497h53m41.561355871s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690292143 (497h53m41.561375188s) [seg_pos={0 0}, upload=436993]: Segment reached optimal threshold of 13 pieces. Dec 26 09:22:41.690533532 (497h53m41.561616577s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection Dec 26 09:22:41.690545716 (497h53m41.561628751s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690586363 (497h53m41.561669398s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection Dec 26 09:22:41.690602573 (497h53m41.561685618s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690748803 (497h53m41.561831848s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 09:22:41.690758992 (497h53m41.561842037s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690901523 (497h53m41.561984568s) [seg_pos={0 0}, upload=436993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:22:41.691053442 (497h53m41.562136487s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:22:41.691060506 (497h53m41.562143551s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:22:41.976817686 (497h53m41.847900731s) [upload=436993]: Done uploading segments... Dec 26 09:22:41.976839939 (497h53m41.847922984s) [upload=436993]: Upload finished. err: <nil> Duration: 6.136177272s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.XXX....................X....XXX..X.......X.....XX.XX.XXX..] duration: 6.062871796s trace: https://jaeger.dev.storj.io/trace/389ac4bc3ff91e5c FAIL (2025-12-26 09:23:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding segments... Dec 26 09:22:39.145892195 (497h53m39.01697523s) [upload=436993]: Got next segment. Inline: false Dec 26 09:22:39.145927702 (497h53m39.017010737s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:22:39.145931580 (497h53m39.017014625s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:22:39.388524600 (497h53m39.259607645s) [seg_pos={0 0}, upload=436993]: Begin upload segment... Dec 26 09:22:39.388594132 (497h53m39.259677167s) [seg_pos={0 0}, upload=436993]: Done begin upload segment. Dec 26 09:22:39.388632365 (497h53m39.25971541s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.388653324 (497h53m39.259736359s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389447877 (497h53m39.260530922s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389850545 (497h53m39.26093358s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390368431 (497h53m39.261451476s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390681477 (497h53m39.261764512s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390976849 (497h53m39.262059884s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.392704890 (497h53m39.263787935s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.394668378 (497h53m39.265751413s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.396705798 (497h53m39.267788833s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.399751377 (497h53m39.270834412s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.400608700 (497h53m39.271691745s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.403933580 (497h53m39.275016626s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406414265 (497h53m39.2774973s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406579328 (497h53m39.277662374s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.408705978 (497h53m39.279789023s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:41.096910661 (497h53m40.967993696s) [upload=436993]: Next returned nil segment Dec 26 09:22:41.096923887 (497h53m40.968006922s) [upload=436993]: Waiting for error group managing segments... Dec 26 09:22:41.381072225 (497h53m41.25215526s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.383949273 (497h53m41.255032308s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.396690755 (497h53m41.26777379s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.411478754 (497h53m41.282561799s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421077892 (497h53m41.292160938s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421372643 (497h53m41.292455688s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.434903529 (497h53m41.305986574s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.435526675 (497h53m41.30660972s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.438982786 (497h53m41.310065821s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.625051454 (497h53m41.496134489s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.646488653 (497h53m41.517571688s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.687221335 (497h53m41.55830438s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690272836 (497h53m41.561355871s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690292143 (497h53m41.561375188s) [seg_pos={0 0}, upload=436993]: Segment reached optimal threshold of 13 pieces. Dec 26 09:22:41.690533532 (497h53m41.561616577s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection Dec 26 09:22:41.690545716 (497h53m41.561628751s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690586363 (497h53m41.561669398s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection Dec 26 09:22:41.690602573 (497h53m41.561685618s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690748803 (497h53m41.561831848s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 09:22:41.690758992 (497h53m41.561842037s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690901523 (497h53m41.561984568s) [seg_pos={0 0}, upload=436993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:22:41.691053442 (497h53m41.562136487s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:22:41.691060506 (497h53m41.562143551s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:22:41.976817686 (497h53m41.847900731s) [upload=436993]: Done uploading segments... Dec 26 09:22:41.976839939 (497h53m41.847922984s) [upload=436993]: Upload finished. err: <nil> Duration: 6.062902484s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X.XXX....................X....XXX..X.......X.....XX.XX.XXX.] duration: 6.052775902s trace: https://jaeger.dev.storj.io/trace/389ac4bc3ff91e5c FAIL (2025-12-26 09:23:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding segments... Dec 26 09:22:39.145892195 (497h53m39.01697523s) [upload=436993]: Got next segment. Inline: false Dec 26 09:22:39.145927702 (497h53m39.017010737s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:22:39.145931580 (497h53m39.017014625s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:22:39.388524600 (497h53m39.259607645s) [seg_pos={0 0}, upload=436993]: Begin upload segment... Dec 26 09:22:39.388594132 (497h53m39.259677167s) [seg_pos={0 0}, upload=436993]: Done begin upload segment. Dec 26 09:22:39.388632365 (497h53m39.25971541s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.388653324 (497h53m39.259736359s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389447877 (497h53m39.260530922s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389850545 (497h53m39.26093358s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390368431 (497h53m39.261451476s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390681477 (497h53m39.261764512s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390976849 (497h53m39.262059884s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.392704890 (497h53m39.263787935s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.394668378 (497h53m39.265751413s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.396705798 (497h53m39.267788833s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.399751377 (497h53m39.270834412s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.400608700 (497h53m39.271691745s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.403933580 (497h53m39.275016626s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406414265 (497h53m39.2774973s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406579328 (497h53m39.277662374s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.408705978 (497h53m39.279789023s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:41.096910661 (497h53m40.967993696s) [upload=436993]: Next returned nil segment Dec 26 09:22:41.096923887 (497h53m40.968006922s) [upload=436993]: Waiting for error group managing segments... Dec 26 09:22:41.381072225 (497h53m41.25215526s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.383949273 (497h53m41.255032308s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.396690755 (497h53m41.26777379s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.411478754 (497h53m41.282561799s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421077892 (497h53m41.292160938s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421372643 (497h53m41.292455688s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.434903529 (497h53m41.305986574s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.435526675 (497h53m41.30660972s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.438982786 (497h53m41.310065821s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.625051454 (497h53m41.496134489s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.646488653 (497h53m41.517571688s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.687221335 (497h53m41.55830438s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690272836 (497h53m41.561355871s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690292143 (497h53m41.561375188s) [seg_pos={0 0}, upload=436993]: Segment reached optimal threshold of 13 pieces. Dec 26 09:22:41.690533532 (497h53m41.561616577s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection Dec 26 09:22:41.690545716 (497h53m41.561628751s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690586363 (497h53m41.561669398s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection Dec 26 09:22:41.690602573 (497h53m41.561685618s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690748803 (497h53m41.561831848s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 09:22:41.690758992 (497h53m41.561842037s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690901523 (497h53m41.561984568s) [seg_pos={0 0}, upload=436993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:22:41.691053442 (497h53m41.562136487s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:22:41.691060506 (497h53m41.562143551s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:22:41.976817686 (497h53m41.847900731s) [upload=436993]: Done uploading segments... Dec 26 09:22:41.976839939 (497h53m41.847922984s) [upload=436993]: Upload finished. err: <nil> Duration: 6.052808184s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X.XXX....................X....XXX..X.......X.....XX.XX.XXX] duration: 30.003004703s trace: https://jaeger.dev.storj.io/trace/389ac4bc3ff91e5c FAIL (2025-12-26 09:23:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding segments... Dec 26 09:22:39.145892195 (497h53m39.01697523s) [upload=436993]: Got next segment. Inline: false Dec 26 09:22:39.145927702 (497h53m39.017010737s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:22:39.145931580 (497h53m39.017014625s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:22:39.388524600 (497h53m39.259607645s) [seg_pos={0 0}, upload=436993]: Begin upload segment... Dec 26 09:22:39.388594132 (497h53m39.259677167s) [seg_pos={0 0}, upload=436993]: Done begin upload segment. Dec 26 09:22:39.388632365 (497h53m39.25971541s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.388653324 (497h53m39.259736359s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389447877 (497h53m39.260530922s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.389850545 (497h53m39.26093358s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390368431 (497h53m39.261451476s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390681477 (497h53m39.261764512s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.390976849 (497h53m39.262059884s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.392704890 (497h53m39.263787935s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.394668378 (497h53m39.265751413s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.396705798 (497h53m39.267788833s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.399751377 (497h53m39.270834412s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.400608700 (497h53m39.271691745s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.403933580 (497h53m39.275016626s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406414265 (497h53m39.2774973s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.406579328 (497h53m39.277662374s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:39.408705978 (497h53m39.279789023s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Uploading piece... Dec 26 09:22:41.096910661 (497h53m40.967993696s) [upload=436993]: Next returned nil segment Dec 26 09:22:41.096923887 (497h53m40.968006922s) [upload=436993]: Waiting for error group managing segments... Dec 26 09:22:41.381072225 (497h53m41.25215526s) [noise=true , address=145.79.227.37:4565 , piece_id=HCXKZHJAWK7HDUGTPGNZQTYFOPMPEMLMGY5KJQKVZGBU5RZPEMWQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.383949273 (497h53m41.255032308s) [noise=true , address=145.79.227.51:4409 , piece_id=LHASXQJY4PWF7AUTJCMLNVETWDOLCZWXTLKAXF3N2DK3FNWS7LNA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.396690755 (497h53m41.26777379s) [noise=true , address=145.79.227.12:4767 , piece_id=AY654BY2DSVT5T225UL77XUJNB4J6T6GCGFHVHMRKU33XF7Q3KOA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.411478754 (497h53m41.282561799s) [noise=true , address=145.79.227.11:4730 , piece_id=RPMTQAEVCK6WGKMYPMOJZEOF7W5JHHZLHZR5CTSMEKIUKLXNUP3Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421077892 (497h53m41.292160938s) [noise=true , address=145.79.227.46:4601 , piece_id=AVI75JEPBAH54Q6BFSJNCMUK7BSZXGDKYA5WLANHMZLCX7VWOZ7A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.421372643 (497h53m41.292455688s) [noise=true , address=145.79.227.41:4086 , piece_id=7ZTGTKVTSN4DZIHLA7N44VO5JVZW6O4J6VRV6AAE24KZIB5OAPHA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.434903529 (497h53m41.305986574s) [noise=true , address=145.79.227.5:4222 , piece_id=V2XBQYTXD6FMWS4GKSZYKZNFXWA4MAWM3BDEFHQ22KHNJH7IOCHQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.435526675 (497h53m41.30660972s) [noise=true , address=145.79.227.52:4445 , piece_id=HBT2DH5LJA7VUCGBLDIPLNMWLA4O4XWUTEYQ5HSARURDYR3KLKVQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.438982786 (497h53m41.310065821s) [noise=true , address=145.79.227.45:4230 , piece_id=HQAAP7X2TZK5BJZ2DGGOZG7WPXOZNE5BMVN3TMDWSW5RLIDNG73Q, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.625051454 (497h53m41.496134489s) [noise=true , address=45.91.116.11:4117 , piece_id=4FKD7CRRRLDFKBTKDZCKQOBLCISB7QNLZOPOGL33FTRGXRBMCEZQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.646488653 (497h53m41.517571688s) [noise=true , address=77.90.31.17:4773 , piece_id=LZMU2PHTCMYJPJUPL4ORTV6ZVCWUJ37OTZUORHUOK6DSL3PQPE6A, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.687221335 (497h53m41.55830438s) [noise=true , address=77.90.31.21:4878 , piece_id=AHCJ6HMPZQ6V3DIFLYHBJS6NL7MSZCR4MQRQT3RNKVQ2NK45YVCA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690272836 (497h53m41.561355871s) [noise=true , address=77.90.31.19:4844 , piece_id=Y4C77W7VF3R6ZNTWI6FMWVHFBKJGCH5ZOVIMGZJTDT56YXLEKZGA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: <nil> Dec 26 09:22:41.690292143 (497h53m41.561375188s) [seg_pos={0 0}, upload=436993]: Segment reached optimal threshold of 13 pieces. Dec 26 09:22:41.690533532 (497h53m41.561616577s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53444->45.91.116.5:4300: use of closed network connection Dec 26 09:22:41.690545716 (497h53m41.561628751s) [noise=true , address=45.91.116.5:4300 , piece_id=MWYXMTTTC5XJ6LIQMWT2AMNRU5CBLRVP66GY6UKT7ZWZNZ5GM2LQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690586363 (497h53m41.561669398s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57056->45.91.116.8:4046: use of closed network connection Dec 26 09:22:41.690602573 (497h53m41.561685618s) [noise=true , address=45.91.116.8:4046 , piece_id=OLOLPEBL5EFBOEIJC7UXQ7CECSOX4AZZJVR7FT7B5JG7S46WAIDA, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690748803 (497h53m41.561831848s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 09:22:41.690758992 (497h53m41.561842037s) [noise=true , address=45.91.116.14:4189 , piece_id=SYKNYYQN5OUHF3S6D2FHZJ6IKKCYBS6DBCBHLGAHLLIZIBHFRXCQ, seg_pos={0 0}, upload=436993]: Long tail cancellation due to optimal threshold Dec 26 09:22:41.690901523 (497h53m41.561984568s) [seg_pos={0 0}, upload=436993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:22:41.691053442 (497h53m41.562136487s) [upload=436993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:22:41.691060506 (497h53m41.562143551s) [upload=436993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:22:41.976817686 (497h53m41.847900731s) [upload=436993]: Done uploading segments... Dec 26 09:22:41.976839939 (497h53m41.847922984s) [upload=436993]: Upload finished. err: <nil> Duration: 30.00302881s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X..X.XXX....................X....XXX..X.......X.....XX.XX.XX] duration: 30.002325463s trace: https://jaeger.dev.storj.io/trace/635edce129cc0db7 FAIL (2025-12-26 09:22:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: VFRPHFVFFUSGYPRN7VI3S2BLHVVIIABYRLDXRNUWV4JAW6WXOB3Q) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: AW3EWNN4BZ4NC5WSVXQ2O6NL5O6GIITPO2JZQTN2JCC4EISJV3LA) context deadline exceeded ======LOGS====== 26 09:21:46.403461474 (497h52m46.274544519s) [upload=436981]: Starting upload Dec 26 09:21:46.403511469 (497h52m46.274594514s) [upload=436981]: Done starting upload Dec 26 09:21:46.403858680 (497h52m46.274941715s) [upload=436981]: Uploading segments... Dec 26 09:21:46.403919727 (497h52m46.275002762s) [upload=436981]: Got next segment. Inline: false Dec 26 09:21:46.403948922 (497h52m46.275031957s) [upload=436981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:21:46.403952960 (497h52m46.275035995s) [upload=436981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:21:46.647306216 (497h52m46.518389251s) [seg_pos={0 0}, upload=436981]: Begin upload segment... Dec 26 09:21:46.647349459 (497h52m46.518432494s) [seg_pos={0 0}, upload=436981]: Done begin upload segment. Dec 26 09:21:46.647402579 (497h52m46.518485625s) [noise=true , address=145.79.227.43:4159 , piece_id=5FID3F3OPXSKHYLCO3TZ7VQISKSDAXA7ACIECDQ627NB4AARCN3Q, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.647412368 (497h52m46.518495403s) [noise=true , address=77.90.31.11:4556 , piece_id=KR2ZCHXOSE5R6H2CM5WBN24MROZGJRYRXLYYD7H4BWW66AF6FMJA, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.647572744 (497h52m46.518655779s) [noise=true , address=45.91.116.5:4300 , piece_id=AW3EWNN4BZ4NC5WSVXQ2O6NL5O6GIITPO2JZQTN2JCC4EISJV3LA, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.648030816 (497h52m46.519113851s) [noise=true , address=145.79.227.24:4583 , piece_id=2AZ2HYGMB25UKP2JPEO6G3CC7MMPUVBWUAYXJDU3XDU5GER4EITQ, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.648362508 (497h52m46.519445543s) [noise=true , address=45.91.116.8:4047 , piece_id=5XTTMLK4OXO65TPGLQPJB4KUC6FPFUSJQYTOCOIGM3DABKIBMBLA, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.648696142 (497h52m46.519779177s) [noise=true , address=145.79.227.16:4910 , piece_id=YM5S6PC5XJZUL2AJADKIQ7RFWQTDSPHKAYU5WUWWW35CIDW6CN5A, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.648950698 (497h52m46.520033733s) [noise=true , address=45.91.116.9:4261 , piece_id=ZHHYGQ66RFKTSY7N2JOMNRRJDNO4ZYGARDD423VMQZIHQCFC2GVQ, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.649234328 (497h52m46.520317373s) [noise=true , address=145.79.227.27:4690 , piece_id=D72AGER7VQXQEGLBMXOOIJLLV5A5P5VZT5V6RO6XGSKJ342E5WQA, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.649552293 (497h52m46.520635328s) [noise=true , address=145.79.227.36:4517 , piece_id=DDT2B6DTVIHDBN6ZLZI7LMSHHNLS2P5XXKEYDDMIMWY7HUD7WPEQ, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.649898542 (497h52m46.520981578s) [noise=true , address=145.79.227.11:4732 , piece_id=KQUVJXZWSA2QOX56ACRPJ73O2WKHB4SVQWYVDWTZ5SKRBYEDGNBQ, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.650004073 (497h52m46.521087108s) [noise=true , address=145.79.227.33:4870 , piece_id=QWGK2HCF2O4H6URGUENDGJ4X46XDJRZ5QHGDUPQUGGCO3BUPD54Q, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.650441406 (497h52m46.521524451s) [noise=true , address=45.91.116.13:4154 , piece_id=VFRPHFVFFUSGYPRN7VI3S2BLHVVIIABYRLDXRNUWV4JAW6WXOB3Q, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.650701972 (497h52m46.521785007s) [noise=true , address=45.91.116.4:4940 , piece_id=OJWDYUEOWDKNW5ZFEMPVAX2VTHUOVWIBOUY6MVVHYGUIHTVRKAAA, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.650756926 (497h52m46.521839961s) [noise=true , address=145.79.227.4:4186 , piece_id=HKFOMWU56YFWNXUDIFEVLCZO2I6TP76E667KSV5CS5V57XHROR5A, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.651049103 (497h52m46.522132138s) [noise=true , address=77.90.31.21:4880 , piece_id=WKAMWYGSYFX2OSONUTTQNXYGRRBWLI2TFVFB6OCG4JGLFFLXKXTQ, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:46.651396695 (497h52m46.52247973s) [noise=true , address=145.79.227.50:4373 , piece_id=P56Y6NXR56TU2S6XN5J37VIKZ4XPCM6DXQO7HDQKLGS4UFDZHFXQ, seg_pos={0 0}, upload=436981]: Uploading piece... Dec 26 09:21:48.264473462 (497h52m48.135556497s) [upload=436981]: Next returned nil segment Dec 26 09:21:48.264487469 (497h52m48.135570514s) [upload=436981]: Waiting for error group managing segments... Dec 26 09:21:48.653456122 (497h52m48.524539157s) [noise=true , address=145.79.227.16:4910 , piece_id=YM5S6PC5XJZUL2AJADKIQ7RFWQTDSPHKAYU5WUWWW35CIDW6CN5A, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:48.653597572 (497h52m48.524680617s) [noise=true , address=145.79.227.24:4583 , piece_id=2AZ2HYGMB25UKP2JPEO6G3CC7MMPUVBWUAYXJDU3XDU5GER4EITQ, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:48.660055890 (497h52m48.531138925s) [noise=true , address=145.79.227.36:4517 , piece_id=DDT2B6DTVIHDBN6ZLZI7LMSHHNLS2P5XXKEYDDMIMWY7HUD7WPEQ, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:48.667083301 (497h52m48.538166336s) [noise=true , address=145.79.227.4:4186 , piece_id=HKFOMWU56YFWNXUDIFEVLCZO2I6TP76E667KSV5CS5V57XHROR5A, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:48.672307369 (497h52m48.543390414s) [noise=true , address=145.79.227.50:4373 , piece_id=P56Y6NXR56TU2S6XN5J37VIKZ4XPCM6DXQO7HDQKLGS4UFDZHFXQ, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:48.675114244 (497h52m48.546197279s) [noise=true , address=145.79.227.27:4690 , piece_id=D72AGER7VQXQEGLBMXOOIJLLV5A5P5VZT5V6RO6XGSKJ342E5WQA, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:48.678730930 (497h52m48.549813965s) [noise=true , address=145.79.227.11:4732 , piece_id=KQUVJXZWSA2QOX56ACRPJ73O2WKHB4SVQWYVDWTZ5SKRBYEDGNBQ, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:48.784465517 (497h52m48.655548552s) [noise=true , address=145.79.227.43:4159 , piece_id=5FID3F3OPXSKHYLCO3TZ7VQISKSDAXA7ACIECDQ627NB4AARCN3Q, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:48.905033629 (497h52m48.776116664s) [noise=true , address=145.79.227.33:4870 , piece_id=QWGK2HCF2O4H6URGUENDGJ4X46XDJRZ5QHGDUPQUGGCO3BUPD54Q, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:48.919792152 (497h52m48.790875197s) [noise=true , address=45.91.116.5:4300 , piece_id=AW3EWNN4BZ4NC5WSVXQ2O6NL5O6GIITPO2JZQTN2JCC4EISJV3LA, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:49.024066258 (497h52m48.895149304s) [noise=true , address=77.90.31.11:4556 , piece_id=KR2ZCHXOSE5R6H2CM5WBN24MROZGJRYRXLYYD7H4BWW66AF6FMJA, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:49.048193240 (497h52m48.919276285s) [noise=true , address=45.91.116.4:4940 , piece_id=OJWDYUEOWDKNW5ZFEMPVAX2VTHUOVWIBOUY6MVVHYGUIHTVRKAAA, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:49.763601441 (497h52m49.634684476s) [noise=true , address=45.91.116.13:4154 , piece_id=VFRPHFVFFUSGYPRN7VI3S2BLHVVIIABYRLDXRNUWV4JAW6WXOB3Q, seg_pos={0 0}, upload=436981]: Done uploading piece. err: <nil> Dec 26 09:21:49.763623944 (497h52m49.634706979s) [seg_pos={0 0}, upload=436981]: Segment reached optimal threshold of 13 pieces. Dec 26 09:21:49.763887766 (497h52m49.634970801s) [noise=true , address=77.90.31.21:4880 , piece_id=WKAMWYGSYFX2OSONUTTQNXYGRRBWLI2TFVFB6OCG4JGLFFLXKXTQ, seg_pos={0 0}, upload=436981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:46612->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:46612->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46612->77.90.31.21:4880: use of closed network connection Dec 26 09:21:49.763896643 (497h52m49.634979688s) [noise=true , address=77.90.31.21:4880 , piece_id=WKAMWYGSYFX2OSONUTTQNXYGRRBWLI2TFVFB6OCG4JGLFFLXKXTQ, seg_pos={0 0}, upload=436981]: Long tail cancellation due to optimal threshold Dec 26 09:21:49.763958261 (497h52m49.635041306s) [noise=true , address=45.91.116.9:4261 , piece_id=ZHHYGQ66RFKTSY7N2JOMNRRJDNO4ZYGARDD423VMQZIHQCFC2GVQ, seg_pos={0 0}, upload=436981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 09:21:49.763968891 (497h52m49.635051956s) [noise=true , address=45.91.116.9:4261 , piece_id=ZHHYGQ66RFKTSY7N2JOMNRRJDNO4ZYGARDD423VMQZIHQCFC2GVQ, seg_pos={0 0}, upload=436981]: Long tail cancellation due to optimal threshold Dec 26 09:21:49.764188899 (497h52m49.635271935s) [noise=true , address=45.91.116.8:4047 , piece_id=5XTTMLK4OXO65TPGLQPJB4KUC6FPFUSJQYTOCOIGM3DABKIBMBLA, seg_pos={0 0}, upload=436981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 09:21:49.764198507 (497h52m49.635281553s) [noise=true , address=45.91.116.8:4047 , piece_id=5XTTMLK4OXO65TPGLQPJB4KUC6FPFUSJQYTOCOIGM3DABKIBMBLA, seg_pos={0 0}, upload=436981]: Long tail cancellation due to optimal threshold Dec 26 09:21:49.764222283 (497h52m49.635305328s) [seg_pos={0 0}, upload=436981]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:21:49.764327614 (497h52m49.635410649s) [upload=436981]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:21:49.764334797 (497h52m49.635417832s) [upload=436981]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:21:50.061515470 (497h52m49.932598505s) [upload=436981]: Done uploading segments... Dec 26 09:21:50.061533525 (497h52m49.93261656s) [upload=436981]: Upload finished. err: <nil> Duration: 30.002349898s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX..X.XXX....................X....XXX..X.......X.....XX.XX.X] duration: 30.003183353s trace: https://jaeger.dev.storj.io/trace/4f8203c617f957b1 FAIL (2025-12-26 09:21:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: 25RBE2EHWTYANKZ43DNGOA5DY557NEQYAWLZLFHWQJ7DFE4H43MQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 2ND2LKYAJ524K6BEO5TU3KGQHJ6ZF7MZT2JJ2ES3VWSUTCGBHX4Q) context deadline exceeded ======LOGS====== 97h51m46.506041203s) [upload=436968]: Done starting upload Dec 26 09:20:46.635092524 (497h51m46.506175559s) [upload=436968]: Uploading segments... Dec 26 09:20:46.635257146 (497h51m46.506340181s) [upload=436968]: Got next segment. Inline: false Dec 26 09:20:46.635303966 (497h51m46.506387001s) [upload=436968]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:20:46.635310869 (497h51m46.506393904s) [upload=436968]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:20:46.880233915 (497h51m46.75131696s) [seg_pos={0 0}, upload=436968]: Begin upload segment... Dec 26 09:20:46.880298448 (497h51m46.751381493s) [seg_pos={0 0}, upload=436968]: Done begin upload segment. Dec 26 09:20:46.880376086 (497h51m46.751459121s) [noise=true , address=45.91.116.5:4300 , piece_id=RHDK5XINIGJSU33ZL7U45RP537ECE5QFC6TUCSI7FHP4XPYP4OPQ, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.880412675 (497h51m46.75149572s) [noise=true , address=145.79.227.45:4230 , piece_id=NVPE4OQRKKHBN2TOX475UJSDWUHTTFTBFISKKIOQJP6BNYHI4G2Q, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.880923558 (497h51m46.752006593s) [noise=true , address=145.79.227.46:4601 , piece_id=4Y4E3NFZT5CL3A4X23O4OJLH2QZD7HHC74KN3ON6LV37KHT2MXPA, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.881327978 (497h51m46.752411013s) [noise=true , address=145.79.227.15:4875 , piece_id=DTEYPG3RIGL3ADEINNI7AQIFD2YZ52KPXYRFZ4PV4FXTWCNXXTFA, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.881738971 (497h51m46.752822016s) [noise=true , address=45.91.116.7:4012 , piece_id=BV7LPZTAJYJ3F2PDHYOYSR7TYPPZGAUT5VTT6BCQYGHSD64FUAZA, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.882266004 (497h51m46.753349049s) [noise=true , address=145.79.227.41:4085 , piece_id=PK2GU7N3ANQ3SUTRWECXO3FEKLQE53KCUUDYRVMTL5H4LQ7RYNXQ, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.882699660 (497h51m46.753782695s) [noise=true , address=145.79.227.19:4079 , piece_id=7G6HP436N5QNWRT56S4HVJ4UFFGKEM4WY3L42TJNY57CHQ3GHRFA, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.883123627 (497h51m46.754206673s) [noise=true , address=145.79.227.36:4519 , piece_id=OKPLE32ZHCGAXG6RXCKLS7EU6QDYRDOW24FLVNI5XSZ356GXOH4A, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.883646082 (497h51m46.754729127s) [noise=true , address=45.91.116.9:4264 , piece_id=2ND2LKYAJ524K6BEO5TU3KGQHJ6ZF7MZT2JJ2ES3VWSUTCGBHX4Q, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.884950786 (497h51m46.756033821s) [noise=true , address=145.79.227.2:4113 , piece_id=LCE2WXLLK47ZNIBRDV6PWRLSEVN3Y35HKROO3M73FH2QU3YDZH4Q, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.885235760 (497h51m46.756318805s) [noise=true , address=145.79.227.17:4971 , piece_id=O4RUOKWECAN6X46WFMLFMO47WKIUOP6C33I6XFDIAO3ZI2F7NU7A, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.886712912 (497h51m46.757795957s) [noise=true , address=45.91.116.13:4155 , piece_id=25RBE2EHWTYANKZ43DNGOA5DY557NEQYAWLZLFHWQJ7DFE4H43MQ, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.886935636 (497h51m46.758018681s) [noise=true , address=45.91.116.10:4083 , piece_id=WF73W5YJ2GGPUCSPOQJNSH4RRMLCMFM6DCEF46WQNKXEKGV5H43A, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.888667403 (497h51m46.759750439s) [noise=true , address=77.90.31.13:4629 , piece_id=JMUD2VDIJI42NZNLHY7ZCLLPYR6ZZRFTVOPCIPRG4WTPBFNFFXAQ, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.891870473 (497h51m46.762953518s) [noise=true , address=145.79.227.49:4339 , piece_id=AFZSOFD5ODAEYJXHEAXCY4YZ7UV63J7JDNJG5BZJGZ2JD3ZP5SOA, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:46.895627537 (497h51m46.766710582s) [noise=true , address=45.91.116.14:4192 , piece_id=3Y6WEL7OSABCDL6G5C3HA53NAA6JWTIZACXJNWF36F2NKHWZNE7Q, seg_pos={0 0}, upload=436968]: Uploading piece... Dec 26 09:20:48.610310399 (497h51m48.481393434s) [upload=436968]: Next returned nil segment Dec 26 09:20:48.610332281 (497h51m48.481415316s) [upload=436968]: Waiting for error group managing segments... Dec 26 09:20:48.897403625 (497h51m48.76848667s) [noise=true , address=145.79.227.2:4113 , piece_id=LCE2WXLLK47ZNIBRDV6PWRLSEVN3Y35HKROO3M73FH2QU3YDZH4Q, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:48.898371066 (497h51m48.769454111s) [noise=true , address=145.79.227.46:4601 , piece_id=4Y4E3NFZT5CL3A4X23O4OJLH2QZD7HHC74KN3ON6LV37KHT2MXPA, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:48.898631402 (497h51m48.769714447s) [noise=true , address=145.79.227.19:4079 , piece_id=7G6HP436N5QNWRT56S4HVJ4UFFGKEM4WY3L42TJNY57CHQ3GHRFA, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:48.899402430 (497h51m48.770485465s) [noise=true , address=145.79.227.41:4085 , piece_id=PK2GU7N3ANQ3SUTRWECXO3FEKLQE53KCUUDYRVMTL5H4LQ7RYNXQ, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:48.901736314 (497h51m48.772819349s) [noise=true , address=145.79.227.15:4875 , piece_id=DTEYPG3RIGL3ADEINNI7AQIFD2YZ52KPXYRFZ4PV4FXTWCNXXTFA, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:48.905576196 (497h51m48.776659231s) [noise=true , address=145.79.227.45:4230 , piece_id=NVPE4OQRKKHBN2TOX475UJSDWUHTTFTBFISKKIOQJP6BNYHI4G2Q, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:48.911288242 (497h51m48.782371287s) [noise=true , address=145.79.227.36:4519 , piece_id=OKPLE32ZHCGAXG6RXCKLS7EU6QDYRDOW24FLVNI5XSZ356GXOH4A, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:48.912763823 (497h51m48.783846858s) [noise=true , address=145.79.227.49:4339 , piece_id=AFZSOFD5ODAEYJXHEAXCY4YZ7UV63J7JDNJG5BZJGZ2JD3ZP5SOA, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:48.913918571 (497h51m48.785001606s) [noise=true , address=145.79.227.17:4971 , piece_id=O4RUOKWECAN6X46WFMLFMO47WKIUOP6C33I6XFDIAO3ZI2F7NU7A, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:49.151944564 (497h51m49.023027609s) [noise=true , address=45.91.116.13:4155 , piece_id=25RBE2EHWTYANKZ43DNGOA5DY557NEQYAWLZLFHWQJ7DFE4H43MQ, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:49.182025213 (497h51m49.053108248s) [noise=true , address=45.91.116.5:4300 , piece_id=RHDK5XINIGJSU33ZL7U45RP537ECE5QFC6TUCSI7FHP4XPYP4OPQ, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:49.268589917 (497h51m49.139672962s) [noise=true , address=45.91.116.9:4264 , piece_id=2ND2LKYAJ524K6BEO5TU3KGQHJ6ZF7MZT2JJ2ES3VWSUTCGBHX4Q, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:49.372518365 (497h51m49.2436014s) [noise=true , address=45.91.116.14:4192 , piece_id=3Y6WEL7OSABCDL6G5C3HA53NAA6JWTIZACXJNWF36F2NKHWZNE7Q, seg_pos={0 0}, upload=436968]: Done uploading piece. err: <nil> Dec 26 09:20:49.372542651 (497h51m49.243625696s) [seg_pos={0 0}, upload=436968]: Segment reached optimal threshold of 13 pieces. Dec 26 09:20:49.372783179 (497h51m49.243866214s) [noise=true , address=77.90.31.13:4629 , piece_id=JMUD2VDIJI42NZNLHY7ZCLLPYR6ZZRFTVOPCIPRG4WTPBFNFFXAQ, seg_pos={0 0}, upload=436968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 26 09:20:49.372793038 (497h51m49.243876093s) [noise=true , address=77.90.31.13:4629 , piece_id=JMUD2VDIJI42NZNLHY7ZCLLPYR6ZZRFTVOPCIPRG4WTPBFNFFXAQ, seg_pos={0 0}, upload=436968]: Long tail cancellation due to optimal threshold Dec 26 09:20:49.373018356 (497h51m49.244101391s) [noise=true , address=45.91.116.10:4083 , piece_id=WF73W5YJ2GGPUCSPOQJNSH4RRMLCMFM6DCEF46WQNKXEKGV5H43A, seg_pos={0 0}, upload=436968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35020->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35020->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35020->45.91.116.10:4083: use of closed network connection Dec 26 09:20:49.373039357 (497h51m49.244122402s) [noise=true , address=45.91.116.10:4083 , piece_id=WF73W5YJ2GGPUCSPOQJNSH4RRMLCMFM6DCEF46WQNKXEKGV5H43A, seg_pos={0 0}, upload=436968]: Long tail cancellation due to optimal threshold Dec 26 09:20:49.373256840 (497h51m49.244339906s) [noise=true , address=45.91.116.7:4012 , piece_id=BV7LPZTAJYJ3F2PDHYOYSR7TYPPZGAUT5VTT6BCQYGHSD64FUAZA, seg_pos={0 0}, upload=436968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: read tcp 188.166.225.170:51752->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 26 09:20:49.373267041 (497h51m49.244350086s) [noise=true , address=45.91.116.7:4012 , piece_id=BV7LPZTAJYJ3F2PDHYOYSR7TYPPZGAUT5VTT6BCQYGHSD64FUAZA, seg_pos={0 0}, upload=436968]: Long tail cancellation due to optimal threshold Dec 26 09:20:49.373293250 (497h51m49.244376285s) [seg_pos={0 0}, upload=436968]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:20:49.373374585 (497h51m49.24445763s) [upload=436968]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:20:49.373380556 (497h51m49.244463602s) [upload=436968]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:20:49.659951867 (497h51m49.531034903s) [upload=436968]: Done uploading segments... Dec 26 09:20:49.659970984 (497h51m49.531054019s) [upload=436968]: Upload finished. err: <nil> Duration: 30.003205645s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 42.668587794s trace: https://jaeger.dev.storj.io/trace/6820217d73ecf2a FAIL (2025-12-26 09:20:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments... Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces. Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments... Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil> Duration: 42.668613933s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.XX..X.XXX....................X....XXX..X.......X.....XX.XX.] duration: 8.484784465s trace: https://jaeger.dev.storj.io/trace/6cf45297e66e6078 FAIL (2025-12-26 09:20:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: QNOMIDO2QA25K6NDBCOCUFOD4OXJCCRBWXN7M3D4W6IGELC6YYYA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: BHUA4UOWWCKWN7XFPMIXPXLEQNJL4BLXBFZ44USNIF7M2NJOQTQA) context deadline exceeded ======LOGS====== ec 26 09:19:34.243162290 (497h50m34.114245325s) [upload=436951]: Starting upload Dec 26 09:19:34.243234257 (497h50m34.114317302s) [upload=436951]: Done starting upload Dec 26 09:19:34.243351129 (497h50m34.114434174s) [upload=436951]: Uploading segments... Dec 26 09:19:34.243475266 (497h50m34.114558311s) [upload=436951]: Got next segment. Inline: false Dec 26 09:19:34.243532084 (497h50m34.114615119s) [upload=436951]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:19:34.243537033 (497h50m34.114620068s) [upload=436951]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:19:34.493547967 (497h50m34.364631012s) [seg_pos={0 0}, upload=436951]: Begin upload segment... Dec 26 09:19:34.493604515 (497h50m34.36468756s) [seg_pos={0 0}, upload=436951]: Done begin upload segment. Dec 26 09:19:34.493682744 (497h50m34.364765779s) [noise=true , address=45.91.116.4:4939 , piece_id=Q5O3CNTDJM5N3BW4HLOWMJ2HTEM3ATHMCJYP5AV5OIAKZG5K5P3Q, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.493697582 (497h50m34.364780618s) [noise=true , address=145.79.227.39:4015 , piece_id=N3RCQRN3KALLO6XJGI5GFQRA2VT6CD6DKZSPZYD5YAZZWRJ6SRAA, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.494371916 (497h50m34.365454961s) [noise=true , address=145.79.227.19:4078 , piece_id=NY2XMF5NLKMBN4FONPW3J4HN45NP2IBJKLSVWXTDWXQ4XQUNK4TQ, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.494723516 (497h50m34.365806551s) [noise=true , address=145.79.227.4:4187 , piece_id=UZJXUK6CYOLYWTTU7FE5OT6VH6C6PRCGPNYPUMJLAG5EKIKW2I6A, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.495048936 (497h50m34.366131971s) [noise=true , address=145.79.227.41:4085 , piece_id=6YYVGFSDHPTQVHNTLCHPJG2SGVU4G7FC2GF35IRB27344YOQGNSA, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.495383352 (497h50m34.366466387s) [noise=true , address=145.79.227.33:4871 , piece_id=RCLP635VNAI2DM2LAWOHBMB5KJXDUD7ZBKVWO4OMZSAH5IQLP6BA, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.495700406 (497h50m34.366783441s) [noise=true , address=145.79.227.5:4223 , piece_id=EJIBWPM2QZBKH6UFMAFYG6SK5YD6DBCCUBY6VSC5GHQPFYDITGMA, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.496041465 (497h50m34.3671245s) [noise=true , address=45.91.116.9:4263 , piece_id=VIFNP7GPPVILODXWTXYFI5I7RNVFDLBGYGCLVQF7YDYTSN6YDTSQ, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.496401470 (497h50m34.367484506s) [noise=true , address=145.79.227.13:4804 , piece_id=USXXM6IFES56NRIMPAYZSHGTSWQSOWXVMT6JQNIWOIPNU7THEEIQ, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.497715623 (497h50m34.368798668s) [noise=true , address=77.90.31.19:4844 , piece_id=QNOMIDO2QA25K6NDBCOCUFOD4OXJCCRBWXN7M3D4W6IGELC6YYYA, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.499133834 (497h50m34.370216869s) [noise=true , address=45.91.116.12:4225 , piece_id=BHUA4UOWWCKWN7XFPMIXPXLEQNJL4BLXBFZ44USNIF7M2NJOQTQA, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.500473124 (497h50m34.371556159s) [noise=true , address=77.90.31.10:4518 , piece_id=SGFGD626DSAWGQ6S7DTTEQQBVRBBLRBHGDGLE43DRZSJSURDNDCA, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.503103823 (497h50m34.374186858s) [noise=true , address=145.79.227.10:4403 , piece_id=N7PXIYM4OUYPG4SC4EFXCR6TA3RFEFP2OB3CFV35BWTR6O6UDWDQ, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.504165134 (497h50m34.375248169s) [noise=true , address=145.79.227.49:4337 , piece_id=HZZNJY57PXFMPIGBM7JNWRT4TL4XF2E2D2JGKP24T3CE3AS7G4EQ, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.507468755 (497h50m34.37855179s) [noise=true , address=145.79.227.20:4437 , piece_id=2FAAM7X5PNB42VYAMIF5UY7OGDBYTZXH5A4FWEC5CWEQ5KK4AOTA, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.509623317 (497h50m34.380706352s) [noise=true , address=45.91.116.13:4153 , piece_id=WMDD5PIQ5XT3WLXFMCM2F62ZEXU5TWAZVTMBI4YLLNJAMVU53CIQ, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:36.130524193 (497h50m36.001607238s) [upload=436951]: Next returned nil segment Dec 26 09:19:36.130544551 (497h50m36.001627586s) [upload=436951]: Waiting for error group managing segments... Dec 26 09:19:36.514115216 (497h50m36.385198251s) [noise=true , address=145.79.227.13:4804 , piece_id=USXXM6IFES56NRIMPAYZSHGTSWQSOWXVMT6JQNIWOIPNU7THEEIQ, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.520861964 (497h50m36.391944999s) [noise=true , address=145.79.227.5:4223 , piece_id=EJIBWPM2QZBKH6UFMAFYG6SK5YD6DBCCUBY6VSC5GHQPFYDITGMA, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.521014353 (497h50m36.392097398s) [noise=true , address=145.79.227.10:4403 , piece_id=N7PXIYM4OUYPG4SC4EFXCR6TA3RFEFP2OB3CFV35BWTR6O6UDWDQ, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.521551286 (497h50m36.392634331s) [noise=true , address=145.79.227.41:4085 , piece_id=6YYVGFSDHPTQVHNTLCHPJG2SGVU4G7FC2GF35IRB27344YOQGNSA, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.522644047 (497h50m36.393727092s) [noise=true , address=145.79.227.19:4078 , piece_id=NY2XMF5NLKMBN4FONPW3J4HN45NP2IBJKLSVWXTDWXQ4XQUNK4TQ, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.523320115 (497h50m36.39440315s) [noise=true , address=145.79.227.39:4015 , piece_id=N3RCQRN3KALLO6XJGI5GFQRA2VT6CD6DKZSPZYD5YAZZWRJ6SRAA, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.537183583 (497h50m36.408266618s) [noise=true , address=145.79.227.49:4337 , piece_id=HZZNJY57PXFMPIGBM7JNWRT4TL4XF2E2D2JGKP24T3CE3AS7G4EQ, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.540792886 (497h50m36.411875921s) [noise=true , address=145.79.227.20:4437 , piece_id=2FAAM7X5PNB42VYAMIF5UY7OGDBYTZXH5A4FWEC5CWEQ5KK4AOTA, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.552161884 (497h50m36.423244929s) [noise=true , address=145.79.227.4:4187 , piece_id=UZJXUK6CYOLYWTTU7FE5OT6VH6C6PRCGPNYPUMJLAG5EKIKW2I6A, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.749919438 (497h50m36.621002474s) [noise=true , address=45.91.116.13:4153 , piece_id=WMDD5PIQ5XT3WLXFMCM2F62ZEXU5TWAZVTMBI4YLLNJAMVU53CIQ, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.852600672 (497h50m36.723683707s) [noise=true , address=77.90.31.19:4844 , piece_id=QNOMIDO2QA25K6NDBCOCUFOD4OXJCCRBWXN7M3D4W6IGELC6YYYA, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.869190671 (497h50m36.740273706s) [noise=true , address=77.90.31.10:4518 , piece_id=SGFGD626DSAWGQ6S7DTTEQQBVRBBLRBHGDGLE43DRZSJSURDNDCA, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:37.173093355 (497h50m37.04417639s) [noise=true , address=45.91.116.12:4225 , piece_id=BHUA4UOWWCKWN7XFPMIXPXLEQNJL4BLXBFZ44USNIF7M2NJOQTQA, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:37.173117691 (497h50m37.044200736s) [seg_pos={0 0}, upload=436951]: Segment reached optimal threshold of 13 pieces. Dec 26 09:19:37.173424295 (497h50m37.04450734s) [noise=true , address=145.79.227.33:4871 , piece_id=RCLP635VNAI2DM2LAWOHBMB5KJXDUD7ZBKVWO4OMZSAH5IQLP6BA, seg_pos={0 0}, upload=436951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 09:19:37.173435377 (497h50m37.044518442s) [noise=true , address=145.79.227.33:4871 , piece_id=RCLP635VNAI2DM2LAWOHBMB5KJXDUD7ZBKVWO4OMZSAH5IQLP6BA, seg_pos={0 0}, upload=436951]: Long tail cancellation due to optimal threshold Dec 26 09:19:37.173467287 (497h50m37.044550322s) [noise=true , address=45.91.116.4:4939 , piece_id=Q5O3CNTDJM5N3BW4HLOWMJ2HTEM3ATHMCJYP5AV5OIAKZG5K5P3Q, seg_pos={0 0}, upload=436951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 09:19:37.173482216 (497h50m37.044565251s) [noise=true , address=45.91.116.4:4939 , piece_id=Q5O3CNTDJM5N3BW4HLOWMJ2HTEM3ATHMCJYP5AV5OIAKZG5K5P3Q, seg_pos={0 0}, upload=436951]: Long tail cancellation due to optimal threshold Dec 26 09:19:37.173648993 (497h50m37.044732038s) [noise=true , address=45.91.116.9:4263 , piece_id=VIFNP7GPPVILODXWTXYFI5I7RNVFDLBGYGCLVQF7YDYTSN6YDTSQ, seg_pos={0 0}, upload=436951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:32934->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:32934->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32934->45.91.116.9:4263: use of closed network connection Dec 26 09:19:37.173661867 (497h50m37.044744913s) [noise=true , address=45.91.116.9:4263 , piece_id=VIFNP7GPPVILODXWTXYFI5I7RNVFDLBGYGCLVQF7YDYTSN6YDTSQ, seg_pos={0 0}, upload=436951]: Long tail cancellation due to optimal threshold Dec 26 09:19:37.173679512 (497h50m37.044762557s) [seg_pos={0 0}, upload=436951]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:19:37.173754224 (497h50m37.044837259s) [upload=436951]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:19:37.173759133 (497h50m37.044842168s) [upload=436951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:19:37.470303114 (497h50m37.341386149s) [upload=436951]: Done uploading segments... Dec 26 09:19:37.470324103 (497h50m37.341407138s) [upload=436951]: Upload finished. err: <nil> Duration: 8.4848075s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.XX..X.XXX....................X....XXX..X.......X.....XX.XX] duration: 30.005942309s trace: https://jaeger.dev.storj.io/trace/6cf45297e66e6078 FAIL (2025-12-26 09:20:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: QNOMIDO2QA25K6NDBCOCUFOD4OXJCCRBWXN7M3D4W6IGELC6YYYA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: BHUA4UOWWCKWN7XFPMIXPXLEQNJL4BLXBFZ44USNIF7M2NJOQTQA) context deadline exceeded ======LOGS====== ec 26 09:19:34.243162290 (497h50m34.114245325s) [upload=436951]: Starting upload Dec 26 09:19:34.243234257 (497h50m34.114317302s) [upload=436951]: Done starting upload Dec 26 09:19:34.243351129 (497h50m34.114434174s) [upload=436951]: Uploading segments... Dec 26 09:19:34.243475266 (497h50m34.114558311s) [upload=436951]: Got next segment. Inline: false Dec 26 09:19:34.243532084 (497h50m34.114615119s) [upload=436951]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:19:34.243537033 (497h50m34.114620068s) [upload=436951]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:19:34.493547967 (497h50m34.364631012s) [seg_pos={0 0}, upload=436951]: Begin upload segment... Dec 26 09:19:34.493604515 (497h50m34.36468756s) [seg_pos={0 0}, upload=436951]: Done begin upload segment. Dec 26 09:19:34.493682744 (497h50m34.364765779s) [noise=true , address=45.91.116.4:4939 , piece_id=Q5O3CNTDJM5N3BW4HLOWMJ2HTEM3ATHMCJYP5AV5OIAKZG5K5P3Q, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.493697582 (497h50m34.364780618s) [noise=true , address=145.79.227.39:4015 , piece_id=N3RCQRN3KALLO6XJGI5GFQRA2VT6CD6DKZSPZYD5YAZZWRJ6SRAA, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.494371916 (497h50m34.365454961s) [noise=true , address=145.79.227.19:4078 , piece_id=NY2XMF5NLKMBN4FONPW3J4HN45NP2IBJKLSVWXTDWXQ4XQUNK4TQ, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.494723516 (497h50m34.365806551s) [noise=true , address=145.79.227.4:4187 , piece_id=UZJXUK6CYOLYWTTU7FE5OT6VH6C6PRCGPNYPUMJLAG5EKIKW2I6A, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.495048936 (497h50m34.366131971s) [noise=true , address=145.79.227.41:4085 , piece_id=6YYVGFSDHPTQVHNTLCHPJG2SGVU4G7FC2GF35IRB27344YOQGNSA, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.495383352 (497h50m34.366466387s) [noise=true , address=145.79.227.33:4871 , piece_id=RCLP635VNAI2DM2LAWOHBMB5KJXDUD7ZBKVWO4OMZSAH5IQLP6BA, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.495700406 (497h50m34.366783441s) [noise=true , address=145.79.227.5:4223 , piece_id=EJIBWPM2QZBKH6UFMAFYG6SK5YD6DBCCUBY6VSC5GHQPFYDITGMA, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.496041465 (497h50m34.3671245s) [noise=true , address=45.91.116.9:4263 , piece_id=VIFNP7GPPVILODXWTXYFI5I7RNVFDLBGYGCLVQF7YDYTSN6YDTSQ, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.496401470 (497h50m34.367484506s) [noise=true , address=145.79.227.13:4804 , piece_id=USXXM6IFES56NRIMPAYZSHGTSWQSOWXVMT6JQNIWOIPNU7THEEIQ, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.497715623 (497h50m34.368798668s) [noise=true , address=77.90.31.19:4844 , piece_id=QNOMIDO2QA25K6NDBCOCUFOD4OXJCCRBWXN7M3D4W6IGELC6YYYA, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.499133834 (497h50m34.370216869s) [noise=true , address=45.91.116.12:4225 , piece_id=BHUA4UOWWCKWN7XFPMIXPXLEQNJL4BLXBFZ44USNIF7M2NJOQTQA, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.500473124 (497h50m34.371556159s) [noise=true , address=77.90.31.10:4518 , piece_id=SGFGD626DSAWGQ6S7DTTEQQBVRBBLRBHGDGLE43DRZSJSURDNDCA, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.503103823 (497h50m34.374186858s) [noise=true , address=145.79.227.10:4403 , piece_id=N7PXIYM4OUYPG4SC4EFXCR6TA3RFEFP2OB3CFV35BWTR6O6UDWDQ, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.504165134 (497h50m34.375248169s) [noise=true , address=145.79.227.49:4337 , piece_id=HZZNJY57PXFMPIGBM7JNWRT4TL4XF2E2D2JGKP24T3CE3AS7G4EQ, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.507468755 (497h50m34.37855179s) [noise=true , address=145.79.227.20:4437 , piece_id=2FAAM7X5PNB42VYAMIF5UY7OGDBYTZXH5A4FWEC5CWEQ5KK4AOTA, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:34.509623317 (497h50m34.380706352s) [noise=true , address=45.91.116.13:4153 , piece_id=WMDD5PIQ5XT3WLXFMCM2F62ZEXU5TWAZVTMBI4YLLNJAMVU53CIQ, seg_pos={0 0}, upload=436951]: Uploading piece... Dec 26 09:19:36.130524193 (497h50m36.001607238s) [upload=436951]: Next returned nil segment Dec 26 09:19:36.130544551 (497h50m36.001627586s) [upload=436951]: Waiting for error group managing segments... Dec 26 09:19:36.514115216 (497h50m36.385198251s) [noise=true , address=145.79.227.13:4804 , piece_id=USXXM6IFES56NRIMPAYZSHGTSWQSOWXVMT6JQNIWOIPNU7THEEIQ, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.520861964 (497h50m36.391944999s) [noise=true , address=145.79.227.5:4223 , piece_id=EJIBWPM2QZBKH6UFMAFYG6SK5YD6DBCCUBY6VSC5GHQPFYDITGMA, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.521014353 (497h50m36.392097398s) [noise=true , address=145.79.227.10:4403 , piece_id=N7PXIYM4OUYPG4SC4EFXCR6TA3RFEFP2OB3CFV35BWTR6O6UDWDQ, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.521551286 (497h50m36.392634331s) [noise=true , address=145.79.227.41:4085 , piece_id=6YYVGFSDHPTQVHNTLCHPJG2SGVU4G7FC2GF35IRB27344YOQGNSA, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.522644047 (497h50m36.393727092s) [noise=true , address=145.79.227.19:4078 , piece_id=NY2XMF5NLKMBN4FONPW3J4HN45NP2IBJKLSVWXTDWXQ4XQUNK4TQ, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.523320115 (497h50m36.39440315s) [noise=true , address=145.79.227.39:4015 , piece_id=N3RCQRN3KALLO6XJGI5GFQRA2VT6CD6DKZSPZYD5YAZZWRJ6SRAA, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.537183583 (497h50m36.408266618s) [noise=true , address=145.79.227.49:4337 , piece_id=HZZNJY57PXFMPIGBM7JNWRT4TL4XF2E2D2JGKP24T3CE3AS7G4EQ, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.540792886 (497h50m36.411875921s) [noise=true , address=145.79.227.20:4437 , piece_id=2FAAM7X5PNB42VYAMIF5UY7OGDBYTZXH5A4FWEC5CWEQ5KK4AOTA, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.552161884 (497h50m36.423244929s) [noise=true , address=145.79.227.4:4187 , piece_id=UZJXUK6CYOLYWTTU7FE5OT6VH6C6PRCGPNYPUMJLAG5EKIKW2I6A, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.749919438 (497h50m36.621002474s) [noise=true , address=45.91.116.13:4153 , piece_id=WMDD5PIQ5XT3WLXFMCM2F62ZEXU5TWAZVTMBI4YLLNJAMVU53CIQ, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.852600672 (497h50m36.723683707s) [noise=true , address=77.90.31.19:4844 , piece_id=QNOMIDO2QA25K6NDBCOCUFOD4OXJCCRBWXN7M3D4W6IGELC6YYYA, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:36.869190671 (497h50m36.740273706s) [noise=true , address=77.90.31.10:4518 , piece_id=SGFGD626DSAWGQ6S7DTTEQQBVRBBLRBHGDGLE43DRZSJSURDNDCA, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:37.173093355 (497h50m37.04417639s) [noise=true , address=45.91.116.12:4225 , piece_id=BHUA4UOWWCKWN7XFPMIXPXLEQNJL4BLXBFZ44USNIF7M2NJOQTQA, seg_pos={0 0}, upload=436951]: Done uploading piece. err: <nil> Dec 26 09:19:37.173117691 (497h50m37.044200736s) [seg_pos={0 0}, upload=436951]: Segment reached optimal threshold of 13 pieces. Dec 26 09:19:37.173424295 (497h50m37.04450734s) [noise=true , address=145.79.227.33:4871 , piece_id=RCLP635VNAI2DM2LAWOHBMB5KJXDUD7ZBKVWO4OMZSAH5IQLP6BA, seg_pos={0 0}, upload=436951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 09:19:37.173435377 (497h50m37.044518442s) [noise=true , address=145.79.227.33:4871 , piece_id=RCLP635VNAI2DM2LAWOHBMB5KJXDUD7ZBKVWO4OMZSAH5IQLP6BA, seg_pos={0 0}, upload=436951]: Long tail cancellation due to optimal threshold Dec 26 09:19:37.173467287 (497h50m37.044550322s) [noise=true , address=45.91.116.4:4939 , piece_id=Q5O3CNTDJM5N3BW4HLOWMJ2HTEM3ATHMCJYP5AV5OIAKZG5K5P3Q, seg_pos={0 0}, upload=436951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 09:19:37.173482216 (497h50m37.044565251s) [noise=true , address=45.91.116.4:4939 , piece_id=Q5O3CNTDJM5N3BW4HLOWMJ2HTEM3ATHMCJYP5AV5OIAKZG5K5P3Q, seg_pos={0 0}, upload=436951]: Long tail cancellation due to optimal threshold Dec 26 09:19:37.173648993 (497h50m37.044732038s) [noise=true , address=45.91.116.9:4263 , piece_id=VIFNP7GPPVILODXWTXYFI5I7RNVFDLBGYGCLVQF7YDYTSN6YDTSQ, seg_pos={0 0}, upload=436951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:32934->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:32934->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32934->45.91.116.9:4263: use of closed network connection Dec 26 09:19:37.173661867 (497h50m37.044744913s) [noise=true , address=45.91.116.9:4263 , piece_id=VIFNP7GPPVILODXWTXYFI5I7RNVFDLBGYGCLVQF7YDYTSN6YDTSQ, seg_pos={0 0}, upload=436951]: Long tail cancellation due to optimal threshold Dec 26 09:19:37.173679512 (497h50m37.044762557s) [seg_pos={0 0}, upload=436951]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:19:37.173754224 (497h50m37.044837259s) [upload=436951]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:19:37.173759133 (497h50m37.044842168s) [upload=436951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:19:37.470303114 (497h50m37.341386149s) [upload=436951]: Done uploading segments... Dec 26 09:19:37.470324103 (497h50m37.341407138s) [upload=436951]: Upload finished. err: <nil> Duration: 30.005958009s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 28.612840644s trace: https://jaeger.dev.storj.io/trace/6820217d73ecf2a FAIL (2025-12-26 09:19:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments... Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces. Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments... Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil> Duration: 28.612864338s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.XX..X.XXX....................X....XXX..X.......X.....XX.X] duration: 30.002760358s trace: https://jaeger.dev.storj.io/trace/9918bb74091e6a8 FAIL (2025-12-26 09:19:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 5CKTAS4I5X37CR4YBGHV752KVMCMQU6VCEAW2RLIOOZMQBTMLJVQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: KZHOYEHYOIXUALTMTFSFQSFBRF4UGXA7FTCCOSMCDTJHI5LJ3BOA) context deadline exceeded ======LOGS====== Dec 26 09:19:03.990924512 (497h50m3.862007557s) [upload=436942]: Starting upload Dec 26 09:19:03.991001408 (497h50m3.862084453s) [upload=436942]: Done starting upload Dec 26 09:19:03.991235694 (497h50m3.862318739s) [upload=436942]: Uploading segments... Dec 26 09:19:03.991292181 (497h50m3.862375246s) [upload=436942]: Got next segment. Inline: false Dec 26 09:19:03.991329482 (497h50m3.862412527s) [upload=436942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:19:03.991335665 (497h50m3.86241871s) [upload=436942]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:19:04.238737781 (497h50m4.109820836s) [seg_pos={0 0}, upload=436942]: Begin upload segment... Dec 26 09:19:04.238809979 (497h50m4.109893024s) [seg_pos={0 0}, upload=436942]: Done begin upload segment. Dec 26 09:19:04.238864382 (497h50m4.109947427s) [noise=true , address=145.79.227.16:4910 , piece_id=MM5KOERT2WCFTRALXGZB4HJ5X2COBCLWKGHRVB4HRJJNIAQVI5MQ, seg_pos={0 0}, upload=436942]: Uploading piece... Dec 26 09:19:04.239293219 (497h50m4.110376264s) [noise=true , address=77.90.31.10:4517 , piece_id=FHPCHAP6CRLUWCALUURYURX2SHNQJ3HLLX56CQUKRF735LRGPCUA, seg_pos={0 0}, upload=436942]: Uploading piece... Dec 26 09:19:04.240119962 (497h50m4.111203007s) [noise=true , address=145.79.227.47:4266 , piece_id=MRJDDGI6OSFCWV3634MXCBFQZNFQVQZNOX6FATHK74LKOTD46VDA, seg_pos={0 0}, upload=436942]: Uploading piece... Dec 26 09:19:04.240381912 (497h50m4.111464957s) [noise=true , address=145.79.227.50:4374 , piece_id=TYC6BBUCVGBIWHFSRBCDOUMAFWYID6X5BVNA7GV6KGSELTK3YWXQ, seg_pos={0 0}, upload=436942]: Uploading piece... Dec 26 09:19:04.240602592 (497h50m4.111685637s) [noise=true , address=145.79.227.36:4519 , piece_id=3FNPMNWFEY75C7VZLEM4OXH2DPRVYOMRNPAYFR4QMARAIAPTZQOA, seg_pos={0 0}, upload=436942]: Uploading piece... Dec 26 09:19:04.241215990 (497h50m4.112299035s) [noise=true , address=145.79.227.41:4087 , piece_id=KCSPTT4MUK5CFFCZ7UAYNXHDBVGKZDSGA6FO5UZS64KPHCVO5DMQ, seg_pos={0 0}, upload=436942]: Uploading piece... Dec 26 09:19:04.241697116 (497h50m4.112780161s) [noise=true , address=45.91.116.13:4153 , piece_id=DUGBEADU7RELSD2QKVMOXWUGIUFUQGWQ5MKH76JKXWQ4EGNVSUIA, seg_pos={0 0}, upload=436942]: Uploading piece... Dec 26 09:19:04.243068147 (497h50m4.114151202s) [noise=true , address=45.91.116.9:4264 , piece_id=5CKTAS4I5X37CR4YBGHV752KVMCMQU6VCEAW2RLIOOZMQBTMLJVQ, seg_pos={0 0}, upload=436942]: Uploading piece... Dec 26 09:19:04.243487325 (497h50m4.11457037s) [noise=true , address=145.79.227.6:4257 , piece_id=X46L2UQ7PYRLU4UHWNHFFGN3UARMT2WLCGR46LNQH7RT5T52L5OA, seg_pos={0 0}, upload=436942]: Uploading piece... Dec 26 09:19:04.243897556 (497h50m4.114980601s) [noise=true , address=45.91.116.7:4009 , piece_id=GSYOOLXHZAB7BPNNUPNKLHFNUQ3VMYXM4KTUGR37RECC5W2MG54A, seg_pos={0 0}, upload=436942]: Uploading piece... Dec 26 09:19:04.244444237 (497h50m4.115527282s) [noise=true , address=45.91.116.10:4082 , piece_id=DYQBSVCT2KBJNFVT2UAUO2VYCVFGLLKN5CSZTPOZE2DSQ2M2JDGQ, seg_pos={0 0}, upload=436942]: Uploading piece... Dec 26 09:19:04.244895226 (497h50m4.115978271s) [noise=true , address=145.79.227.4:4185 , piece_id=MBZKG6NOJY62L4AE2MKJMHS3LL5XTFYNW5THKIETP4CPZBH567EQ, seg_pos={0 0}, upload=436942]: Uploading piece... Dec 26 09:19:04.245348289 (497h50m4.116431334s) [noise=true , address=77.90.31.21:4880 , piece_id=KZHOYEHYOIXUALTMTFSFQSFBRF4UGXA7FTCCOSMCDTJHI5LJ3BOA, seg_pos={0 0}, upload=436942]: Uploading piece... Dec 26 09:19:04.245950225 (497h50m4.11703327s) [noise=true , address=45.91.116.4:4940 , piece_id=DRHHHHRUTBMG5R2IZ2WDKVWZAAOJ453LZI5X3TX3OVADQ3S5LZXA, seg_pos={0 0}, upload=436942]: Uploading piece... Dec 26 09:19:04.246798560 (497h50m4.117881625s) [noise=true , address=145.79.227.2:4113 , piece_id=ABIPRHOAZWJITESUC5A6FGQ2K4LB34IOB4A7DFOS4JDVH4W5RG4A, seg_pos={0 0}, upload=436942]: Uploading piece... Dec 26 09:19:04.247536925 (497h50m4.118619971s) [noise=true , address=145.79.227.5:4223 , piece_id=TNS44GAPMLSCRMZNTCQ2ORSGXGGV3VTSJBX5MIJNLNHR2P7TSRRQ, seg_pos={0 0}, upload=436942]: Uploading piece... Dec 26 09:19:05.960755430 (497h50m5.831838475s) [upload=436942]: Next returned nil segment Dec 26 09:19:05.960770650 (497h50m5.831853695s) [upload=436942]: Waiting for error group managing segments... Dec 26 09:19:06.240183319 (497h50m6.111266364s) [noise=true , address=145.79.227.36:4519 , piece_id=3FNPMNWFEY75C7VZLEM4OXH2DPRVYOMRNPAYFR4QMARAIAPTZQOA, seg_pos={0 0}, upload=436942]: Done uploading piece. err: <nil> Dec 26 09:19:06.247000049 (497h50m6.118083084s) [noise=true , address=145.79.227.50:4374 , piece_id=TYC6BBUCVGBIWHFSRBCDOUMAFWYID6X5BVNA7GV6KGSELTK3YWXQ, seg_pos={0 0}, upload=436942]: Done uploading piece. err: <nil> Dec 26 09:19:06.252041269 (497h50m6.123124304s) [noise=true , address=145.79.227.41:4087 , piece_id=KCSPTT4MUK5CFFCZ7UAYNXHDBVGKZDSGA6FO5UZS64KPHCVO5DMQ, seg_pos={0 0}, upload=436942]: Done uploading piece. err: <nil> Dec 26 09:19:06.256510519 (497h50m6.127593554s) [noise=true , address=145.79.227.5:4223 , piece_id=TNS44GAPMLSCRMZNTCQ2ORSGXGGV3VTSJBX5MIJNLNHR2P7TSRRQ, seg_pos={0 0}, upload=436942]: Done uploading piece. err: <nil> Dec 26 09:19:06.262479706 (497h50m6.133562741s) [noise=true , address=145.79.227.2:4113 , piece_id=ABIPRHOAZWJITESUC5A6FGQ2K4LB34IOB4A7DFOS4JDVH4W5RG4A, seg_pos={0 0}, upload=436942]: Done uploading piece. err: <nil> Dec 26 09:19:06.267605536 (497h50m6.138688581s) [noise=true , address=145.79.227.16:4910 , piece_id=MM5KOERT2WCFTRALXGZB4HJ5X2COBCLWKGHRVB4HRJJNIAQVI5MQ, seg_pos={0 0}, upload=436942]: Done uploading piece. err: <nil> Dec 26 09:19:06.268038039 (497h50m6.139121084s) [noise=true , address=145.79.227.47:4266 , piece_id=MRJDDGI6OSFCWV3634MXCBFQZNFQVQZNOX6FATHK74LKOTD46VDA, seg_pos={0 0}, upload=436942]: Done uploading piece. err: <nil> Dec 26 09:19:06.270776654 (497h50m6.141859689s) [noise=true , address=145.79.227.4:4185 , piece_id=MBZKG6NOJY62L4AE2MKJMHS3LL5XTFYNW5THKIETP4CPZBH567EQ, seg_pos={0 0}, upload=436942]: Done uploading piece. err: <nil> Dec 26 09:19:06.313364099 (497h50m6.184447134s) [noise=true , address=145.79.227.6:4257 , piece_id=X46L2UQ7PYRLU4UHWNHFFGN3UARMT2WLCGR46LNQH7RT5T52L5OA, seg_pos={0 0}, upload=436942]: Done uploading piece. err: <nil> Dec 26 09:19:06.474769782 (497h50m6.345852827s) [noise=true , address=77.90.31.21:4880 , piece_id=KZHOYEHYOIXUALTMTFSFQSFBRF4UGXA7FTCCOSMCDTJHI5LJ3BOA, seg_pos={0 0}, upload=436942]: Done uploading piece. err: <nil> Dec 26 09:19:06.546365233 (497h50m6.417448278s) [noise=true , address=77.90.31.10:4517 , piece_id=FHPCHAP6CRLUWCALUURYURX2SHNQJ3HLLX56CQUKRF735LRGPCUA, seg_pos={0 0}, upload=436942]: Done uploading piece. err: <nil> Dec 26 09:19:06.568853073 (497h50m6.439936339s) [noise=true , address=45.91.116.4:4940 , piece_id=DRHHHHRUTBMG5R2IZ2WDKVWZAAOJ453LZI5X3TX3OVADQ3S5LZXA, seg_pos={0 0}, upload=436942]: Done uploading piece. err: <nil> Dec 26 09:19:06.628884955 (497h50m6.499968s) [noise=true , address=45.91.116.9:4264 , piece_id=5CKTAS4I5X37CR4YBGHV752KVMCMQU6VCEAW2RLIOOZMQBTMLJVQ, seg_pos={0 0}, upload=436942]: Done uploading piece. err: <nil> Dec 26 09:19:06.628906104 (497h50m6.49998915s) [seg_pos={0 0}, upload=436942]: Segment reached optimal threshold of 13 pieces. Dec 26 09:19:06.629147033 (497h50m6.500230068s) [noise=true , address=45.91.116.7:4009 , piece_id=GSYOOLXHZAB7BPNNUPNKLHFNUQ3VMYXM4KTUGR37RECC5W2MG54A, seg_pos={0 0}, upload=436942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 09:19:06.629157252 (497h50m6.500240288s) [noise=true , address=45.91.116.7:4009 , piece_id=GSYOOLXHZAB7BPNNUPNKLHFNUQ3VMYXM4KTUGR37RECC5W2MG54A, seg_pos={0 0}, upload=436942]: Long tail cancellation due to optimal threshold Dec 26 09:19:06.629192910 (497h50m6.500275955s) [noise=true , address=45.91.116.13:4153 , piece_id=DUGBEADU7RELSD2QKVMOXWUGIUFUQGWQ5MKH76JKXWQ4EGNVSUIA, seg_pos={0 0}, upload=436942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:60242->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:60242->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60242->45.91.116.13:4153: use of closed network connection Dec 26 09:19:06.629201437 (497h50m6.500284482s) [noise=true , address=45.91.116.13:4153 , piece_id=DUGBEADU7RELSD2QKVMOXWUGIUFUQGWQ5MKH76JKXWQ4EGNVSUIA, seg_pos={0 0}, upload=436942]: Long tail cancellation due to optimal threshold Dec 26 09:19:06.629360750 (497h50m6.500443785s) [noise=true , address=45.91.116.10:4082 , piece_id=DYQBSVCT2KBJNFVT2UAUO2VYCVFGLLKN5CSZTPOZE2DSQ2M2JDGQ, seg_pos={0 0}, upload=436942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 26 09:19:06.629367192 (497h50m6.500450227s) [noise=true , address=45.91.116.10:4082 , piece_id=DYQBSVCT2KBJNFVT2UAUO2VYCVFGLLKN5CSZTPOZE2DSQ2M2JDGQ, seg_pos={0 0}, upload=436942]: Long tail cancellation due to optimal threshold Dec 26 09:19:06.629384716 (497h50m6.500467761s) [seg_pos={0 0}, upload=436942]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:19:06.629451924 (497h50m6.500534959s) [upload=436942]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:19:06.629456523 (497h50m6.500539558s) [upload=436942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:19:06.920143612 (497h50m6.791226657s) [upload=436942]: Done uploading segments... Dec 26 09:19:06.920167116 (497h50m6.791250161s) [upload=436942]: Upload finished. err: <nil> Duration: 30.002785798s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 41.409853087s trace: https://jaeger.dev.storj.io/trace/6820217d73ecf2a FAIL (2025-12-26 09:19:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments... Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces. Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments... Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil> Duration: 41.409884515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 33.569328413s trace: https://jaeger.dev.storj.io/trace/6820217d73ecf2a FAIL (2025-12-26 09:18:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments... Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces. Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments... Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil> Duration: 33.569344956s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X.XX..X.XXX....................X....XXX..X.......X.....XX.] duration: 5.688882585s trace: https://jaeger.dev.storj.io/trace/3d7dc99d15085113 FAIL (2025-12-26 09:17:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: YKPNEL4RN2DT5RN3DDU6Y5JVCEE7ZMUBJALVCF4R4LN4RVIDLHIA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: II4ZEBALB5Z74RXZERX4IWOBZRKLXAXXBAQKK52XF3HAHNMQGIRQ) context deadline exceeded ======LOGS====== Dec 26 09:16:48.692126167 (497h47m48.563209202s) [upload=436907]: Starting upload Dec 26 09:16:48.692247958 (497h47m48.563331023s) [upload=436907]: Done starting upload Dec 26 09:16:48.692402222 (497h47m48.563485257s) [upload=436907]: Uploading segments... Dec 26 09:16:48.692471835 (497h47m48.56355488s) [upload=436907]: Got next segment. Inline: false Dec 26 09:16:48.692513614 (497h47m48.563596659s) [upload=436907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:16:48.692519916 (497h47m48.563602961s) [upload=436907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:16:48.937565426 (497h47m48.808648461s) [seg_pos={0 0}, upload=436907]: Begin upload segment... Dec 26 09:16:48.937619439 (497h47m48.808702484s) [seg_pos={0 0}, upload=436907]: Done begin upload segment. Dec 26 09:16:48.937685834 (497h47m48.808768869s) [noise=true , address=145.79.227.6:4257 , piece_id=ZOO4GULHRI2XQJN6R56TVZKV2V3C5XNPJ2OM5YIJTZIA5ROR7TSQ, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.937706554 (497h47m48.808789599s) [noise=true , address=77.90.31.17:4770 , piece_id=YISY4QNUTF74THD5ODHOOYK3GQFLOEHHIYC5TRVE5GQWCMFERS6Q, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.938316306 (497h47m48.809399341s) [noise=true , address=145.79.227.16:4909 , piece_id=3NLMVMATSDGB7QQZCSB5ZF2OQI2VNIL2TNHJMY4O63VY663QRKJA, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.938346643 (497h47m48.809429688s) [noise=true , address=145.79.227.15:4876 , piece_id=4LEXKT5NSFPIUTT3OMGEMNWG3LUJYVHRLVYIY6I6VJ6AHJ5AWT6Q, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.938653998 (497h47m48.809737043s) [noise=true , address=45.91.116.9:4261 , piece_id=II4ZEBALB5Z74RXZERX4IWOBZRKLXAXXBAQKK52XF3HAHNMQGIRQ, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.938737037 (497h47m48.809820072s) [noise=true , address=145.79.227.38:4978 , piece_id=76RWZ5QMOTTT6GUIDBDWUDJSFRF5657F7K6TJNBIZGBVSJXLKZOQ, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.938937708 (497h47m48.810020783s) [noise=true , address=45.91.116.5:4298 , piece_id=XEFJVYTAESCMJRKT5FQ7VYW5Y2TAS2UPNRTN2VMBANCRTCHO44IQ, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.939141817 (497h47m48.810224862s) [noise=true , address=45.91.116.7:4012 , piece_id=YFNLWNEAZBFP3Y6OJKE6PBYMBRFSVPWNRGEDGI7YISI2RXRQC3JA, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.939297294 (497h47m48.810380329s) [noise=true , address=145.79.227.19:4079 , piece_id=HLEM4S5BFBG3FAGFZXQS65AVQMXQU332OBJMNGZDZ5VPZV4XUAVQ, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.939493847 (497h47m48.810576892s) [noise=true , address=45.91.116.11:4120 , piece_id=XISFW5UUIOJASCHUGMHG3INS3VUGNKZ7A4CCHNAH4GBABRNSSFRA, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.939581324 (497h47m48.810664359s) [noise=true , address=145.79.227.48:4303 , piece_id=E5R23XR4EOGGQEPQQ6ZHANNHPS2QJVFOZ4SC4S37VMRRAE7Q67QA, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.939787818 (497h47m48.810870863s) [noise=true , address=145.79.227.4:4187 , piece_id=AFEINTEMGGGNHZA6FIILQ7JTKY5T32OGPF2P5YNMDMPRNCGB3PBA, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.939897085 (497h47m48.810980121s) [noise=true , address=145.79.227.52:4445 , piece_id=BS3JBDVIYJYNQV3QCLZAVNJSCEMXTTUOF7GSQIQ7HA3VM3TLBL5A, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.940078030 (497h47m48.811161065s) [noise=true , address=77.90.31.18:4805 , piece_id=F2V73DE3AWTLRUYDBUPWYMYRQZJUM3ZTEXPNOI7XKNUCE35H57XA, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.940169845 (497h47m48.81125289s) [noise=true , address=145.79.227.45:4231 , piece_id=MKMOEYKPBRPIY57IO6VXGGQXW5A7CWXTNTHFUADEZOUMEIAIOQ2Q, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.940455729 (497h47m48.811538774s) [noise=true , address=45.91.116.13:4155 , piece_id=YKPNEL4RN2DT5RN3DDU6Y5JVCEE7ZMUBJALVCF4R4LN4RVIDLHIA, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:50.147129464 (497h47m50.01821251s) [noise=true , address=77.90.31.18:4805 , piece_id=F2V73DE3AWTLRUYDBUPWYMYRQZJUM3ZTEXPNOI7XKNUCE35H57XA, seg_pos={0 0}, upload=436907]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 09:16:50.651129899 (497h47m50.522212944s) [upload=436907]: Next returned nil segment Dec 26 09:16:50.651144336 (497h47m50.522227381s) [upload=436907]: Waiting for error group managing segments... Dec 26 09:16:50.934752946 (497h47m50.805835981s) [noise=true , address=145.79.227.45:4231 , piece_id=MKMOEYKPBRPIY57IO6VXGGQXW5A7CWXTNTHFUADEZOUMEIAIOQ2Q, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:50.954237408 (497h47m50.825320453s) [noise=true , address=145.79.227.19:4079 , piece_id=HLEM4S5BFBG3FAGFZXQS65AVQMXQU332OBJMNGZDZ5VPZV4XUAVQ, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:50.959530206 (497h47m50.830613251s) [noise=true , address=145.79.227.6:4257 , piece_id=ZOO4GULHRI2XQJN6R56TVZKV2V3C5XNPJ2OM5YIJTZIA5ROR7TSQ, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:50.962496765 (497h47m50.83357981s) [noise=true , address=145.79.227.38:4978 , piece_id=76RWZ5QMOTTT6GUIDBDWUDJSFRF5657F7K6TJNBIZGBVSJXLKZOQ, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:50.963923321 (497h47m50.835006356s) [noise=true , address=145.79.227.48:4303 , piece_id=E5R23XR4EOGGQEPQQ6ZHANNHPS2QJVFOZ4SC4S37VMRRAE7Q67QA, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:50.965394502 (497h47m50.836477537s) [noise=true , address=145.79.227.15:4876 , piece_id=4LEXKT5NSFPIUTT3OMGEMNWG3LUJYVHRLVYIY6I6VJ6AHJ5AWT6Q, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:50.974992910 (497h47m50.846075955s) [noise=true , address=145.79.227.52:4445 , piece_id=BS3JBDVIYJYNQV3QCLZAVNJSCEMXTTUOF7GSQIQ7HA3VM3TLBL5A, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:50.975248506 (497h47m50.846331551s) [noise=true , address=145.79.227.16:4909 , piece_id=3NLMVMATSDGB7QQZCSB5ZF2OQI2VNIL2TNHJMY4O63VY663QRKJA, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:50.977325321 (497h47m50.848408356s) [noise=true , address=145.79.227.4:4187 , piece_id=AFEINTEMGGGNHZA6FIILQ7JTKY5T32OGPF2P5YNMDMPRNCGB3PBA, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:51.175394449 (497h47m51.046477484s) [noise=true , address=45.91.116.13:4155 , piece_id=YKPNEL4RN2DT5RN3DDU6Y5JVCEE7ZMUBJALVCF4R4LN4RVIDLHIA, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:51.278121048 (497h47m51.149204093s) [noise=true , address=45.91.116.5:4298 , piece_id=XEFJVYTAESCMJRKT5FQ7VYW5Y2TAS2UPNRTN2VMBANCRTCHO44IQ, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:51.316414766 (497h47m51.187497811s) [noise=true , address=77.90.31.17:4770 , piece_id=YISY4QNUTF74THD5ODHOOYK3GQFLOEHHIYC5TRVE5GQWCMFERS6Q, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:51.330398964 (497h47m51.201482009s) [noise=true , address=45.91.116.9:4261 , piece_id=II4ZEBALB5Z74RXZERX4IWOBZRKLXAXXBAQKK52XF3HAHNMQGIRQ, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:51.330423550 (497h47m51.201506586s) [seg_pos={0 0}, upload=436907]: Segment reached optimal threshold of 13 pieces. Dec 26 09:16:51.330659640 (497h47m51.201742675s) [noise=true , address=45.91.116.11:4120 , piece_id=XISFW5UUIOJASCHUGMHG3INS3VUGNKZ7A4CCHNAH4GBABRNSSFRA, seg_pos={0 0}, upload=436907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 09:16:51.330669840 (497h47m51.201752885s) [noise=true , address=45.91.116.11:4120 , piece_id=XISFW5UUIOJASCHUGMHG3INS3VUGNKZ7A4CCHNAH4GBABRNSSFRA, seg_pos={0 0}, upload=436907]: Long tail cancellation due to optimal threshold Dec 26 09:16:51.330959602 (497h47m51.202042647s) [noise=true , address=45.91.116.7:4012 , piece_id=YFNLWNEAZBFP3Y6OJKE6PBYMBRFSVPWNRGEDGI7YISI2RXRQC3JA, seg_pos={0 0}, upload=436907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 26 09:16:51.330966635 (497h47m51.20204967s) [noise=true , address=45.91.116.7:4012 , piece_id=YFNLWNEAZBFP3Y6OJKE6PBYMBRFSVPWNRGEDGI7YISI2RXRQC3JA, seg_pos={0 0}, upload=436907]: Long tail cancellation due to optimal threshold Dec 26 09:16:51.330984109 (497h47m51.202067144s) [seg_pos={0 0}, upload=436907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:16:51.331052289 (497h47m51.202135324s) [upload=436907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:16:51.331057148 (497h47m51.202140183s) [upload=436907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:16:51.625659311 (497h47m51.496742356s) [upload=436907]: Done uploading segments... Dec 26 09:16:51.625683927 (497h47m51.496766972s) [upload=436907]: Upload finished. err: <nil> Duration: 5.68891112s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 42.992740468s trace: https://jaeger.dev.storj.io/trace/6820217d73ecf2a FAIL (2025-12-26 09:17:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments... Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces. Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments... Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil> Duration: 42.992761368s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X.XX..X.XXX....................X....XXX..X.......X.....XX] duration: 30.003878239s trace: https://jaeger.dev.storj.io/trace/3d7dc99d15085113 FAIL (2025-12-26 09:17:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: YKPNEL4RN2DT5RN3DDU6Y5JVCEE7ZMUBJALVCF4R4LN4RVIDLHIA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: II4ZEBALB5Z74RXZERX4IWOBZRKLXAXXBAQKK52XF3HAHNMQGIRQ) context deadline exceeded ======LOGS====== Dec 26 09:16:48.692126167 (497h47m48.563209202s) [upload=436907]: Starting upload Dec 26 09:16:48.692247958 (497h47m48.563331023s) [upload=436907]: Done starting upload Dec 26 09:16:48.692402222 (497h47m48.563485257s) [upload=436907]: Uploading segments... Dec 26 09:16:48.692471835 (497h47m48.56355488s) [upload=436907]: Got next segment. Inline: false Dec 26 09:16:48.692513614 (497h47m48.563596659s) [upload=436907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:16:48.692519916 (497h47m48.563602961s) [upload=436907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:16:48.937565426 (497h47m48.808648461s) [seg_pos={0 0}, upload=436907]: Begin upload segment... Dec 26 09:16:48.937619439 (497h47m48.808702484s) [seg_pos={0 0}, upload=436907]: Done begin upload segment. Dec 26 09:16:48.937685834 (497h47m48.808768869s) [noise=true , address=145.79.227.6:4257 , piece_id=ZOO4GULHRI2XQJN6R56TVZKV2V3C5XNPJ2OM5YIJTZIA5ROR7TSQ, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.937706554 (497h47m48.808789599s) [noise=true , address=77.90.31.17:4770 , piece_id=YISY4QNUTF74THD5ODHOOYK3GQFLOEHHIYC5TRVE5GQWCMFERS6Q, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.938316306 (497h47m48.809399341s) [noise=true , address=145.79.227.16:4909 , piece_id=3NLMVMATSDGB7QQZCSB5ZF2OQI2VNIL2TNHJMY4O63VY663QRKJA, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.938346643 (497h47m48.809429688s) [noise=true , address=145.79.227.15:4876 , piece_id=4LEXKT5NSFPIUTT3OMGEMNWG3LUJYVHRLVYIY6I6VJ6AHJ5AWT6Q, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.938653998 (497h47m48.809737043s) [noise=true , address=45.91.116.9:4261 , piece_id=II4ZEBALB5Z74RXZERX4IWOBZRKLXAXXBAQKK52XF3HAHNMQGIRQ, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.938737037 (497h47m48.809820072s) [noise=true , address=145.79.227.38:4978 , piece_id=76RWZ5QMOTTT6GUIDBDWUDJSFRF5657F7K6TJNBIZGBVSJXLKZOQ, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.938937708 (497h47m48.810020783s) [noise=true , address=45.91.116.5:4298 , piece_id=XEFJVYTAESCMJRKT5FQ7VYW5Y2TAS2UPNRTN2VMBANCRTCHO44IQ, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.939141817 (497h47m48.810224862s) [noise=true , address=45.91.116.7:4012 , piece_id=YFNLWNEAZBFP3Y6OJKE6PBYMBRFSVPWNRGEDGI7YISI2RXRQC3JA, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.939297294 (497h47m48.810380329s) [noise=true , address=145.79.227.19:4079 , piece_id=HLEM4S5BFBG3FAGFZXQS65AVQMXQU332OBJMNGZDZ5VPZV4XUAVQ, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.939493847 (497h47m48.810576892s) [noise=true , address=45.91.116.11:4120 , piece_id=XISFW5UUIOJASCHUGMHG3INS3VUGNKZ7A4CCHNAH4GBABRNSSFRA, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.939581324 (497h47m48.810664359s) [noise=true , address=145.79.227.48:4303 , piece_id=E5R23XR4EOGGQEPQQ6ZHANNHPS2QJVFOZ4SC4S37VMRRAE7Q67QA, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.939787818 (497h47m48.810870863s) [noise=true , address=145.79.227.4:4187 , piece_id=AFEINTEMGGGNHZA6FIILQ7JTKY5T32OGPF2P5YNMDMPRNCGB3PBA, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.939897085 (497h47m48.810980121s) [noise=true , address=145.79.227.52:4445 , piece_id=BS3JBDVIYJYNQV3QCLZAVNJSCEMXTTUOF7GSQIQ7HA3VM3TLBL5A, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.940078030 (497h47m48.811161065s) [noise=true , address=77.90.31.18:4805 , piece_id=F2V73DE3AWTLRUYDBUPWYMYRQZJUM3ZTEXPNOI7XKNUCE35H57XA, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.940169845 (497h47m48.81125289s) [noise=true , address=145.79.227.45:4231 , piece_id=MKMOEYKPBRPIY57IO6VXGGQXW5A7CWXTNTHFUADEZOUMEIAIOQ2Q, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:48.940455729 (497h47m48.811538774s) [noise=true , address=45.91.116.13:4155 , piece_id=YKPNEL4RN2DT5RN3DDU6Y5JVCEE7ZMUBJALVCF4R4LN4RVIDLHIA, seg_pos={0 0}, upload=436907]: Uploading piece... Dec 26 09:16:50.147129464 (497h47m50.01821251s) [noise=true , address=77.90.31.18:4805 , piece_id=F2V73DE3AWTLRUYDBUPWYMYRQZJUM3ZTEXPNOI7XKNUCE35H57XA, seg_pos={0 0}, upload=436907]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 09:16:50.651129899 (497h47m50.522212944s) [upload=436907]: Next returned nil segment Dec 26 09:16:50.651144336 (497h47m50.522227381s) [upload=436907]: Waiting for error group managing segments... Dec 26 09:16:50.934752946 (497h47m50.805835981s) [noise=true , address=145.79.227.45:4231 , piece_id=MKMOEYKPBRPIY57IO6VXGGQXW5A7CWXTNTHFUADEZOUMEIAIOQ2Q, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:50.954237408 (497h47m50.825320453s) [noise=true , address=145.79.227.19:4079 , piece_id=HLEM4S5BFBG3FAGFZXQS65AVQMXQU332OBJMNGZDZ5VPZV4XUAVQ, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:50.959530206 (497h47m50.830613251s) [noise=true , address=145.79.227.6:4257 , piece_id=ZOO4GULHRI2XQJN6R56TVZKV2V3C5XNPJ2OM5YIJTZIA5ROR7TSQ, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:50.962496765 (497h47m50.83357981s) [noise=true , address=145.79.227.38:4978 , piece_id=76RWZ5QMOTTT6GUIDBDWUDJSFRF5657F7K6TJNBIZGBVSJXLKZOQ, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:50.963923321 (497h47m50.835006356s) [noise=true , address=145.79.227.48:4303 , piece_id=E5R23XR4EOGGQEPQQ6ZHANNHPS2QJVFOZ4SC4S37VMRRAE7Q67QA, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:50.965394502 (497h47m50.836477537s) [noise=true , address=145.79.227.15:4876 , piece_id=4LEXKT5NSFPIUTT3OMGEMNWG3LUJYVHRLVYIY6I6VJ6AHJ5AWT6Q, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:50.974992910 (497h47m50.846075955s) [noise=true , address=145.79.227.52:4445 , piece_id=BS3JBDVIYJYNQV3QCLZAVNJSCEMXTTUOF7GSQIQ7HA3VM3TLBL5A, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:50.975248506 (497h47m50.846331551s) [noise=true , address=145.79.227.16:4909 , piece_id=3NLMVMATSDGB7QQZCSB5ZF2OQI2VNIL2TNHJMY4O63VY663QRKJA, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:50.977325321 (497h47m50.848408356s) [noise=true , address=145.79.227.4:4187 , piece_id=AFEINTEMGGGNHZA6FIILQ7JTKY5T32OGPF2P5YNMDMPRNCGB3PBA, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:51.175394449 (497h47m51.046477484s) [noise=true , address=45.91.116.13:4155 , piece_id=YKPNEL4RN2DT5RN3DDU6Y5JVCEE7ZMUBJALVCF4R4LN4RVIDLHIA, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:51.278121048 (497h47m51.149204093s) [noise=true , address=45.91.116.5:4298 , piece_id=XEFJVYTAESCMJRKT5FQ7VYW5Y2TAS2UPNRTN2VMBANCRTCHO44IQ, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:51.316414766 (497h47m51.187497811s) [noise=true , address=77.90.31.17:4770 , piece_id=YISY4QNUTF74THD5ODHOOYK3GQFLOEHHIYC5TRVE5GQWCMFERS6Q, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:51.330398964 (497h47m51.201482009s) [noise=true , address=45.91.116.9:4261 , piece_id=II4ZEBALB5Z74RXZERX4IWOBZRKLXAXXBAQKK52XF3HAHNMQGIRQ, seg_pos={0 0}, upload=436907]: Done uploading piece. err: <nil> Dec 26 09:16:51.330423550 (497h47m51.201506586s) [seg_pos={0 0}, upload=436907]: Segment reached optimal threshold of 13 pieces. Dec 26 09:16:51.330659640 (497h47m51.201742675s) [noise=true , address=45.91.116.11:4120 , piece_id=XISFW5UUIOJASCHUGMHG3INS3VUGNKZ7A4CCHNAH4GBABRNSSFRA, seg_pos={0 0}, upload=436907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 09:16:51.330669840 (497h47m51.201752885s) [noise=true , address=45.91.116.11:4120 , piece_id=XISFW5UUIOJASCHUGMHG3INS3VUGNKZ7A4CCHNAH4GBABRNSSFRA, seg_pos={0 0}, upload=436907]: Long tail cancellation due to optimal threshold Dec 26 09:16:51.330959602 (497h47m51.202042647s) [noise=true , address=45.91.116.7:4012 , piece_id=YFNLWNEAZBFP3Y6OJKE6PBYMBRFSVPWNRGEDGI7YISI2RXRQC3JA, seg_pos={0 0}, upload=436907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 26 09:16:51.330966635 (497h47m51.20204967s) [noise=true , address=45.91.116.7:4012 , piece_id=YFNLWNEAZBFP3Y6OJKE6PBYMBRFSVPWNRGEDGI7YISI2RXRQC3JA, seg_pos={0 0}, upload=436907]: Long tail cancellation due to optimal threshold Dec 26 09:16:51.330984109 (497h47m51.202067144s) [seg_pos={0 0}, upload=436907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:16:51.331052289 (497h47m51.202135324s) [upload=436907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:16:51.331057148 (497h47m51.202140183s) [upload=436907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:16:51.625659311 (497h47m51.496742356s) [upload=436907]: Done uploading segments... Dec 26 09:16:51.625683927 (497h47m51.496766972s) [upload=436907]: Upload finished. err: <nil> Duration: 30.003905571s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 39.306753259s trace: https://jaeger.dev.storj.io/trace/6820217d73ecf2a FAIL (2025-12-26 09:17:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments... Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces. Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments... Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil> Duration: 39.306768337s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 59.851359697s trace: https://jaeger.dev.storj.io/trace/6820217d73ecf2a FAIL (2025-12-26 09:16:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments... Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces. Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments... Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil> Duration: 59.851376729s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X...X.XX..X.XXX....................X....XXX..X.......X.....X] duration: 30.003996189s trace: https://jaeger.dev.storj.io/trace/490afb1f3757f78f FAIL (2025-12-26 09:16:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 4AYHTEURFUHGCNOYIJUD45NOUUGJKAJ3GFNYR7KWANYIPTTKTUGQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: JJ4K6IX476KTYCI3QE3YEYMRHQFN3M7FHFKNH4RKXJYBE5BUVN3A) context deadline exceeded ======LOGS====== 09:15:42.504827418 (497h46m42.375910453s) [upload=436895]: Done starting upload Dec 26 09:15:42.505084327 (497h46m42.376167362s) [upload=436895]: Uploading segments... Dec 26 09:15:42.505123993 (497h46m42.376207028s) [upload=436895]: Got next segment. Inline: false Dec 26 09:15:42.505148670 (497h46m42.376231705s) [upload=436895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:15:42.505152898 (497h46m42.376235933s) [upload=436895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:15:42.752295340 (497h46m42.623378375s) [seg_pos={0 0}, upload=436895]: Begin upload segment... Dec 26 09:15:42.752341538 (497h46m42.623424573s) [seg_pos={0 0}, upload=436895]: Done begin upload segment. Dec 26 09:15:42.752370674 (497h46m42.623453709s) [noise=true , address=145.79.227.5:4223 , piece_id=XNW4RWSWYF5HLE6RMJ3MSW57QPTKT64XLXFGIQNXZBPL4X6N4VOA, seg_pos={0 0}, upload=436895]: Uploading piece... Dec 26 09:15:42.752386173 (497h46m42.623469208s) [noise=true , address=145.79.227.47:4266 , piece_id=BDVLZ7P6LRBAL2AX4MFISLEKO5LUQU3BYV7OWHUXFSRLGVUAS6TA, seg_pos={0 0}, upload=436895]: Uploading piece... Dec 26 09:15:42.752900172 (497h46m42.623983207s) [noise=true , address=77.90.31.13:4629 , piece_id=55HQASOGPSSSG63PHZXE2NCDSMDGGXCR4TA63NJGKEKBJR2YZCOA, seg_pos={0 0}, upload=436895]: Uploading piece... Dec 26 09:15:42.753209881 (497h46m42.624292927s) [noise=true , address=145.79.227.46:4603 , piece_id=IWBGF4B3EGGNYFI5OMYYZ7O34DEZRMHB2W3VRU2Y5G2IIM2G3LAA, seg_pos={0 0}, upload=436895]: Uploading piece... Dec 26 09:15:42.753717679 (497h46m42.624800714s) [noise=true , address=145.79.227.19:4077 , piece_id=V5SEZCMZCBTRFXCODFECHNZUUFEHHHKVBINJFWRO55AIOUUDIUOA, seg_pos={0 0}, upload=436895]: Uploading piece... Dec 26 09:15:42.753971933 (497h46m42.625054968s) [noise=true , address=45.91.116.11:4118 , piece_id=4AYHTEURFUHGCNOYIJUD45NOUUGJKAJ3GFNYR7KWANYIPTTKTUGQ, seg_pos={0 0}, upload=436895]: Uploading piece... Dec 26 09:15:42.754103283 (497h46m42.625186378s) [noise=true , address=145.79.227.36:4517 , piece_id=I7BRRAPDUNOAQGYDUPDU6O52YCCRMSF5SOEFSWFXX4LXJFZSLSLQ, seg_pos={0 0}, upload=436895]: Uploading piece... Dec 26 09:15:42.754513585 (497h46m42.62559662s) [noise=true , address=145.79.227.16:4909 , piece_id=PQNT6DWQTUKWLVOFEL2GQMEVRNK6KIQIQYEB2WSNYXNJHHCNLSPQ, seg_pos={0 0}, upload=436895]: Uploading piece... Dec 26 09:15:42.754817312 (497h46m42.625900357s) [noise=true , address=145.79.227.52:4446 , piece_id=J4FELDZEGN5LNRXSHKZI2WGX4M6C5IOLTHFSEUCCPJYGAIB4ZURA, seg_pos={0 0}, upload=436895]: Uploading piece... Dec 26 09:15:42.754850345 (497h46m42.62593338s) [noise=true , address=45.91.116.4:4940 , piece_id=ERGUY7KSQ7X7TYI2CKV3IMYIPHT2UA6VFSZXUVRWLQ4NYFGTOR3Q, seg_pos={0 0}, upload=436895]: Uploading piece... Dec 26 09:15:42.755093107 (497h46m42.626176142s) [noise=true , address=145.79.227.29:4941 , piece_id=SEKAX7IQCPMODN6QYIZPOLQY4JA6AXLYK3RNAX5I6QTM6OD77LRQ, seg_pos={0 0}, upload=436895]: Uploading piece... Dec 26 09:15:42.755732115 (497h46m42.62681515s) [noise=true , address=45.91.116.8:4047 , piece_id=CN65EEATO7FYSRTCONA4OOLHCGDHXGBIQH77NBSVC45JCVE3JW2Q, seg_pos={0 0}, upload=436895]: Uploading piece... Dec 26 09:15:42.756118951 (497h46m42.627201986s) [noise=true , address=45.91.116.7:4010 , piece_id=JJ4K6IX476KTYCI3QE3YEYMRHQFN3M7FHFKNH4RKXJYBE5BUVN3A, seg_pos={0 0}, upload=436895]: Uploading piece... Dec 26 09:15:42.756508783 (497h46m42.627591818s) [noise=true , address=77.90.31.18:4805 , piece_id=R4QOQLTEMSRBOWVPA6F3BYPY5AOWVDAV3SSN7MSQ5J62GC2G33DQ, seg_pos={0 0}, upload=436895]: Uploading piece... Dec 26 09:15:42.756873277 (497h46m42.627956322s) [noise=true , address=145.79.227.13:4802 , piece_id=DPHTJPPAXUAZMFRYWSYQDLIDAEZD4HSPEQNYMNIHXNOWFU6DZ5MQ, seg_pos={0 0}, upload=436895]: Uploading piece... Dec 26 09:15:42.757193237 (497h46m42.628276272s) [noise=true , address=77.90.31.10:4520 , piece_id=FX4UDAPCIXJE7F5L6NQKFUHANKFUWYFPHO36JAQ4OTUOHCD5HU7Q, seg_pos={0 0}, upload=436895]: Uploading piece... Dec 26 09:15:44.472320826 (497h46m44.343403871s) [upload=436895]: Next returned nil segment Dec 26 09:15:44.472332058 (497h46m44.343415093s) [upload=436895]: Waiting for error group managing segments... Dec 26 09:15:44.749409391 (497h46m44.620492446s) [noise=true , address=145.79.227.29:4941 , piece_id=SEKAX7IQCPMODN6QYIZPOLQY4JA6AXLYK3RNAX5I6QTM6OD77LRQ, seg_pos={0 0}, upload=436895]: Done uploading piece. err: <nil> Dec 26 09:15:44.763028394 (497h46m44.634111429s) [noise=true , address=145.79.227.13:4802 , piece_id=DPHTJPPAXUAZMFRYWSYQDLIDAEZD4HSPEQNYMNIHXNOWFU6DZ5MQ, seg_pos={0 0}, upload=436895]: Done uploading piece. err: <nil> Dec 26 09:15:44.763684803 (497h46m44.634767838s) [noise=true , address=145.79.227.36:4517 , piece_id=I7BRRAPDUNOAQGYDUPDU6O52YCCRMSF5SOEFSWFXX4LXJFZSLSLQ, seg_pos={0 0}, upload=436895]: Done uploading piece. err: <nil> Dec 26 09:15:44.763782630 (497h46m44.634865675s) [noise=true , address=145.79.227.19:4077 , piece_id=V5SEZCMZCBTRFXCODFECHNZUUFEHHHKVBINJFWRO55AIOUUDIUOA, seg_pos={0 0}, upload=436895]: Done uploading piece. err: <nil> Dec 26 09:15:44.764246193 (497h46m44.635329228s) [noise=true , address=145.79.227.16:4909 , piece_id=PQNT6DWQTUKWLVOFEL2GQMEVRNK6KIQIQYEB2WSNYXNJHHCNLSPQ, seg_pos={0 0}, upload=436895]: Done uploading piece. err: <nil> Dec 26 09:15:44.766033697 (497h46m44.637116742s) [noise=true , address=145.79.227.46:4603 , piece_id=IWBGF4B3EGGNYFI5OMYYZ7O34DEZRMHB2W3VRU2Y5G2IIM2G3LAA, seg_pos={0 0}, upload=436895]: Done uploading piece. err: <nil> Dec 26 09:15:44.773030840 (497h46m44.644113885s) [noise=true , address=145.79.227.47:4266 , piece_id=BDVLZ7P6LRBAL2AX4MFISLEKO5LUQU3BYV7OWHUXFSRLGVUAS6TA, seg_pos={0 0}, upload=436895]: Done uploading piece. err: <nil> Dec 26 09:15:44.782564784 (497h46m44.653647829s) [noise=true , address=145.79.227.52:4446 , piece_id=J4FELDZEGN5LNRXSHKZI2WGX4M6C5IOLTHFSEUCCPJYGAIB4ZURA, seg_pos={0 0}, upload=436895]: Done uploading piece. err: <nil> Dec 26 09:15:44.792333065 (497h46m44.66341611s) [noise=true , address=145.79.227.5:4223 , piece_id=XNW4RWSWYF5HLE6RMJ3MSW57QPTKT64XLXFGIQNXZBPL4X6N4VOA, seg_pos={0 0}, upload=436895]: Done uploading piece. err: <nil> Dec 26 09:15:45.017657932 (497h46m44.888740967s) [noise=true , address=77.90.31.13:4629 , piece_id=55HQASOGPSSSG63PHZXE2NCDSMDGGXCR4TA63NJGKEKBJR2YZCOA, seg_pos={0 0}, upload=436895]: Done uploading piece. err: <nil> Dec 26 09:15:45.107388996 (497h46m44.978472051s) [noise=true , address=45.91.116.8:4047 , piece_id=CN65EEATO7FYSRTCONA4OOLHCGDHXGBIQH77NBSVC45JCVE3JW2Q, seg_pos={0 0}, upload=436895]: Done uploading piece. err: <nil> Dec 26 09:15:45.156663222 (497h46m45.027746267s) [noise=true , address=45.91.116.11:4118 , piece_id=4AYHTEURFUHGCNOYIJUD45NOUUGJKAJ3GFNYR7KWANYIPTTKTUGQ, seg_pos={0 0}, upload=436895]: Done uploading piece. err: <nil> Dec 26 09:15:45.399602921 (497h46m45.270685966s) [noise=true , address=45.91.116.7:4010 , piece_id=JJ4K6IX476KTYCI3QE3YEYMRHQFN3M7FHFKNH4RKXJYBE5BUVN3A, seg_pos={0 0}, upload=436895]: Done uploading piece. err: <nil> Dec 26 09:15:45.399624152 (497h46m45.270707187s) [seg_pos={0 0}, upload=436895]: Segment reached optimal threshold of 13 pieces. Dec 26 09:15:45.399756765 (497h46m45.2708398s) [noise=true , address=77.90.31.18:4805 , piece_id=R4QOQLTEMSRBOWVPA6F3BYPY5AOWVDAV3SSN7MSQ5J62GC2G33DQ, seg_pos={0 0}, upload=436895]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:47850->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 09:15:45.399762696 (497h46m45.270845741s) [noise=true , address=77.90.31.18:4805 , piece_id=R4QOQLTEMSRBOWVPA6F3BYPY5AOWVDAV3SSN7MSQ5J62GC2G33DQ, seg_pos={0 0}, upload=436895]: Long tail cancellation due to optimal threshold Dec 26 09:15:45.399969370 (497h46m45.271052405s) [noise=true , address=45.91.116.4:4940 , piece_id=ERGUY7KSQ7X7TYI2CKV3IMYIPHT2UA6VFSZXUVRWLQ4NYFGTOR3Q, seg_pos={0 0}, upload=436895]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 09:15:45.399975231 (497h46m45.271058266s) [noise=true , address=45.91.116.4:4940 , piece_id=ERGUY7KSQ7X7TYI2CKV3IMYIPHT2UA6VFSZXUVRWLQ4NYFGTOR3Q, seg_pos={0 0}, upload=436895]: Long tail cancellation due to optimal threshold Dec 26 09:15:45.400064099 (497h46m45.271147144s) [noise=true , address=77.90.31.10:4520 , piece_id=FX4UDAPCIXJE7F5L6NQKFUHANKFUWYFPHO36JAQ4OTUOHCD5HU7Q, seg_pos={0 0}, upload=436895]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:52090->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:52090->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52090->77.90.31.10:4520: use of closed network connection Dec 26 09:15:45.400070923 (497h46m45.271153968s) [noise=true , address=77.90.31.10:4520 , piece_id=FX4UDAPCIXJE7F5L6NQKFUHANKFUWYFPHO36JAQ4OTUOHCD5HU7Q, seg_pos={0 0}, upload=436895]: Long tail cancellation due to optimal threshold Dec 26 09:15:45.400084258 (497h46m45.271167303s) [seg_pos={0 0}, upload=436895]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:15:45.400148190 (497h46m45.271231225s) [upload=436895]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:15:45.400153039 (497h46m45.271236084s) [upload=436895]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:15:45.689254087 (497h46m45.560337133s) [upload=436895]: Done uploading segments... Dec 26 09:15:45.689277422 (497h46m45.560360477s) [upload=436895]: Upload finished. err: <nil> Duration: 30.004021899s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 57.259965498s trace: https://jaeger.dev.storj.io/trace/6820217d73ecf2a FAIL (2025-12-26 09:15:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments... Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces. Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments... Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil> Duration: 57.259981739s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X...X.XX..X.XXX....................X....XXX..X.......X.....] duration: 5.989999864s trace: https://jaeger.dev.storj.io/trace/7e377ce9a9417865 FAIL (2025-12-26 09:14:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 10:54.879406022 (497h41m54.750489067s) [upload=436819]: Starting upload Dec 26 09:10:54.879541820 (497h41m54.750624865s) [upload=436819]: Done starting upload Dec 26 09:10:54.880024890 (497h41m54.751107935s) [upload=436819]: Uploading segments... Dec 26 09:10:54.880104391 (497h41m54.751187436s) [upload=436819]: Got next segment. Inline: false Dec 26 09:10:54.880146241 (497h41m54.751229276s) [upload=436819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:10:54.880151912 (497h41m54.751234947s) [upload=436819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:10:55.125750343 (497h41m54.996833378s) [seg_pos={0 0}, upload=436819]: Begin upload segment... Dec 26 09:10:55.125857047 (497h41m54.996940092s) [seg_pos={0 0}, upload=436819]: Done begin upload segment. Dec 26 09:10:55.125902423 (497h41m54.996985468s) [noise=true , address=145.79.227.40:4049 , piece_id=C47V2HF6RFGKAMFSOR3TE3KR7VRYJT354MLH7HCHVZGTQ2V74FUA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.125986493 (497h41m54.997069548s) [noise=true , address=45.91.116.7:4011 , piece_id=YRB6UO3BNQTAUU6MK62UZM4ZOGKOY7S6ZGUNKLKV4V5SONJL7CYQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.126687488 (497h41m54.997770523s) [noise=true , address=45.91.116.10:4083 , piece_id=IDYGUWJVXMZUGWL3THC5SWAH7GXBA7PSU7HX3U2GNPMGKXHCTNHA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.127155399 (497h41m54.998238444s) [noise=true , address=45.91.116.6:4974 , piece_id=DHWCO3O4E2TAWPEK3S5MO72AJO7R3XAQHOEBRI3NDTYDBTANK7MA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.127488613 (497h41m54.998571658s) [noise=true , address=145.79.227.45:4231 , piece_id=GQ2VZPUN2CLSVVSCZPZIAHMAPSWEYJ73S2HJB6LJWHCTV3LQGRYQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.128064139 (497h41m54.999147185s) [noise=true , address=145.79.227.46:4603 , piece_id=DQQYF2F4YHM7N6HLNHE5IETLSZEVVPWHODK5T3DGSVNS4AIOIUOA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.128505691 (497h41m54.999588736s) [noise=true , address=145.79.227.15:4875 , piece_id=RH7BSRUX6AHU6LK4IQQ7WYEWEEE3CC5EJQPHJAVGMM4D5XSZXS2Q, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.129048043 (497h41m55.000131078s) [noise=true , address=145.79.227.48:4302 , piece_id=CFD2SPCLIPJ6BMN55BZJKQSI5E7CKQWT36AT6M4RLDZWMKY6GIHQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.129593071 (497h41m55.000676116s) [noise=true , address=45.91.116.5:4299 , piece_id=UPPBKRTUTELYILKUOCI722ZTEFW6ABBQ3UC73FZMP3O4ZVS26NJA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.130183575 (497h41m55.00126662s) [noise=true , address=45.91.116.4:4939 , piece_id=D4JHH7YQIMDLXPNGYGWBURUEA6F3UE56UX3SSVGSYJAGCFQNAYEA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.130610648 (497h41m55.001693683s) [noise=true , address=145.79.227.39:4013 , piece_id=WI6ZYL643STKYE2AL2K4SO3F3RQVDHHLNAGMN4IAGPZE4FXPLRDQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.130718774 (497h41m55.001801809s) [noise=true , address=45.91.116.12:4227 , piece_id=DLGWVCIBJSUDKMUGPUQLP4R7H5ICU2XLM5RXD5LWPJSYQJHQP4EA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.131075473 (497h41m55.002158508s) [noise=true , address=145.79.227.13:4804 , piece_id=H6OHBG75MUVTHOQXHK5QGDMYRLN5Q2HLTGZCP4RQF6KA7QFX5JZA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.131473952 (497h41m55.002556987s) [noise=true , address=45.91.116.9:4261 , piece_id=76BFSEUKR6OZ23M3GI5KMWIKBLLS65AYL6BHXO5ZTOZWPZWHF3SA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.131833537 (497h41m55.002916572s) [noise=true , address=145.79.227.16:4909 , piece_id=I62QZXZXMRTIP4N6ZDDUEIQTHNWNFTQHQ2CALTYLAMUTL4HZ46BA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.132165620 (497h41m55.003248685s) [noise=true , address=77.90.31.17:4773 , piece_id=SM3WUMEECGVEEUJCDPGRS66WUYCZZ6PGNEZXD3PUB7RHRDGX53YA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:56.837614629 (497h41m56.708697674s) [upload=436819]: Next returned nil segment Dec 26 09:10:56.837655677 (497h41m56.708738722s) [upload=436819]: Waiting for error group managing segments... Dec 26 09:10:57.120464374 (497h41m56.991547419s) [noise=true , address=145.79.227.16:4909 , piece_id=I62QZXZXMRTIP4N6ZDDUEIQTHNWNFTQHQ2CALTYLAMUTL4HZ46BA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.131683237 (497h41m57.002766282s) [noise=true , address=145.79.227.15:4875 , piece_id=RH7BSRUX6AHU6LK4IQQ7WYEWEEE3CC5EJQPHJAVGMM4D5XSZXS2Q, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.132496956 (497h41m57.003579991s) [noise=true , address=145.79.227.13:4804 , piece_id=H6OHBG75MUVTHOQXHK5QGDMYRLN5Q2HLTGZCP4RQF6KA7QFX5JZA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.135213488 (497h41m57.006296523s) [noise=true , address=145.79.227.48:4302 , piece_id=CFD2SPCLIPJ6BMN55BZJKQSI5E7CKQWT36AT6M4RLDZWMKY6GIHQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.139782589 (497h41m57.010865624s) [noise=true , address=145.79.227.39:4013 , piece_id=WI6ZYL643STKYE2AL2K4SO3F3RQVDHHLNAGMN4IAGPZE4FXPLRDQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.153979191 (497h41m57.025062227s) [noise=true , address=145.79.227.46:4603 , piece_id=DQQYF2F4YHM7N6HLNHE5IETLSZEVVPWHODK5T3DGSVNS4AIOIUOA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.160470912 (497h41m57.031553947s) [noise=true , address=145.79.227.45:4231 , piece_id=GQ2VZPUN2CLSVVSCZPZIAHMAPSWEYJ73S2HJB6LJWHCTV3LQGRYQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.395125116 (497h41m57.266208161s) [noise=true , address=77.90.31.17:4773 , piece_id=SM3WUMEECGVEEUJCDPGRS66WUYCZZ6PGNEZXD3PUB7RHRDGX53YA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.445018932 (497h41m57.316101977s) [noise=true , address=45.91.116.6:4974 , piece_id=DHWCO3O4E2TAWPEK3S5MO72AJO7R3XAQHOEBRI3NDTYDBTANK7MA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.469099114 (497h41m57.340182159s) [noise=true , address=45.91.116.5:4299 , piece_id=UPPBKRTUTELYILKUOCI722ZTEFW6ABBQ3UC73FZMP3O4ZVS26NJA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.470093207 (497h41m57.341176252s) [noise=true , address=45.91.116.9:4261 , piece_id=76BFSEUKR6OZ23M3GI5KMWIKBLLS65AYL6BHXO5ZTOZWPZWHF3SA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.504656261 (497h41m57.375739306s) [noise=true , address=45.91.116.7:4011 , piece_id=YRB6UO3BNQTAUU6MK62UZM4ZOGKOY7S6ZGUNKLKV4V5SONJL7CYQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:58.081565226 (497h41m57.952648271s) [noise=true , address=45.91.116.4:4939 , piece_id=D4JHH7YQIMDLXPNGYGWBURUEA6F3UE56UX3SSVGSYJAGCFQNAYEA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:58.081589312 (497h41m57.952672347s) [seg_pos={0 0}, upload=436819]: Segment reached optimal threshold of 13 pieces. Dec 26 09:10:58.081930351 (497h41m57.953013396s) [noise=true , address=45.91.116.10:4083 , piece_id=IDYGUWJVXMZUGWL3THC5SWAH7GXBA7PSU7HX3U2GNPMGKXHCTNHA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:49172->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:49172->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49172->45.91.116.10:4083: use of closed network connection Dec 26 09:10:58.081940210 (497h41m57.953023245s) [noise=true , address=45.91.116.10:4083 , piece_id=IDYGUWJVXMZUGWL3THC5SWAH7GXBA7PSU7HX3U2GNPMGKXHCTNHA, seg_pos={0 0}, upload=436819]: Long tail cancellation due to optimal threshold Dec 26 09:10:58.082018820 (497h41m57.953101855s) [noise=true , address=45.91.116.12:4227 , piece_id=DLGWVCIBJSUDKMUGPUQLP4R7H5ICU2XLM5RXD5LWPJSYQJHQP4EA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 26 09:10:58.082038857 (497h41m57.953121902s) [noise=true , address=45.91.116.12:4227 , piece_id=DLGWVCIBJSUDKMUGPUQLP4R7H5ICU2XLM5RXD5LWPJSYQJHQP4EA, seg_pos={0 0}, upload=436819]: Long tail cancellation due to optimal threshold Dec 26 09:10:58.082351643 (497h41m57.953434698s) [noise=true , address=145.79.227.40:4049 , piece_id=C47V2HF6RFGKAMFSOR3TE3KR7VRYJT354MLH7HCHVZGTQ2V74FUA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 09:10:58.082360631 (497h41m57.953443666s) [noise=true , address=145.79.227.40:4049 , piece_id=C47V2HF6RFGKAMFSOR3TE3KR7VRYJT354MLH7HCHVZGTQ2V74FUA, seg_pos={0 0}, upload=436819]: Long tail cancellation due to optimal threshold Dec 26 09:10:58.082387421 (497h41m57.953470456s) [seg_pos={0 0}, upload=436819]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:10:58.082487081 (497h41m57.953570116s) [upload=436819]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:10:58.082495227 (497h41m57.953578272s) [upload=436819]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:10:58.378274491 (497h41m58.249357536s) [upload=436819]: Done uploading segments... Dec 26 09:10:58.378296644 (497h41m58.249379689s) [upload=436819]: Upload finished. err: <nil> Duration: 5.990025955s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 50.671349915s trace: https://jaeger.dev.storj.io/trace/6820217d73ecf2a FAIL (2025-12-26 09:14:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments... Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces. Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments... Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil> Duration: 50.671381947s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.X...X.XX..X.XXX....................X....XXX..X.......X....] duration: 6.005058309s trace: https://jaeger.dev.storj.io/trace/7e377ce9a9417865 FAIL (2025-12-26 09:14:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 10:54.879406022 (497h41m54.750489067s) [upload=436819]: Starting upload Dec 26 09:10:54.879541820 (497h41m54.750624865s) [upload=436819]: Done starting upload Dec 26 09:10:54.880024890 (497h41m54.751107935s) [upload=436819]: Uploading segments... Dec 26 09:10:54.880104391 (497h41m54.751187436s) [upload=436819]: Got next segment. Inline: false Dec 26 09:10:54.880146241 (497h41m54.751229276s) [upload=436819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:10:54.880151912 (497h41m54.751234947s) [upload=436819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:10:55.125750343 (497h41m54.996833378s) [seg_pos={0 0}, upload=436819]: Begin upload segment... Dec 26 09:10:55.125857047 (497h41m54.996940092s) [seg_pos={0 0}, upload=436819]: Done begin upload segment. Dec 26 09:10:55.125902423 (497h41m54.996985468s) [noise=true , address=145.79.227.40:4049 , piece_id=C47V2HF6RFGKAMFSOR3TE3KR7VRYJT354MLH7HCHVZGTQ2V74FUA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.125986493 (497h41m54.997069548s) [noise=true , address=45.91.116.7:4011 , piece_id=YRB6UO3BNQTAUU6MK62UZM4ZOGKOY7S6ZGUNKLKV4V5SONJL7CYQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.126687488 (497h41m54.997770523s) [noise=true , address=45.91.116.10:4083 , piece_id=IDYGUWJVXMZUGWL3THC5SWAH7GXBA7PSU7HX3U2GNPMGKXHCTNHA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.127155399 (497h41m54.998238444s) [noise=true , address=45.91.116.6:4974 , piece_id=DHWCO3O4E2TAWPEK3S5MO72AJO7R3XAQHOEBRI3NDTYDBTANK7MA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.127488613 (497h41m54.998571658s) [noise=true , address=145.79.227.45:4231 , piece_id=GQ2VZPUN2CLSVVSCZPZIAHMAPSWEYJ73S2HJB6LJWHCTV3LQGRYQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.128064139 (497h41m54.999147185s) [noise=true , address=145.79.227.46:4603 , piece_id=DQQYF2F4YHM7N6HLNHE5IETLSZEVVPWHODK5T3DGSVNS4AIOIUOA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.128505691 (497h41m54.999588736s) [noise=true , address=145.79.227.15:4875 , piece_id=RH7BSRUX6AHU6LK4IQQ7WYEWEEE3CC5EJQPHJAVGMM4D5XSZXS2Q, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.129048043 (497h41m55.000131078s) [noise=true , address=145.79.227.48:4302 , piece_id=CFD2SPCLIPJ6BMN55BZJKQSI5E7CKQWT36AT6M4RLDZWMKY6GIHQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.129593071 (497h41m55.000676116s) [noise=true , address=45.91.116.5:4299 , piece_id=UPPBKRTUTELYILKUOCI722ZTEFW6ABBQ3UC73FZMP3O4ZVS26NJA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.130183575 (497h41m55.00126662s) [noise=true , address=45.91.116.4:4939 , piece_id=D4JHH7YQIMDLXPNGYGWBURUEA6F3UE56UX3SSVGSYJAGCFQNAYEA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.130610648 (497h41m55.001693683s) [noise=true , address=145.79.227.39:4013 , piece_id=WI6ZYL643STKYE2AL2K4SO3F3RQVDHHLNAGMN4IAGPZE4FXPLRDQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.130718774 (497h41m55.001801809s) [noise=true , address=45.91.116.12:4227 , piece_id=DLGWVCIBJSUDKMUGPUQLP4R7H5ICU2XLM5RXD5LWPJSYQJHQP4EA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.131075473 (497h41m55.002158508s) [noise=true , address=145.79.227.13:4804 , piece_id=H6OHBG75MUVTHOQXHK5QGDMYRLN5Q2HLTGZCP4RQF6KA7QFX5JZA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.131473952 (497h41m55.002556987s) [noise=true , address=45.91.116.9:4261 , piece_id=76BFSEUKR6OZ23M3GI5KMWIKBLLS65AYL6BHXO5ZTOZWPZWHF3SA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.131833537 (497h41m55.002916572s) [noise=true , address=145.79.227.16:4909 , piece_id=I62QZXZXMRTIP4N6ZDDUEIQTHNWNFTQHQ2CALTYLAMUTL4HZ46BA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.132165620 (497h41m55.003248685s) [noise=true , address=77.90.31.17:4773 , piece_id=SM3WUMEECGVEEUJCDPGRS66WUYCZZ6PGNEZXD3PUB7RHRDGX53YA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:56.837614629 (497h41m56.708697674s) [upload=436819]: Next returned nil segment Dec 26 09:10:56.837655677 (497h41m56.708738722s) [upload=436819]: Waiting for error group managing segments... Dec 26 09:10:57.120464374 (497h41m56.991547419s) [noise=true , address=145.79.227.16:4909 , piece_id=I62QZXZXMRTIP4N6ZDDUEIQTHNWNFTQHQ2CALTYLAMUTL4HZ46BA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.131683237 (497h41m57.002766282s) [noise=true , address=145.79.227.15:4875 , piece_id=RH7BSRUX6AHU6LK4IQQ7WYEWEEE3CC5EJQPHJAVGMM4D5XSZXS2Q, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.132496956 (497h41m57.003579991s) [noise=true , address=145.79.227.13:4804 , piece_id=H6OHBG75MUVTHOQXHK5QGDMYRLN5Q2HLTGZCP4RQF6KA7QFX5JZA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.135213488 (497h41m57.006296523s) [noise=true , address=145.79.227.48:4302 , piece_id=CFD2SPCLIPJ6BMN55BZJKQSI5E7CKQWT36AT6M4RLDZWMKY6GIHQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.139782589 (497h41m57.010865624s) [noise=true , address=145.79.227.39:4013 , piece_id=WI6ZYL643STKYE2AL2K4SO3F3RQVDHHLNAGMN4IAGPZE4FXPLRDQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.153979191 (497h41m57.025062227s) [noise=true , address=145.79.227.46:4603 , piece_id=DQQYF2F4YHM7N6HLNHE5IETLSZEVVPWHODK5T3DGSVNS4AIOIUOA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.160470912 (497h41m57.031553947s) [noise=true , address=145.79.227.45:4231 , piece_id=GQ2VZPUN2CLSVVSCZPZIAHMAPSWEYJ73S2HJB6LJWHCTV3LQGRYQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.395125116 (497h41m57.266208161s) [noise=true , address=77.90.31.17:4773 , piece_id=SM3WUMEECGVEEUJCDPGRS66WUYCZZ6PGNEZXD3PUB7RHRDGX53YA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.445018932 (497h41m57.316101977s) [noise=true , address=45.91.116.6:4974 , piece_id=DHWCO3O4E2TAWPEK3S5MO72AJO7R3XAQHOEBRI3NDTYDBTANK7MA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.469099114 (497h41m57.340182159s) [noise=true , address=45.91.116.5:4299 , piece_id=UPPBKRTUTELYILKUOCI722ZTEFW6ABBQ3UC73FZMP3O4ZVS26NJA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.470093207 (497h41m57.341176252s) [noise=true , address=45.91.116.9:4261 , piece_id=76BFSEUKR6OZ23M3GI5KMWIKBLLS65AYL6BHXO5ZTOZWPZWHF3SA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.504656261 (497h41m57.375739306s) [noise=true , address=45.91.116.7:4011 , piece_id=YRB6UO3BNQTAUU6MK62UZM4ZOGKOY7S6ZGUNKLKV4V5SONJL7CYQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:58.081565226 (497h41m57.952648271s) [noise=true , address=45.91.116.4:4939 , piece_id=D4JHH7YQIMDLXPNGYGWBURUEA6F3UE56UX3SSVGSYJAGCFQNAYEA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:58.081589312 (497h41m57.952672347s) [seg_pos={0 0}, upload=436819]: Segment reached optimal threshold of 13 pieces. Dec 26 09:10:58.081930351 (497h41m57.953013396s) [noise=true , address=45.91.116.10:4083 , piece_id=IDYGUWJVXMZUGWL3THC5SWAH7GXBA7PSU7HX3U2GNPMGKXHCTNHA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:49172->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:49172->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49172->45.91.116.10:4083: use of closed network connection Dec 26 09:10:58.081940210 (497h41m57.953023245s) [noise=true , address=45.91.116.10:4083 , piece_id=IDYGUWJVXMZUGWL3THC5SWAH7GXBA7PSU7HX3U2GNPMGKXHCTNHA, seg_pos={0 0}, upload=436819]: Long tail cancellation due to optimal threshold Dec 26 09:10:58.082018820 (497h41m57.953101855s) [noise=true , address=45.91.116.12:4227 , piece_id=DLGWVCIBJSUDKMUGPUQLP4R7H5ICU2XLM5RXD5LWPJSYQJHQP4EA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 26 09:10:58.082038857 (497h41m57.953121902s) [noise=true , address=45.91.116.12:4227 , piece_id=DLGWVCIBJSUDKMUGPUQLP4R7H5ICU2XLM5RXD5LWPJSYQJHQP4EA, seg_pos={0 0}, upload=436819]: Long tail cancellation due to optimal threshold Dec 26 09:10:58.082351643 (497h41m57.953434698s) [noise=true , address=145.79.227.40:4049 , piece_id=C47V2HF6RFGKAMFSOR3TE3KR7VRYJT354MLH7HCHVZGTQ2V74FUA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 09:10:58.082360631 (497h41m57.953443666s) [noise=true , address=145.79.227.40:4049 , piece_id=C47V2HF6RFGKAMFSOR3TE3KR7VRYJT354MLH7HCHVZGTQ2V74FUA, seg_pos={0 0}, upload=436819]: Long tail cancellation due to optimal threshold Dec 26 09:10:58.082387421 (497h41m57.953470456s) [seg_pos={0 0}, upload=436819]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:10:58.082487081 (497h41m57.953570116s) [upload=436819]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:10:58.082495227 (497h41m57.953578272s) [upload=436819]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:10:58.378274491 (497h41m58.249357536s) [upload=436819]: Done uploading segments... Dec 26 09:10:58.378296644 (497h41m58.249379689s) [upload=436819]: Upload finished. err: <nil> Duration: 6.005110748s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 41.172837936s trace: https://jaeger.dev.storj.io/trace/6820217d73ecf2a FAIL (2025-12-26 09:13:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments... Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces. Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments... Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil> Duration: 41.172852454s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.X...X.XX..X.XXX....................X....XXX..X.......X...] duration: 6.002944041s trace: https://jaeger.dev.storj.io/trace/7e377ce9a9417865 FAIL (2025-12-26 09:13:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 10:54.879406022 (497h41m54.750489067s) [upload=436819]: Starting upload Dec 26 09:10:54.879541820 (497h41m54.750624865s) [upload=436819]: Done starting upload Dec 26 09:10:54.880024890 (497h41m54.751107935s) [upload=436819]: Uploading segments... Dec 26 09:10:54.880104391 (497h41m54.751187436s) [upload=436819]: Got next segment. Inline: false Dec 26 09:10:54.880146241 (497h41m54.751229276s) [upload=436819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:10:54.880151912 (497h41m54.751234947s) [upload=436819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:10:55.125750343 (497h41m54.996833378s) [seg_pos={0 0}, upload=436819]: Begin upload segment... Dec 26 09:10:55.125857047 (497h41m54.996940092s) [seg_pos={0 0}, upload=436819]: Done begin upload segment. Dec 26 09:10:55.125902423 (497h41m54.996985468s) [noise=true , address=145.79.227.40:4049 , piece_id=C47V2HF6RFGKAMFSOR3TE3KR7VRYJT354MLH7HCHVZGTQ2V74FUA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.125986493 (497h41m54.997069548s) [noise=true , address=45.91.116.7:4011 , piece_id=YRB6UO3BNQTAUU6MK62UZM4ZOGKOY7S6ZGUNKLKV4V5SONJL7CYQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.126687488 (497h41m54.997770523s) [noise=true , address=45.91.116.10:4083 , piece_id=IDYGUWJVXMZUGWL3THC5SWAH7GXBA7PSU7HX3U2GNPMGKXHCTNHA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.127155399 (497h41m54.998238444s) [noise=true , address=45.91.116.6:4974 , piece_id=DHWCO3O4E2TAWPEK3S5MO72AJO7R3XAQHOEBRI3NDTYDBTANK7MA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.127488613 (497h41m54.998571658s) [noise=true , address=145.79.227.45:4231 , piece_id=GQ2VZPUN2CLSVVSCZPZIAHMAPSWEYJ73S2HJB6LJWHCTV3LQGRYQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.128064139 (497h41m54.999147185s) [noise=true , address=145.79.227.46:4603 , piece_id=DQQYF2F4YHM7N6HLNHE5IETLSZEVVPWHODK5T3DGSVNS4AIOIUOA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.128505691 (497h41m54.999588736s) [noise=true , address=145.79.227.15:4875 , piece_id=RH7BSRUX6AHU6LK4IQQ7WYEWEEE3CC5EJQPHJAVGMM4D5XSZXS2Q, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.129048043 (497h41m55.000131078s) [noise=true , address=145.79.227.48:4302 , piece_id=CFD2SPCLIPJ6BMN55BZJKQSI5E7CKQWT36AT6M4RLDZWMKY6GIHQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.129593071 (497h41m55.000676116s) [noise=true , address=45.91.116.5:4299 , piece_id=UPPBKRTUTELYILKUOCI722ZTEFW6ABBQ3UC73FZMP3O4ZVS26NJA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.130183575 (497h41m55.00126662s) [noise=true , address=45.91.116.4:4939 , piece_id=D4JHH7YQIMDLXPNGYGWBURUEA6F3UE56UX3SSVGSYJAGCFQNAYEA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.130610648 (497h41m55.001693683s) [noise=true , address=145.79.227.39:4013 , piece_id=WI6ZYL643STKYE2AL2K4SO3F3RQVDHHLNAGMN4IAGPZE4FXPLRDQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.130718774 (497h41m55.001801809s) [noise=true , address=45.91.116.12:4227 , piece_id=DLGWVCIBJSUDKMUGPUQLP4R7H5ICU2XLM5RXD5LWPJSYQJHQP4EA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.131075473 (497h41m55.002158508s) [noise=true , address=145.79.227.13:4804 , piece_id=H6OHBG75MUVTHOQXHK5QGDMYRLN5Q2HLTGZCP4RQF6KA7QFX5JZA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.131473952 (497h41m55.002556987s) [noise=true , address=45.91.116.9:4261 , piece_id=76BFSEUKR6OZ23M3GI5KMWIKBLLS65AYL6BHXO5ZTOZWPZWHF3SA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.131833537 (497h41m55.002916572s) [noise=true , address=145.79.227.16:4909 , piece_id=I62QZXZXMRTIP4N6ZDDUEIQTHNWNFTQHQ2CALTYLAMUTL4HZ46BA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.132165620 (497h41m55.003248685s) [noise=true , address=77.90.31.17:4773 , piece_id=SM3WUMEECGVEEUJCDPGRS66WUYCZZ6PGNEZXD3PUB7RHRDGX53YA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:56.837614629 (497h41m56.708697674s) [upload=436819]: Next returned nil segment Dec 26 09:10:56.837655677 (497h41m56.708738722s) [upload=436819]: Waiting for error group managing segments... Dec 26 09:10:57.120464374 (497h41m56.991547419s) [noise=true , address=145.79.227.16:4909 , piece_id=I62QZXZXMRTIP4N6ZDDUEIQTHNWNFTQHQ2CALTYLAMUTL4HZ46BA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.131683237 (497h41m57.002766282s) [noise=true , address=145.79.227.15:4875 , piece_id=RH7BSRUX6AHU6LK4IQQ7WYEWEEE3CC5EJQPHJAVGMM4D5XSZXS2Q, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.132496956 (497h41m57.003579991s) [noise=true , address=145.79.227.13:4804 , piece_id=H6OHBG75MUVTHOQXHK5QGDMYRLN5Q2HLTGZCP4RQF6KA7QFX5JZA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.135213488 (497h41m57.006296523s) [noise=true , address=145.79.227.48:4302 , piece_id=CFD2SPCLIPJ6BMN55BZJKQSI5E7CKQWT36AT6M4RLDZWMKY6GIHQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.139782589 (497h41m57.010865624s) [noise=true , address=145.79.227.39:4013 , piece_id=WI6ZYL643STKYE2AL2K4SO3F3RQVDHHLNAGMN4IAGPZE4FXPLRDQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.153979191 (497h41m57.025062227s) [noise=true , address=145.79.227.46:4603 , piece_id=DQQYF2F4YHM7N6HLNHE5IETLSZEVVPWHODK5T3DGSVNS4AIOIUOA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.160470912 (497h41m57.031553947s) [noise=true , address=145.79.227.45:4231 , piece_id=GQ2VZPUN2CLSVVSCZPZIAHMAPSWEYJ73S2HJB6LJWHCTV3LQGRYQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.395125116 (497h41m57.266208161s) [noise=true , address=77.90.31.17:4773 , piece_id=SM3WUMEECGVEEUJCDPGRS66WUYCZZ6PGNEZXD3PUB7RHRDGX53YA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.445018932 (497h41m57.316101977s) [noise=true , address=45.91.116.6:4974 , piece_id=DHWCO3O4E2TAWPEK3S5MO72AJO7R3XAQHOEBRI3NDTYDBTANK7MA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.469099114 (497h41m57.340182159s) [noise=true , address=45.91.116.5:4299 , piece_id=UPPBKRTUTELYILKUOCI722ZTEFW6ABBQ3UC73FZMP3O4ZVS26NJA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.470093207 (497h41m57.341176252s) [noise=true , address=45.91.116.9:4261 , piece_id=76BFSEUKR6OZ23M3GI5KMWIKBLLS65AYL6BHXO5ZTOZWPZWHF3SA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.504656261 (497h41m57.375739306s) [noise=true , address=45.91.116.7:4011 , piece_id=YRB6UO3BNQTAUU6MK62UZM4ZOGKOY7S6ZGUNKLKV4V5SONJL7CYQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:58.081565226 (497h41m57.952648271s) [noise=true , address=45.91.116.4:4939 , piece_id=D4JHH7YQIMDLXPNGYGWBURUEA6F3UE56UX3SSVGSYJAGCFQNAYEA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:58.081589312 (497h41m57.952672347s) [seg_pos={0 0}, upload=436819]: Segment reached optimal threshold of 13 pieces. Dec 26 09:10:58.081930351 (497h41m57.953013396s) [noise=true , address=45.91.116.10:4083 , piece_id=IDYGUWJVXMZUGWL3THC5SWAH7GXBA7PSU7HX3U2GNPMGKXHCTNHA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:49172->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:49172->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49172->45.91.116.10:4083: use of closed network connection Dec 26 09:10:58.081940210 (497h41m57.953023245s) [noise=true , address=45.91.116.10:4083 , piece_id=IDYGUWJVXMZUGWL3THC5SWAH7GXBA7PSU7HX3U2GNPMGKXHCTNHA, seg_pos={0 0}, upload=436819]: Long tail cancellation due to optimal threshold Dec 26 09:10:58.082018820 (497h41m57.953101855s) [noise=true , address=45.91.116.12:4227 , piece_id=DLGWVCIBJSUDKMUGPUQLP4R7H5ICU2XLM5RXD5LWPJSYQJHQP4EA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 26 09:10:58.082038857 (497h41m57.953121902s) [noise=true , address=45.91.116.12:4227 , piece_id=DLGWVCIBJSUDKMUGPUQLP4R7H5ICU2XLM5RXD5LWPJSYQJHQP4EA, seg_pos={0 0}, upload=436819]: Long tail cancellation due to optimal threshold Dec 26 09:10:58.082351643 (497h41m57.953434698s) [noise=true , address=145.79.227.40:4049 , piece_id=C47V2HF6RFGKAMFSOR3TE3KR7VRYJT354MLH7HCHVZGTQ2V74FUA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 09:10:58.082360631 (497h41m57.953443666s) [noise=true , address=145.79.227.40:4049 , piece_id=C47V2HF6RFGKAMFSOR3TE3KR7VRYJT354MLH7HCHVZGTQ2V74FUA, seg_pos={0 0}, upload=436819]: Long tail cancellation due to optimal threshold Dec 26 09:10:58.082387421 (497h41m57.953470456s) [seg_pos={0 0}, upload=436819]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:10:58.082487081 (497h41m57.953570116s) [upload=436819]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:10:58.082495227 (497h41m57.953578272s) [upload=436819]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:10:58.378274491 (497h41m58.249357536s) [upload=436819]: Done uploading segments... Dec 26 09:10:58.378296644 (497h41m58.249379689s) [upload=436819]: Upload finished. err: <nil> Duration: 6.002981724s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 41.769309542s trace: https://jaeger.dev.storj.io/trace/6820217d73ecf2a FAIL (2025-12-26 09:12:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments... Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces. Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments... Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil> Duration: 41.769328828s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X.X...X.XX..X.XXX....................X....XXX..X.......X..] duration: 5.727394239s trace: https://jaeger.dev.storj.io/trace/7e377ce9a9417865 FAIL (2025-12-26 09:12:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 10:54.879406022 (497h41m54.750489067s) [upload=436819]: Starting upload Dec 26 09:10:54.879541820 (497h41m54.750624865s) [upload=436819]: Done starting upload Dec 26 09:10:54.880024890 (497h41m54.751107935s) [upload=436819]: Uploading segments... Dec 26 09:10:54.880104391 (497h41m54.751187436s) [upload=436819]: Got next segment. Inline: false Dec 26 09:10:54.880146241 (497h41m54.751229276s) [upload=436819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:10:54.880151912 (497h41m54.751234947s) [upload=436819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:10:55.125750343 (497h41m54.996833378s) [seg_pos={0 0}, upload=436819]: Begin upload segment... Dec 26 09:10:55.125857047 (497h41m54.996940092s) [seg_pos={0 0}, upload=436819]: Done begin upload segment. Dec 26 09:10:55.125902423 (497h41m54.996985468s) [noise=true , address=145.79.227.40:4049 , piece_id=C47V2HF6RFGKAMFSOR3TE3KR7VRYJT354MLH7HCHVZGTQ2V74FUA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.125986493 (497h41m54.997069548s) [noise=true , address=45.91.116.7:4011 , piece_id=YRB6UO3BNQTAUU6MK62UZM4ZOGKOY7S6ZGUNKLKV4V5SONJL7CYQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.126687488 (497h41m54.997770523s) [noise=true , address=45.91.116.10:4083 , piece_id=IDYGUWJVXMZUGWL3THC5SWAH7GXBA7PSU7HX3U2GNPMGKXHCTNHA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.127155399 (497h41m54.998238444s) [noise=true , address=45.91.116.6:4974 , piece_id=DHWCO3O4E2TAWPEK3S5MO72AJO7R3XAQHOEBRI3NDTYDBTANK7MA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.127488613 (497h41m54.998571658s) [noise=true , address=145.79.227.45:4231 , piece_id=GQ2VZPUN2CLSVVSCZPZIAHMAPSWEYJ73S2HJB6LJWHCTV3LQGRYQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.128064139 (497h41m54.999147185s) [noise=true , address=145.79.227.46:4603 , piece_id=DQQYF2F4YHM7N6HLNHE5IETLSZEVVPWHODK5T3DGSVNS4AIOIUOA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.128505691 (497h41m54.999588736s) [noise=true , address=145.79.227.15:4875 , piece_id=RH7BSRUX6AHU6LK4IQQ7WYEWEEE3CC5EJQPHJAVGMM4D5XSZXS2Q, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.129048043 (497h41m55.000131078s) [noise=true , address=145.79.227.48:4302 , piece_id=CFD2SPCLIPJ6BMN55BZJKQSI5E7CKQWT36AT6M4RLDZWMKY6GIHQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.129593071 (497h41m55.000676116s) [noise=true , address=45.91.116.5:4299 , piece_id=UPPBKRTUTELYILKUOCI722ZTEFW6ABBQ3UC73FZMP3O4ZVS26NJA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.130183575 (497h41m55.00126662s) [noise=true , address=45.91.116.4:4939 , piece_id=D4JHH7YQIMDLXPNGYGWBURUEA6F3UE56UX3SSVGSYJAGCFQNAYEA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.130610648 (497h41m55.001693683s) [noise=true , address=145.79.227.39:4013 , piece_id=WI6ZYL643STKYE2AL2K4SO3F3RQVDHHLNAGMN4IAGPZE4FXPLRDQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.130718774 (497h41m55.001801809s) [noise=true , address=45.91.116.12:4227 , piece_id=DLGWVCIBJSUDKMUGPUQLP4R7H5ICU2XLM5RXD5LWPJSYQJHQP4EA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.131075473 (497h41m55.002158508s) [noise=true , address=145.79.227.13:4804 , piece_id=H6OHBG75MUVTHOQXHK5QGDMYRLN5Q2HLTGZCP4RQF6KA7QFX5JZA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.131473952 (497h41m55.002556987s) [noise=true , address=45.91.116.9:4261 , piece_id=76BFSEUKR6OZ23M3GI5KMWIKBLLS65AYL6BHXO5ZTOZWPZWHF3SA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.131833537 (497h41m55.002916572s) [noise=true , address=145.79.227.16:4909 , piece_id=I62QZXZXMRTIP4N6ZDDUEIQTHNWNFTQHQ2CALTYLAMUTL4HZ46BA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.132165620 (497h41m55.003248685s) [noise=true , address=77.90.31.17:4773 , piece_id=SM3WUMEECGVEEUJCDPGRS66WUYCZZ6PGNEZXD3PUB7RHRDGX53YA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:56.837614629 (497h41m56.708697674s) [upload=436819]: Next returned nil segment Dec 26 09:10:56.837655677 (497h41m56.708738722s) [upload=436819]: Waiting for error group managing segments... Dec 26 09:10:57.120464374 (497h41m56.991547419s) [noise=true , address=145.79.227.16:4909 , piece_id=I62QZXZXMRTIP4N6ZDDUEIQTHNWNFTQHQ2CALTYLAMUTL4HZ46BA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.131683237 (497h41m57.002766282s) [noise=true , address=145.79.227.15:4875 , piece_id=RH7BSRUX6AHU6LK4IQQ7WYEWEEE3CC5EJQPHJAVGMM4D5XSZXS2Q, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.132496956 (497h41m57.003579991s) [noise=true , address=145.79.227.13:4804 , piece_id=H6OHBG75MUVTHOQXHK5QGDMYRLN5Q2HLTGZCP4RQF6KA7QFX5JZA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.135213488 (497h41m57.006296523s) [noise=true , address=145.79.227.48:4302 , piece_id=CFD2SPCLIPJ6BMN55BZJKQSI5E7CKQWT36AT6M4RLDZWMKY6GIHQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.139782589 (497h41m57.010865624s) [noise=true , address=145.79.227.39:4013 , piece_id=WI6ZYL643STKYE2AL2K4SO3F3RQVDHHLNAGMN4IAGPZE4FXPLRDQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.153979191 (497h41m57.025062227s) [noise=true , address=145.79.227.46:4603 , piece_id=DQQYF2F4YHM7N6HLNHE5IETLSZEVVPWHODK5T3DGSVNS4AIOIUOA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.160470912 (497h41m57.031553947s) [noise=true , address=145.79.227.45:4231 , piece_id=GQ2VZPUN2CLSVVSCZPZIAHMAPSWEYJ73S2HJB6LJWHCTV3LQGRYQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.395125116 (497h41m57.266208161s) [noise=true , address=77.90.31.17:4773 , piece_id=SM3WUMEECGVEEUJCDPGRS66WUYCZZ6PGNEZXD3PUB7RHRDGX53YA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.445018932 (497h41m57.316101977s) [noise=true , address=45.91.116.6:4974 , piece_id=DHWCO3O4E2TAWPEK3S5MO72AJO7R3XAQHOEBRI3NDTYDBTANK7MA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.469099114 (497h41m57.340182159s) [noise=true , address=45.91.116.5:4299 , piece_id=UPPBKRTUTELYILKUOCI722ZTEFW6ABBQ3UC73FZMP3O4ZVS26NJA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.470093207 (497h41m57.341176252s) [noise=true , address=45.91.116.9:4261 , piece_id=76BFSEUKR6OZ23M3GI5KMWIKBLLS65AYL6BHXO5ZTOZWPZWHF3SA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.504656261 (497h41m57.375739306s) [noise=true , address=45.91.116.7:4011 , piece_id=YRB6UO3BNQTAUU6MK62UZM4ZOGKOY7S6ZGUNKLKV4V5SONJL7CYQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:58.081565226 (497h41m57.952648271s) [noise=true , address=45.91.116.4:4939 , piece_id=D4JHH7YQIMDLXPNGYGWBURUEA6F3UE56UX3SSVGSYJAGCFQNAYEA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:58.081589312 (497h41m57.952672347s) [seg_pos={0 0}, upload=436819]: Segment reached optimal threshold of 13 pieces. Dec 26 09:10:58.081930351 (497h41m57.953013396s) [noise=true , address=45.91.116.10:4083 , piece_id=IDYGUWJVXMZUGWL3THC5SWAH7GXBA7PSU7HX3U2GNPMGKXHCTNHA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:49172->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:49172->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49172->45.91.116.10:4083: use of closed network connection Dec 26 09:10:58.081940210 (497h41m57.953023245s) [noise=true , address=45.91.116.10:4083 , piece_id=IDYGUWJVXMZUGWL3THC5SWAH7GXBA7PSU7HX3U2GNPMGKXHCTNHA, seg_pos={0 0}, upload=436819]: Long tail cancellation due to optimal threshold Dec 26 09:10:58.082018820 (497h41m57.953101855s) [noise=true , address=45.91.116.12:4227 , piece_id=DLGWVCIBJSUDKMUGPUQLP4R7H5ICU2XLM5RXD5LWPJSYQJHQP4EA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 26 09:10:58.082038857 (497h41m57.953121902s) [noise=true , address=45.91.116.12:4227 , piece_id=DLGWVCIBJSUDKMUGPUQLP4R7H5ICU2XLM5RXD5LWPJSYQJHQP4EA, seg_pos={0 0}, upload=436819]: Long tail cancellation due to optimal threshold Dec 26 09:10:58.082351643 (497h41m57.953434698s) [noise=true , address=145.79.227.40:4049 , piece_id=C47V2HF6RFGKAMFSOR3TE3KR7VRYJT354MLH7HCHVZGTQ2V74FUA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 09:10:58.082360631 (497h41m57.953443666s) [noise=true , address=145.79.227.40:4049 , piece_id=C47V2HF6RFGKAMFSOR3TE3KR7VRYJT354MLH7HCHVZGTQ2V74FUA, seg_pos={0 0}, upload=436819]: Long tail cancellation due to optimal threshold Dec 26 09:10:58.082387421 (497h41m57.953470456s) [seg_pos={0 0}, upload=436819]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:10:58.082487081 (497h41m57.953570116s) [upload=436819]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:10:58.082495227 (497h41m57.953578272s) [upload=436819]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:10:58.378274491 (497h41m58.249357536s) [upload=436819]: Done uploading segments... Dec 26 09:10:58.378296644 (497h41m58.249379689s) [upload=436819]: Upload finished. err: <nil> Duration: 5.72741613s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X.X...X.XX..X.XXX....................X....XXX..X.......X.] duration: 5.96913552s trace: https://jaeger.dev.storj.io/trace/7e377ce9a9417865 FAIL (2025-12-26 09:11:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 10:54.879406022 (497h41m54.750489067s) [upload=436819]: Starting upload Dec 26 09:10:54.879541820 (497h41m54.750624865s) [upload=436819]: Done starting upload Dec 26 09:10:54.880024890 (497h41m54.751107935s) [upload=436819]: Uploading segments... Dec 26 09:10:54.880104391 (497h41m54.751187436s) [upload=436819]: Got next segment. Inline: false Dec 26 09:10:54.880146241 (497h41m54.751229276s) [upload=436819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:10:54.880151912 (497h41m54.751234947s) [upload=436819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:10:55.125750343 (497h41m54.996833378s) [seg_pos={0 0}, upload=436819]: Begin upload segment... Dec 26 09:10:55.125857047 (497h41m54.996940092s) [seg_pos={0 0}, upload=436819]: Done begin upload segment. Dec 26 09:10:55.125902423 (497h41m54.996985468s) [noise=true , address=145.79.227.40:4049 , piece_id=C47V2HF6RFGKAMFSOR3TE3KR7VRYJT354MLH7HCHVZGTQ2V74FUA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.125986493 (497h41m54.997069548s) [noise=true , address=45.91.116.7:4011 , piece_id=YRB6UO3BNQTAUU6MK62UZM4ZOGKOY7S6ZGUNKLKV4V5SONJL7CYQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.126687488 (497h41m54.997770523s) [noise=true , address=45.91.116.10:4083 , piece_id=IDYGUWJVXMZUGWL3THC5SWAH7GXBA7PSU7HX3U2GNPMGKXHCTNHA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.127155399 (497h41m54.998238444s) [noise=true , address=45.91.116.6:4974 , piece_id=DHWCO3O4E2TAWPEK3S5MO72AJO7R3XAQHOEBRI3NDTYDBTANK7MA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.127488613 (497h41m54.998571658s) [noise=true , address=145.79.227.45:4231 , piece_id=GQ2VZPUN2CLSVVSCZPZIAHMAPSWEYJ73S2HJB6LJWHCTV3LQGRYQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.128064139 (497h41m54.999147185s) [noise=true , address=145.79.227.46:4603 , piece_id=DQQYF2F4YHM7N6HLNHE5IETLSZEVVPWHODK5T3DGSVNS4AIOIUOA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.128505691 (497h41m54.999588736s) [noise=true , address=145.79.227.15:4875 , piece_id=RH7BSRUX6AHU6LK4IQQ7WYEWEEE3CC5EJQPHJAVGMM4D5XSZXS2Q, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.129048043 (497h41m55.000131078s) [noise=true , address=145.79.227.48:4302 , piece_id=CFD2SPCLIPJ6BMN55BZJKQSI5E7CKQWT36AT6M4RLDZWMKY6GIHQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.129593071 (497h41m55.000676116s) [noise=true , address=45.91.116.5:4299 , piece_id=UPPBKRTUTELYILKUOCI722ZTEFW6ABBQ3UC73FZMP3O4ZVS26NJA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.130183575 (497h41m55.00126662s) [noise=true , address=45.91.116.4:4939 , piece_id=D4JHH7YQIMDLXPNGYGWBURUEA6F3UE56UX3SSVGSYJAGCFQNAYEA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.130610648 (497h41m55.001693683s) [noise=true , address=145.79.227.39:4013 , piece_id=WI6ZYL643STKYE2AL2K4SO3F3RQVDHHLNAGMN4IAGPZE4FXPLRDQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.130718774 (497h41m55.001801809s) [noise=true , address=45.91.116.12:4227 , piece_id=DLGWVCIBJSUDKMUGPUQLP4R7H5ICU2XLM5RXD5LWPJSYQJHQP4EA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.131075473 (497h41m55.002158508s) [noise=true , address=145.79.227.13:4804 , piece_id=H6OHBG75MUVTHOQXHK5QGDMYRLN5Q2HLTGZCP4RQF6KA7QFX5JZA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.131473952 (497h41m55.002556987s) [noise=true , address=45.91.116.9:4261 , piece_id=76BFSEUKR6OZ23M3GI5KMWIKBLLS65AYL6BHXO5ZTOZWPZWHF3SA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.131833537 (497h41m55.002916572s) [noise=true , address=145.79.227.16:4909 , piece_id=I62QZXZXMRTIP4N6ZDDUEIQTHNWNFTQHQ2CALTYLAMUTL4HZ46BA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.132165620 (497h41m55.003248685s) [noise=true , address=77.90.31.17:4773 , piece_id=SM3WUMEECGVEEUJCDPGRS66WUYCZZ6PGNEZXD3PUB7RHRDGX53YA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:56.837614629 (497h41m56.708697674s) [upload=436819]: Next returned nil segment Dec 26 09:10:56.837655677 (497h41m56.708738722s) [upload=436819]: Waiting for error group managing segments... Dec 26 09:10:57.120464374 (497h41m56.991547419s) [noise=true , address=145.79.227.16:4909 , piece_id=I62QZXZXMRTIP4N6ZDDUEIQTHNWNFTQHQ2CALTYLAMUTL4HZ46BA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.131683237 (497h41m57.002766282s) [noise=true , address=145.79.227.15:4875 , piece_id=RH7BSRUX6AHU6LK4IQQ7WYEWEEE3CC5EJQPHJAVGMM4D5XSZXS2Q, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.132496956 (497h41m57.003579991s) [noise=true , address=145.79.227.13:4804 , piece_id=H6OHBG75MUVTHOQXHK5QGDMYRLN5Q2HLTGZCP4RQF6KA7QFX5JZA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.135213488 (497h41m57.006296523s) [noise=true , address=145.79.227.48:4302 , piece_id=CFD2SPCLIPJ6BMN55BZJKQSI5E7CKQWT36AT6M4RLDZWMKY6GIHQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.139782589 (497h41m57.010865624s) [noise=true , address=145.79.227.39:4013 , piece_id=WI6ZYL643STKYE2AL2K4SO3F3RQVDHHLNAGMN4IAGPZE4FXPLRDQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.153979191 (497h41m57.025062227s) [noise=true , address=145.79.227.46:4603 , piece_id=DQQYF2F4YHM7N6HLNHE5IETLSZEVVPWHODK5T3DGSVNS4AIOIUOA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.160470912 (497h41m57.031553947s) [noise=true , address=145.79.227.45:4231 , piece_id=GQ2VZPUN2CLSVVSCZPZIAHMAPSWEYJ73S2HJB6LJWHCTV3LQGRYQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.395125116 (497h41m57.266208161s) [noise=true , address=77.90.31.17:4773 , piece_id=SM3WUMEECGVEEUJCDPGRS66WUYCZZ6PGNEZXD3PUB7RHRDGX53YA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.445018932 (497h41m57.316101977s) [noise=true , address=45.91.116.6:4974 , piece_id=DHWCO3O4E2TAWPEK3S5MO72AJO7R3XAQHOEBRI3NDTYDBTANK7MA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.469099114 (497h41m57.340182159s) [noise=true , address=45.91.116.5:4299 , piece_id=UPPBKRTUTELYILKUOCI722ZTEFW6ABBQ3UC73FZMP3O4ZVS26NJA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.470093207 (497h41m57.341176252s) [noise=true , address=45.91.116.9:4261 , piece_id=76BFSEUKR6OZ23M3GI5KMWIKBLLS65AYL6BHXO5ZTOZWPZWHF3SA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.504656261 (497h41m57.375739306s) [noise=true , address=45.91.116.7:4011 , piece_id=YRB6UO3BNQTAUU6MK62UZM4ZOGKOY7S6ZGUNKLKV4V5SONJL7CYQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:58.081565226 (497h41m57.952648271s) [noise=true , address=45.91.116.4:4939 , piece_id=D4JHH7YQIMDLXPNGYGWBURUEA6F3UE56UX3SSVGSYJAGCFQNAYEA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:58.081589312 (497h41m57.952672347s) [seg_pos={0 0}, upload=436819]: Segment reached optimal threshold of 13 pieces. Dec 26 09:10:58.081930351 (497h41m57.953013396s) [noise=true , address=45.91.116.10:4083 , piece_id=IDYGUWJVXMZUGWL3THC5SWAH7GXBA7PSU7HX3U2GNPMGKXHCTNHA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:49172->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:49172->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49172->45.91.116.10:4083: use of closed network connection Dec 26 09:10:58.081940210 (497h41m57.953023245s) [noise=true , address=45.91.116.10:4083 , piece_id=IDYGUWJVXMZUGWL3THC5SWAH7GXBA7PSU7HX3U2GNPMGKXHCTNHA, seg_pos={0 0}, upload=436819]: Long tail cancellation due to optimal threshold Dec 26 09:10:58.082018820 (497h41m57.953101855s) [noise=true , address=45.91.116.12:4227 , piece_id=DLGWVCIBJSUDKMUGPUQLP4R7H5ICU2XLM5RXD5LWPJSYQJHQP4EA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 26 09:10:58.082038857 (497h41m57.953121902s) [noise=true , address=45.91.116.12:4227 , piece_id=DLGWVCIBJSUDKMUGPUQLP4R7H5ICU2XLM5RXD5LWPJSYQJHQP4EA, seg_pos={0 0}, upload=436819]: Long tail cancellation due to optimal threshold Dec 26 09:10:58.082351643 (497h41m57.953434698s) [noise=true , address=145.79.227.40:4049 , piece_id=C47V2HF6RFGKAMFSOR3TE3KR7VRYJT354MLH7HCHVZGTQ2V74FUA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 09:10:58.082360631 (497h41m57.953443666s) [noise=true , address=145.79.227.40:4049 , piece_id=C47V2HF6RFGKAMFSOR3TE3KR7VRYJT354MLH7HCHVZGTQ2V74FUA, seg_pos={0 0}, upload=436819]: Long tail cancellation due to optimal threshold Dec 26 09:10:58.082387421 (497h41m57.953470456s) [seg_pos={0 0}, upload=436819]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:10:58.082487081 (497h41m57.953570116s) [upload=436819]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:10:58.082495227 (497h41m57.953578272s) [upload=436819]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:10:58.378274491 (497h41m58.249357536s) [upload=436819]: Done uploading segments... Dec 26 09:10:58.378296644 (497h41m58.249379689s) [upload=436819]: Upload finished. err: <nil> Duration: 5.969168001s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 17.70377355s trace: https://jaeger.dev.storj.io/trace/6820217d73ecf2a FAIL (2025-12-26 09:11:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments... Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces. Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments... Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil> Duration: 17.703803278s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X.X...X.XX..X.XXX....................X....XXX..X.......X] duration: 30.001743193s trace: https://jaeger.dev.storj.io/trace/7e377ce9a9417865 FAIL (2025-12-26 09:11:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 10:54.879406022 (497h41m54.750489067s) [upload=436819]: Starting upload Dec 26 09:10:54.879541820 (497h41m54.750624865s) [upload=436819]: Done starting upload Dec 26 09:10:54.880024890 (497h41m54.751107935s) [upload=436819]: Uploading segments... Dec 26 09:10:54.880104391 (497h41m54.751187436s) [upload=436819]: Got next segment. Inline: false Dec 26 09:10:54.880146241 (497h41m54.751229276s) [upload=436819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:10:54.880151912 (497h41m54.751234947s) [upload=436819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:10:55.125750343 (497h41m54.996833378s) [seg_pos={0 0}, upload=436819]: Begin upload segment... Dec 26 09:10:55.125857047 (497h41m54.996940092s) [seg_pos={0 0}, upload=436819]: Done begin upload segment. Dec 26 09:10:55.125902423 (497h41m54.996985468s) [noise=true , address=145.79.227.40:4049 , piece_id=C47V2HF6RFGKAMFSOR3TE3KR7VRYJT354MLH7HCHVZGTQ2V74FUA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.125986493 (497h41m54.997069548s) [noise=true , address=45.91.116.7:4011 , piece_id=YRB6UO3BNQTAUU6MK62UZM4ZOGKOY7S6ZGUNKLKV4V5SONJL7CYQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.126687488 (497h41m54.997770523s) [noise=true , address=45.91.116.10:4083 , piece_id=IDYGUWJVXMZUGWL3THC5SWAH7GXBA7PSU7HX3U2GNPMGKXHCTNHA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.127155399 (497h41m54.998238444s) [noise=true , address=45.91.116.6:4974 , piece_id=DHWCO3O4E2TAWPEK3S5MO72AJO7R3XAQHOEBRI3NDTYDBTANK7MA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.127488613 (497h41m54.998571658s) [noise=true , address=145.79.227.45:4231 , piece_id=GQ2VZPUN2CLSVVSCZPZIAHMAPSWEYJ73S2HJB6LJWHCTV3LQGRYQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.128064139 (497h41m54.999147185s) [noise=true , address=145.79.227.46:4603 , piece_id=DQQYF2F4YHM7N6HLNHE5IETLSZEVVPWHODK5T3DGSVNS4AIOIUOA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.128505691 (497h41m54.999588736s) [noise=true , address=145.79.227.15:4875 , piece_id=RH7BSRUX6AHU6LK4IQQ7WYEWEEE3CC5EJQPHJAVGMM4D5XSZXS2Q, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.129048043 (497h41m55.000131078s) [noise=true , address=145.79.227.48:4302 , piece_id=CFD2SPCLIPJ6BMN55BZJKQSI5E7CKQWT36AT6M4RLDZWMKY6GIHQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.129593071 (497h41m55.000676116s) [noise=true , address=45.91.116.5:4299 , piece_id=UPPBKRTUTELYILKUOCI722ZTEFW6ABBQ3UC73FZMP3O4ZVS26NJA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.130183575 (497h41m55.00126662s) [noise=true , address=45.91.116.4:4939 , piece_id=D4JHH7YQIMDLXPNGYGWBURUEA6F3UE56UX3SSVGSYJAGCFQNAYEA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.130610648 (497h41m55.001693683s) [noise=true , address=145.79.227.39:4013 , piece_id=WI6ZYL643STKYE2AL2K4SO3F3RQVDHHLNAGMN4IAGPZE4FXPLRDQ, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.130718774 (497h41m55.001801809s) [noise=true , address=45.91.116.12:4227 , piece_id=DLGWVCIBJSUDKMUGPUQLP4R7H5ICU2XLM5RXD5LWPJSYQJHQP4EA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.131075473 (497h41m55.002158508s) [noise=true , address=145.79.227.13:4804 , piece_id=H6OHBG75MUVTHOQXHK5QGDMYRLN5Q2HLTGZCP4RQF6KA7QFX5JZA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.131473952 (497h41m55.002556987s) [noise=true , address=45.91.116.9:4261 , piece_id=76BFSEUKR6OZ23M3GI5KMWIKBLLS65AYL6BHXO5ZTOZWPZWHF3SA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.131833537 (497h41m55.002916572s) [noise=true , address=145.79.227.16:4909 , piece_id=I62QZXZXMRTIP4N6ZDDUEIQTHNWNFTQHQ2CALTYLAMUTL4HZ46BA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:55.132165620 (497h41m55.003248685s) [noise=true , address=77.90.31.17:4773 , piece_id=SM3WUMEECGVEEUJCDPGRS66WUYCZZ6PGNEZXD3PUB7RHRDGX53YA, seg_pos={0 0}, upload=436819]: Uploading piece... Dec 26 09:10:56.837614629 (497h41m56.708697674s) [upload=436819]: Next returned nil segment Dec 26 09:10:56.837655677 (497h41m56.708738722s) [upload=436819]: Waiting for error group managing segments... Dec 26 09:10:57.120464374 (497h41m56.991547419s) [noise=true , address=145.79.227.16:4909 , piece_id=I62QZXZXMRTIP4N6ZDDUEIQTHNWNFTQHQ2CALTYLAMUTL4HZ46BA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.131683237 (497h41m57.002766282s) [noise=true , address=145.79.227.15:4875 , piece_id=RH7BSRUX6AHU6LK4IQQ7WYEWEEE3CC5EJQPHJAVGMM4D5XSZXS2Q, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.132496956 (497h41m57.003579991s) [noise=true , address=145.79.227.13:4804 , piece_id=H6OHBG75MUVTHOQXHK5QGDMYRLN5Q2HLTGZCP4RQF6KA7QFX5JZA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.135213488 (497h41m57.006296523s) [noise=true , address=145.79.227.48:4302 , piece_id=CFD2SPCLIPJ6BMN55BZJKQSI5E7CKQWT36AT6M4RLDZWMKY6GIHQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.139782589 (497h41m57.010865624s) [noise=true , address=145.79.227.39:4013 , piece_id=WI6ZYL643STKYE2AL2K4SO3F3RQVDHHLNAGMN4IAGPZE4FXPLRDQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.153979191 (497h41m57.025062227s) [noise=true , address=145.79.227.46:4603 , piece_id=DQQYF2F4YHM7N6HLNHE5IETLSZEVVPWHODK5T3DGSVNS4AIOIUOA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.160470912 (497h41m57.031553947s) [noise=true , address=145.79.227.45:4231 , piece_id=GQ2VZPUN2CLSVVSCZPZIAHMAPSWEYJ73S2HJB6LJWHCTV3LQGRYQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.395125116 (497h41m57.266208161s) [noise=true , address=77.90.31.17:4773 , piece_id=SM3WUMEECGVEEUJCDPGRS66WUYCZZ6PGNEZXD3PUB7RHRDGX53YA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.445018932 (497h41m57.316101977s) [noise=true , address=45.91.116.6:4974 , piece_id=DHWCO3O4E2TAWPEK3S5MO72AJO7R3XAQHOEBRI3NDTYDBTANK7MA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.469099114 (497h41m57.340182159s) [noise=true , address=45.91.116.5:4299 , piece_id=UPPBKRTUTELYILKUOCI722ZTEFW6ABBQ3UC73FZMP3O4ZVS26NJA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.470093207 (497h41m57.341176252s) [noise=true , address=45.91.116.9:4261 , piece_id=76BFSEUKR6OZ23M3GI5KMWIKBLLS65AYL6BHXO5ZTOZWPZWHF3SA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:57.504656261 (497h41m57.375739306s) [noise=true , address=45.91.116.7:4011 , piece_id=YRB6UO3BNQTAUU6MK62UZM4ZOGKOY7S6ZGUNKLKV4V5SONJL7CYQ, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:58.081565226 (497h41m57.952648271s) [noise=true , address=45.91.116.4:4939 , piece_id=D4JHH7YQIMDLXPNGYGWBURUEA6F3UE56UX3SSVGSYJAGCFQNAYEA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: <nil> Dec 26 09:10:58.081589312 (497h41m57.952672347s) [seg_pos={0 0}, upload=436819]: Segment reached optimal threshold of 13 pieces. Dec 26 09:10:58.081930351 (497h41m57.953013396s) [noise=true , address=45.91.116.10:4083 , piece_id=IDYGUWJVXMZUGWL3THC5SWAH7GXBA7PSU7HX3U2GNPMGKXHCTNHA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:49172->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:49172->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49172->45.91.116.10:4083: use of closed network connection Dec 26 09:10:58.081940210 (497h41m57.953023245s) [noise=true , address=45.91.116.10:4083 , piece_id=IDYGUWJVXMZUGWL3THC5SWAH7GXBA7PSU7HX3U2GNPMGKXHCTNHA, seg_pos={0 0}, upload=436819]: Long tail cancellation due to optimal threshold Dec 26 09:10:58.082018820 (497h41m57.953101855s) [noise=true , address=45.91.116.12:4227 , piece_id=DLGWVCIBJSUDKMUGPUQLP4R7H5ICU2XLM5RXD5LWPJSYQJHQP4EA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 26 09:10:58.082038857 (497h41m57.953121902s) [noise=true , address=45.91.116.12:4227 , piece_id=DLGWVCIBJSUDKMUGPUQLP4R7H5ICU2XLM5RXD5LWPJSYQJHQP4EA, seg_pos={0 0}, upload=436819]: Long tail cancellation due to optimal threshold Dec 26 09:10:58.082351643 (497h41m57.953434698s) [noise=true , address=145.79.227.40:4049 , piece_id=C47V2HF6RFGKAMFSOR3TE3KR7VRYJT354MLH7HCHVZGTQ2V74FUA, seg_pos={0 0}, upload=436819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 09:10:58.082360631 (497h41m57.953443666s) [noise=true , address=145.79.227.40:4049 , piece_id=C47V2HF6RFGKAMFSOR3TE3KR7VRYJT354MLH7HCHVZGTQ2V74FUA, seg_pos={0 0}, upload=436819]: Long tail cancellation due to optimal threshold Dec 26 09:10:58.082387421 (497h41m57.953470456s) [seg_pos={0 0}, upload=436819]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:10:58.082487081 (497h41m57.953570116s) [upload=436819]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:10:58.082495227 (497h41m57.953578272s) [upload=436819]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:10:58.378274491 (497h41m58.249357536s) [upload=436819]: Done uploading segments... Dec 26 09:10:58.378296644 (497h41m58.249379689s) [upload=436819]: Upload finished. err: <nil> Duration: 30.001764113s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 36.811697126s trace: https://jaeger.dev.storj.io/trace/6820217d73ecf2a FAIL (2025-12-26 09:11:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments... Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces. Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments... Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil> Duration: 36.811727525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 33.463412934s trace: https://jaeger.dev.storj.io/trace/6820217d73ecf2a FAIL (2025-12-26 09:10:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments... Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces. Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments... Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil> Duration: 33.463441698s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.....X.X...X.XX..X.XXX....................X....XXX..X.......] duration: 6.082577843s trace: https://jaeger.dev.storj.io/trace/1bc7698723cdc676 FAIL (2025-12-26 09:10:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 09:05:20.203562675 (497h36m20.07464572s) [upload=436739]: Starting upload Dec 26 09:05:20.203657285 (497h36m20.07474032s) [upload=436739]: Done starting upload Dec 26 09:05:20.203740263 (497h36m20.074823308s) [upload=436739]: Uploading segments... Dec 26 09:05:20.203881842 (497h36m20.074964877s) [upload=436739]: Got next segment. Inline: false Dec 26 09:05:20.203915156 (497h36m20.074998201s) [upload=436739]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:05:20.203919946 (497h36m20.075002991s) [upload=436739]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:05:20.451528164 (497h36m20.322611209s) [seg_pos={0 0}, upload=436739]: Begin upload segment... Dec 26 09:05:20.451579712 (497h36m20.322687094s) [seg_pos={0 0}, upload=436739]: Done begin upload segment. Dec 26 09:05:20.451681246 (497h36m20.322764281s) [noise=true , address=145.79.227.45:4230 , piece_id=F6V27RD3FSPA5VEHR3XWQSN65OACRMLQ7BKEKOV433K4UK57F6JA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.451705141 (497h36m20.322788176s) [noise=true , address=45.91.116.11:4117 , piece_id=6P623RZ47UIK4UCUBWRHED7RC45GPU6UDL5OKPA43E4364CP3YAQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.452346592 (497h36m20.323429637s) [noise=true , address=145.79.227.49:4338 , piece_id=F4TGKH3AMO7BXVRCDQ4OTLTJAJZB4LHCF2FQASPUJO2G442B7ENA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.452878154 (497h36m20.323961199s) [noise=true , address=145.79.227.46:4603 , piece_id=7QYHIKK22SFK6Z245HTN6ANBL2LJQ2MGBUZXPA4SAIMO5UYG4BNQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.453292944 (497h36m20.324375989s) [noise=true , address=145.79.227.42:4122 , piece_id=MUW743I4E66WWNEANZPPAJQHHP7F245LYOJK5ELDV3MYOPJLIJKQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.453598226 (497h36m20.324681271s) [noise=true , address=145.79.227.37:4565 , piece_id=RYFRLEO4NB5YV4Y3AS6VRB5VRH5FBHEJEGN6IEMSROTHUWQDWNGQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.453911482 (497h36m20.324994527s) [noise=true , address=45.91.116.8:4047 , piece_id=GZZZACVBAPUKOPYFGYXMXRZQJCK4UYSYOGF74PZS4GFALAFOZOOA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.454291346 (497h36m20.325374381s) [noise=true , address=145.79.227.35:4483 , piece_id=XY3WYAZZFYPAE6OORXCHVGLIXF74PK4YK6L2GU2BEV4NHSZYRPLQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.454619029 (497h36m20.325702064s) [noise=true , address=145.79.227.48:4303 , piece_id=I5C36PABM3IO7Z7LNTVFZGY2XRBQZXS3A7JAXYQ3GIYV6RQ3WKTQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.455510106 (497h36m20.326593151s) [noise=true , address=145.79.227.3:4149 , piece_id=EHGSRXFW6G2D5SH7AAHSGOYMWPJW45KIXPSCGY726CL5V7MAWIWQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.456091063 (497h36m20.327174108s) [noise=true , address=45.91.116.12:4225 , piece_id=J7R5WRDM6HJD2W5K5URZBWVIBL2MP2GOU7ZZ7TXBS3LN4ZHJ42UA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.456776608 (497h36m20.327859643s) [noise=true , address=145.79.227.13:4804 , piece_id=CVUQPTEPXVZULIREN377ONZIQ665NMGBEQCEOUEVZHTEIVGUVRLQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.457207779 (497h36m20.328290824s) [noise=true , address=45.91.116.6:4974 , piece_id=H76F5IVAIO7Q2GDIINQGO25XNROXWNPZ7RSJ242YZPENAZ6DRKLA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.458453440 (497h36m20.329536476s) [noise=true , address=45.91.116.5:4299 , piece_id=FN5HENWXRAUGQZ3DLCSTZJFMICEIECOS4WXT7ZYS25IA46YYXLEA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.463328213 (497h36m20.334411258s) [noise=true , address=145.79.227.50:4375 , piece_id=F2T7S2AXBCZVCSIBV673JB65PTH2KOJ3BGPBT6DHUEM76QDYCHAQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.465622081 (497h36m20.336705126s) [noise=true , address=145.79.227.19:4078 , piece_id=LJ5RXJFEABFIIK7TV5BHHWIA7CDQHPDBRCRXHMN2CVP65ZP3YPCA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:22.163012587 (497h36m22.034095632s) [upload=436739]: Next returned nil segment Dec 26 09:05:22.163041332 (497h36m22.034124377s) [upload=436739]: Waiting for error group managing segments... Dec 26 09:05:22.453559579 (497h36m22.324642624s) [noise=true , address=145.79.227.35:4483 , piece_id=XY3WYAZZFYPAE6OORXCHVGLIXF74PK4YK6L2GU2BEV4NHSZYRPLQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.459896496 (497h36m22.330979531s) [noise=true , address=145.79.227.46:4603 , piece_id=7QYHIKK22SFK6Z245HTN6ANBL2LJQ2MGBUZXPA4SAIMO5UYG4BNQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.464918438 (497h36m22.336001483s) [noise=true , address=145.79.227.49:4338 , piece_id=F4TGKH3AMO7BXVRCDQ4OTLTJAJZB4LHCF2FQASPUJO2G442B7ENA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.466566056 (497h36m22.337649091s) [noise=true , address=145.79.227.37:4565 , piece_id=RYFRLEO4NB5YV4Y3AS6VRB5VRH5FBHEJEGN6IEMSROTHUWQDWNGQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.483914400 (497h36m22.354997435s) [noise=true , address=145.79.227.3:4149 , piece_id=EHGSRXFW6G2D5SH7AAHSGOYMWPJW45KIXPSCGY726CL5V7MAWIWQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.484448046 (497h36m22.355531081s) [noise=true , address=145.79.227.42:4122 , piece_id=MUW743I4E66WWNEANZPPAJQHHP7F245LYOJK5ELDV3MYOPJLIJKQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.485048360 (497h36m22.356131395s) [noise=true , address=145.79.227.48:4303 , piece_id=I5C36PABM3IO7Z7LNTVFZGY2XRBQZXS3A7JAXYQ3GIYV6RQ3WKTQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.485844586 (497h36m22.356927621s) [noise=true , address=145.79.227.13:4804 , piece_id=CVUQPTEPXVZULIREN377ONZIQ665NMGBEQCEOUEVZHTEIVGUVRLQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.491593703 (497h36m22.362676748s) [noise=true , address=145.79.227.45:4230 , piece_id=F6V27RD3FSPA5VEHR3XWQSN65OACRMLQ7BKEKOV433K4UK57F6JA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.494334231 (497h36m22.365417266s) [noise=true , address=145.79.227.19:4078 , piece_id=LJ5RXJFEABFIIK7TV5BHHWIA7CDQHPDBRCRXHMN2CVP65ZP3YPCA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.497173928 (497h36m22.368256973s) [noise=true , address=145.79.227.50:4375 , piece_id=F2T7S2AXBCZVCSIBV673JB65PTH2KOJ3BGPBT6DHUEM76QDYCHAQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.826633215 (497h36m22.697716261s) [noise=true , address=45.91.116.12:4225 , piece_id=J7R5WRDM6HJD2W5K5URZBWVIBL2MP2GOU7ZZ7TXBS3LN4ZHJ42UA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.835764464 (497h36m22.706847499s) [noise=true , address=45.91.116.11:4117 , piece_id=6P623RZ47UIK4UCUBWRHED7RC45GPU6UDL5OKPA43E4364CP3YAQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.835908578 (497h36m22.706991623s) [seg_pos={0 0}, upload=436739]: Segment reached optimal threshold of 13 pieces. Dec 26 09:05:22.836284153 (497h36m22.707367188s) [noise=true , address=45.91.116.5:4299 , piece_id=FN5HENWXRAUGQZ3DLCSTZJFMICEIECOS4WXT7ZYS25IA46YYXLEA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 09:05:22.836295585 (497h36m22.70737862s) [noise=true , address=45.91.116.5:4299 , piece_id=FN5HENWXRAUGQZ3DLCSTZJFMICEIECOS4WXT7ZYS25IA46YYXLEA, seg_pos={0 0}, upload=436739]: Long tail cancellation due to optimal threshold Dec 26 09:05:22.836830483 (497h36m22.707913528s) [noise=true , address=45.91.116.6:4974 , piece_id=H76F5IVAIO7Q2GDIINQGO25XNROXWNPZ7RSJ242YZPENAZ6DRKLA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:49846->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:49846->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->45.91.116.6:4974: use of closed network connection Dec 26 09:05:22.836840893 (497h36m22.707923938s) [noise=true , address=45.91.116.6:4974 , piece_id=H76F5IVAIO7Q2GDIINQGO25XNROXWNPZ7RSJ242YZPENAZ6DRKLA, seg_pos={0 0}, upload=436739]: Long tail cancellation due to optimal threshold Dec 26 09:05:22.837148850 (497h36m22.708231885s) [noise=true , address=45.91.116.8:4047 , piece_id=GZZZACVBAPUKOPYFGYXMXRZQJCK4UYSYOGF74PZS4GFALAFOZOOA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 09:05:22.837159640 (497h36m22.708242675s) [noise=true , address=45.91.116.8:4047 , piece_id=GZZZACVBAPUKOPYFGYXMXRZQJCK4UYSYOGF74PZS4GFALAFOZOOA, seg_pos={0 0}, upload=436739]: Long tail cancellation due to optimal threshold Dec 26 09:05:22.837183636 (497h36m22.708266681s) [seg_pos={0 0}, upload=436739]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:05:22.837302172 (497h36m22.708385207s) [upload=436739]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:05:22.837309886 (497h36m22.708392931s) [upload=436739]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:05:23.143746675 (497h36m23.01482972s) [upload=436739]: Done uploading segments... Dec 26 09:05:23.143766713 (497h36m23.014849748s) [upload=436739]: Upload finished. err: <nil> Duration: 6.082709345s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [......X.X...X.XX..X.XXX....................X....XXX..X......] duration: 5.695545817s trace: https://jaeger.dev.storj.io/trace/1bc7698723cdc676 FAIL (2025-12-26 09:09:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 09:05:20.203562675 (497h36m20.07464572s) [upload=436739]: Starting upload Dec 26 09:05:20.203657285 (497h36m20.07474032s) [upload=436739]: Done starting upload Dec 26 09:05:20.203740263 (497h36m20.074823308s) [upload=436739]: Uploading segments... Dec 26 09:05:20.203881842 (497h36m20.074964877s) [upload=436739]: Got next segment. Inline: false Dec 26 09:05:20.203915156 (497h36m20.074998201s) [upload=436739]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:05:20.203919946 (497h36m20.075002991s) [upload=436739]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:05:20.451528164 (497h36m20.322611209s) [seg_pos={0 0}, upload=436739]: Begin upload segment... Dec 26 09:05:20.451579712 (497h36m20.322687094s) [seg_pos={0 0}, upload=436739]: Done begin upload segment. Dec 26 09:05:20.451681246 (497h36m20.322764281s) [noise=true , address=145.79.227.45:4230 , piece_id=F6V27RD3FSPA5VEHR3XWQSN65OACRMLQ7BKEKOV433K4UK57F6JA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.451705141 (497h36m20.322788176s) [noise=true , address=45.91.116.11:4117 , piece_id=6P623RZ47UIK4UCUBWRHED7RC45GPU6UDL5OKPA43E4364CP3YAQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.452346592 (497h36m20.323429637s) [noise=true , address=145.79.227.49:4338 , piece_id=F4TGKH3AMO7BXVRCDQ4OTLTJAJZB4LHCF2FQASPUJO2G442B7ENA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.452878154 (497h36m20.323961199s) [noise=true , address=145.79.227.46:4603 , piece_id=7QYHIKK22SFK6Z245HTN6ANBL2LJQ2MGBUZXPA4SAIMO5UYG4BNQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.453292944 (497h36m20.324375989s) [noise=true , address=145.79.227.42:4122 , piece_id=MUW743I4E66WWNEANZPPAJQHHP7F245LYOJK5ELDV3MYOPJLIJKQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.453598226 (497h36m20.324681271s) [noise=true , address=145.79.227.37:4565 , piece_id=RYFRLEO4NB5YV4Y3AS6VRB5VRH5FBHEJEGN6IEMSROTHUWQDWNGQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.453911482 (497h36m20.324994527s) [noise=true , address=45.91.116.8:4047 , piece_id=GZZZACVBAPUKOPYFGYXMXRZQJCK4UYSYOGF74PZS4GFALAFOZOOA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.454291346 (497h36m20.325374381s) [noise=true , address=145.79.227.35:4483 , piece_id=XY3WYAZZFYPAE6OORXCHVGLIXF74PK4YK6L2GU2BEV4NHSZYRPLQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.454619029 (497h36m20.325702064s) [noise=true , address=145.79.227.48:4303 , piece_id=I5C36PABM3IO7Z7LNTVFZGY2XRBQZXS3A7JAXYQ3GIYV6RQ3WKTQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.455510106 (497h36m20.326593151s) [noise=true , address=145.79.227.3:4149 , piece_id=EHGSRXFW6G2D5SH7AAHSGOYMWPJW45KIXPSCGY726CL5V7MAWIWQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.456091063 (497h36m20.327174108s) [noise=true , address=45.91.116.12:4225 , piece_id=J7R5WRDM6HJD2W5K5URZBWVIBL2MP2GOU7ZZ7TXBS3LN4ZHJ42UA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.456776608 (497h36m20.327859643s) [noise=true , address=145.79.227.13:4804 , piece_id=CVUQPTEPXVZULIREN377ONZIQ665NMGBEQCEOUEVZHTEIVGUVRLQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.457207779 (497h36m20.328290824s) [noise=true , address=45.91.116.6:4974 , piece_id=H76F5IVAIO7Q2GDIINQGO25XNROXWNPZ7RSJ242YZPENAZ6DRKLA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.458453440 (497h36m20.329536476s) [noise=true , address=45.91.116.5:4299 , piece_id=FN5HENWXRAUGQZ3DLCSTZJFMICEIECOS4WXT7ZYS25IA46YYXLEA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.463328213 (497h36m20.334411258s) [noise=true , address=145.79.227.50:4375 , piece_id=F2T7S2AXBCZVCSIBV673JB65PTH2KOJ3BGPBT6DHUEM76QDYCHAQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.465622081 (497h36m20.336705126s) [noise=true , address=145.79.227.19:4078 , piece_id=LJ5RXJFEABFIIK7TV5BHHWIA7CDQHPDBRCRXHMN2CVP65ZP3YPCA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:22.163012587 (497h36m22.034095632s) [upload=436739]: Next returned nil segment Dec 26 09:05:22.163041332 (497h36m22.034124377s) [upload=436739]: Waiting for error group managing segments... Dec 26 09:05:22.453559579 (497h36m22.324642624s) [noise=true , address=145.79.227.35:4483 , piece_id=XY3WYAZZFYPAE6OORXCHVGLIXF74PK4YK6L2GU2BEV4NHSZYRPLQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.459896496 (497h36m22.330979531s) [noise=true , address=145.79.227.46:4603 , piece_id=7QYHIKK22SFK6Z245HTN6ANBL2LJQ2MGBUZXPA4SAIMO5UYG4BNQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.464918438 (497h36m22.336001483s) [noise=true , address=145.79.227.49:4338 , piece_id=F4TGKH3AMO7BXVRCDQ4OTLTJAJZB4LHCF2FQASPUJO2G442B7ENA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.466566056 (497h36m22.337649091s) [noise=true , address=145.79.227.37:4565 , piece_id=RYFRLEO4NB5YV4Y3AS6VRB5VRH5FBHEJEGN6IEMSROTHUWQDWNGQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.483914400 (497h36m22.354997435s) [noise=true , address=145.79.227.3:4149 , piece_id=EHGSRXFW6G2D5SH7AAHSGOYMWPJW45KIXPSCGY726CL5V7MAWIWQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.484448046 (497h36m22.355531081s) [noise=true , address=145.79.227.42:4122 , piece_id=MUW743I4E66WWNEANZPPAJQHHP7F245LYOJK5ELDV3MYOPJLIJKQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.485048360 (497h36m22.356131395s) [noise=true , address=145.79.227.48:4303 , piece_id=I5C36PABM3IO7Z7LNTVFZGY2XRBQZXS3A7JAXYQ3GIYV6RQ3WKTQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.485844586 (497h36m22.356927621s) [noise=true , address=145.79.227.13:4804 , piece_id=CVUQPTEPXVZULIREN377ONZIQ665NMGBEQCEOUEVZHTEIVGUVRLQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.491593703 (497h36m22.362676748s) [noise=true , address=145.79.227.45:4230 , piece_id=F6V27RD3FSPA5VEHR3XWQSN65OACRMLQ7BKEKOV433K4UK57F6JA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.494334231 (497h36m22.365417266s) [noise=true , address=145.79.227.19:4078 , piece_id=LJ5RXJFEABFIIK7TV5BHHWIA7CDQHPDBRCRXHMN2CVP65ZP3YPCA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.497173928 (497h36m22.368256973s) [noise=true , address=145.79.227.50:4375 , piece_id=F2T7S2AXBCZVCSIBV673JB65PTH2KOJ3BGPBT6DHUEM76QDYCHAQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.826633215 (497h36m22.697716261s) [noise=true , address=45.91.116.12:4225 , piece_id=J7R5WRDM6HJD2W5K5URZBWVIBL2MP2GOU7ZZ7TXBS3LN4ZHJ42UA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.835764464 (497h36m22.706847499s) [noise=true , address=45.91.116.11:4117 , piece_id=6P623RZ47UIK4UCUBWRHED7RC45GPU6UDL5OKPA43E4364CP3YAQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.835908578 (497h36m22.706991623s) [seg_pos={0 0}, upload=436739]: Segment reached optimal threshold of 13 pieces. Dec 26 09:05:22.836284153 (497h36m22.707367188s) [noise=true , address=45.91.116.5:4299 , piece_id=FN5HENWXRAUGQZ3DLCSTZJFMICEIECOS4WXT7ZYS25IA46YYXLEA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 09:05:22.836295585 (497h36m22.70737862s) [noise=true , address=45.91.116.5:4299 , piece_id=FN5HENWXRAUGQZ3DLCSTZJFMICEIECOS4WXT7ZYS25IA46YYXLEA, seg_pos={0 0}, upload=436739]: Long tail cancellation due to optimal threshold Dec 26 09:05:22.836830483 (497h36m22.707913528s) [noise=true , address=45.91.116.6:4974 , piece_id=H76F5IVAIO7Q2GDIINQGO25XNROXWNPZ7RSJ242YZPENAZ6DRKLA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:49846->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:49846->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->45.91.116.6:4974: use of closed network connection Dec 26 09:05:22.836840893 (497h36m22.707923938s) [noise=true , address=45.91.116.6:4974 , piece_id=H76F5IVAIO7Q2GDIINQGO25XNROXWNPZ7RSJ242YZPENAZ6DRKLA, seg_pos={0 0}, upload=436739]: Long tail cancellation due to optimal threshold Dec 26 09:05:22.837148850 (497h36m22.708231885s) [noise=true , address=45.91.116.8:4047 , piece_id=GZZZACVBAPUKOPYFGYXMXRZQJCK4UYSYOGF74PZS4GFALAFOZOOA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 09:05:22.837159640 (497h36m22.708242675s) [noise=true , address=45.91.116.8:4047 , piece_id=GZZZACVBAPUKOPYFGYXMXRZQJCK4UYSYOGF74PZS4GFALAFOZOOA, seg_pos={0 0}, upload=436739]: Long tail cancellation due to optimal threshold Dec 26 09:05:22.837183636 (497h36m22.708266681s) [seg_pos={0 0}, upload=436739]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:05:22.837302172 (497h36m22.708385207s) [upload=436739]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:05:22.837309886 (497h36m22.708392931s) [upload=436739]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:05:23.143746675 (497h36m23.01482972s) [upload=436739]: Done uploading segments... Dec 26 09:05:23.143766713 (497h36m23.014849748s) [upload=436739]: Upload finished. err: <nil> Duration: 5.695584711s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 44.837275331s trace: https://jaeger.dev.storj.io/trace/6820217d73ecf2a FAIL (2025-12-26 09:09:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments... Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces. Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments... Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil> Duration: 44.837302322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X......X.X...X.XX..X.XXX....................X....XXX..X.....] duration: 5.765066928s trace: https://jaeger.dev.storj.io/trace/1bc7698723cdc676 FAIL (2025-12-26 09:08:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 09:05:20.203562675 (497h36m20.07464572s) [upload=436739]: Starting upload Dec 26 09:05:20.203657285 (497h36m20.07474032s) [upload=436739]: Done starting upload Dec 26 09:05:20.203740263 (497h36m20.074823308s) [upload=436739]: Uploading segments... Dec 26 09:05:20.203881842 (497h36m20.074964877s) [upload=436739]: Got next segment. Inline: false Dec 26 09:05:20.203915156 (497h36m20.074998201s) [upload=436739]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:05:20.203919946 (497h36m20.075002991s) [upload=436739]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:05:20.451528164 (497h36m20.322611209s) [seg_pos={0 0}, upload=436739]: Begin upload segment... Dec 26 09:05:20.451579712 (497h36m20.322687094s) [seg_pos={0 0}, upload=436739]: Done begin upload segment. Dec 26 09:05:20.451681246 (497h36m20.322764281s) [noise=true , address=145.79.227.45:4230 , piece_id=F6V27RD3FSPA5VEHR3XWQSN65OACRMLQ7BKEKOV433K4UK57F6JA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.451705141 (497h36m20.322788176s) [noise=true , address=45.91.116.11:4117 , piece_id=6P623RZ47UIK4UCUBWRHED7RC45GPU6UDL5OKPA43E4364CP3YAQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.452346592 (497h36m20.323429637s) [noise=true , address=145.79.227.49:4338 , piece_id=F4TGKH3AMO7BXVRCDQ4OTLTJAJZB4LHCF2FQASPUJO2G442B7ENA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.452878154 (497h36m20.323961199s) [noise=true , address=145.79.227.46:4603 , piece_id=7QYHIKK22SFK6Z245HTN6ANBL2LJQ2MGBUZXPA4SAIMO5UYG4BNQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.453292944 (497h36m20.324375989s) [noise=true , address=145.79.227.42:4122 , piece_id=MUW743I4E66WWNEANZPPAJQHHP7F245LYOJK5ELDV3MYOPJLIJKQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.453598226 (497h36m20.324681271s) [noise=true , address=145.79.227.37:4565 , piece_id=RYFRLEO4NB5YV4Y3AS6VRB5VRH5FBHEJEGN6IEMSROTHUWQDWNGQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.453911482 (497h36m20.324994527s) [noise=true , address=45.91.116.8:4047 , piece_id=GZZZACVBAPUKOPYFGYXMXRZQJCK4UYSYOGF74PZS4GFALAFOZOOA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.454291346 (497h36m20.325374381s) [noise=true , address=145.79.227.35:4483 , piece_id=XY3WYAZZFYPAE6OORXCHVGLIXF74PK4YK6L2GU2BEV4NHSZYRPLQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.454619029 (497h36m20.325702064s) [noise=true , address=145.79.227.48:4303 , piece_id=I5C36PABM3IO7Z7LNTVFZGY2XRBQZXS3A7JAXYQ3GIYV6RQ3WKTQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.455510106 (497h36m20.326593151s) [noise=true , address=145.79.227.3:4149 , piece_id=EHGSRXFW6G2D5SH7AAHSGOYMWPJW45KIXPSCGY726CL5V7MAWIWQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.456091063 (497h36m20.327174108s) [noise=true , address=45.91.116.12:4225 , piece_id=J7R5WRDM6HJD2W5K5URZBWVIBL2MP2GOU7ZZ7TXBS3LN4ZHJ42UA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.456776608 (497h36m20.327859643s) [noise=true , address=145.79.227.13:4804 , piece_id=CVUQPTEPXVZULIREN377ONZIQ665NMGBEQCEOUEVZHTEIVGUVRLQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.457207779 (497h36m20.328290824s) [noise=true , address=45.91.116.6:4974 , piece_id=H76F5IVAIO7Q2GDIINQGO25XNROXWNPZ7RSJ242YZPENAZ6DRKLA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.458453440 (497h36m20.329536476s) [noise=true , address=45.91.116.5:4299 , piece_id=FN5HENWXRAUGQZ3DLCSTZJFMICEIECOS4WXT7ZYS25IA46YYXLEA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.463328213 (497h36m20.334411258s) [noise=true , address=145.79.227.50:4375 , piece_id=F2T7S2AXBCZVCSIBV673JB65PTH2KOJ3BGPBT6DHUEM76QDYCHAQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.465622081 (497h36m20.336705126s) [noise=true , address=145.79.227.19:4078 , piece_id=LJ5RXJFEABFIIK7TV5BHHWIA7CDQHPDBRCRXHMN2CVP65ZP3YPCA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:22.163012587 (497h36m22.034095632s) [upload=436739]: Next returned nil segment Dec 26 09:05:22.163041332 (497h36m22.034124377s) [upload=436739]: Waiting for error group managing segments... Dec 26 09:05:22.453559579 (497h36m22.324642624s) [noise=true , address=145.79.227.35:4483 , piece_id=XY3WYAZZFYPAE6OORXCHVGLIXF74PK4YK6L2GU2BEV4NHSZYRPLQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.459896496 (497h36m22.330979531s) [noise=true , address=145.79.227.46:4603 , piece_id=7QYHIKK22SFK6Z245HTN6ANBL2LJQ2MGBUZXPA4SAIMO5UYG4BNQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.464918438 (497h36m22.336001483s) [noise=true , address=145.79.227.49:4338 , piece_id=F4TGKH3AMO7BXVRCDQ4OTLTJAJZB4LHCF2FQASPUJO2G442B7ENA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.466566056 (497h36m22.337649091s) [noise=true , address=145.79.227.37:4565 , piece_id=RYFRLEO4NB5YV4Y3AS6VRB5VRH5FBHEJEGN6IEMSROTHUWQDWNGQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.483914400 (497h36m22.354997435s) [noise=true , address=145.79.227.3:4149 , piece_id=EHGSRXFW6G2D5SH7AAHSGOYMWPJW45KIXPSCGY726CL5V7MAWIWQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.484448046 (497h36m22.355531081s) [noise=true , address=145.79.227.42:4122 , piece_id=MUW743I4E66WWNEANZPPAJQHHP7F245LYOJK5ELDV3MYOPJLIJKQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.485048360 (497h36m22.356131395s) [noise=true , address=145.79.227.48:4303 , piece_id=I5C36PABM3IO7Z7LNTVFZGY2XRBQZXS3A7JAXYQ3GIYV6RQ3WKTQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.485844586 (497h36m22.356927621s) [noise=true , address=145.79.227.13:4804 , piece_id=CVUQPTEPXVZULIREN377ONZIQ665NMGBEQCEOUEVZHTEIVGUVRLQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.491593703 (497h36m22.362676748s) [noise=true , address=145.79.227.45:4230 , piece_id=F6V27RD3FSPA5VEHR3XWQSN65OACRMLQ7BKEKOV433K4UK57F6JA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.494334231 (497h36m22.365417266s) [noise=true , address=145.79.227.19:4078 , piece_id=LJ5RXJFEABFIIK7TV5BHHWIA7CDQHPDBRCRXHMN2CVP65ZP3YPCA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.497173928 (497h36m22.368256973s) [noise=true , address=145.79.227.50:4375 , piece_id=F2T7S2AXBCZVCSIBV673JB65PTH2KOJ3BGPBT6DHUEM76QDYCHAQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.826633215 (497h36m22.697716261s) [noise=true , address=45.91.116.12:4225 , piece_id=J7R5WRDM6HJD2W5K5URZBWVIBL2MP2GOU7ZZ7TXBS3LN4ZHJ42UA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.835764464 (497h36m22.706847499s) [noise=true , address=45.91.116.11:4117 , piece_id=6P623RZ47UIK4UCUBWRHED7RC45GPU6UDL5OKPA43E4364CP3YAQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.835908578 (497h36m22.706991623s) [seg_pos={0 0}, upload=436739]: Segment reached optimal threshold of 13 pieces. Dec 26 09:05:22.836284153 (497h36m22.707367188s) [noise=true , address=45.91.116.5:4299 , piece_id=FN5HENWXRAUGQZ3DLCSTZJFMICEIECOS4WXT7ZYS25IA46YYXLEA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 09:05:22.836295585 (497h36m22.70737862s) [noise=true , address=45.91.116.5:4299 , piece_id=FN5HENWXRAUGQZ3DLCSTZJFMICEIECOS4WXT7ZYS25IA46YYXLEA, seg_pos={0 0}, upload=436739]: Long tail cancellation due to optimal threshold Dec 26 09:05:22.836830483 (497h36m22.707913528s) [noise=true , address=45.91.116.6:4974 , piece_id=H76F5IVAIO7Q2GDIINQGO25XNROXWNPZ7RSJ242YZPENAZ6DRKLA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:49846->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:49846->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->45.91.116.6:4974: use of closed network connection Dec 26 09:05:22.836840893 (497h36m22.707923938s) [noise=true , address=45.91.116.6:4974 , piece_id=H76F5IVAIO7Q2GDIINQGO25XNROXWNPZ7RSJ242YZPENAZ6DRKLA, seg_pos={0 0}, upload=436739]: Long tail cancellation due to optimal threshold Dec 26 09:05:22.837148850 (497h36m22.708231885s) [noise=true , address=45.91.116.8:4047 , piece_id=GZZZACVBAPUKOPYFGYXMXRZQJCK4UYSYOGF74PZS4GFALAFOZOOA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 09:05:22.837159640 (497h36m22.708242675s) [noise=true , address=45.91.116.8:4047 , piece_id=GZZZACVBAPUKOPYFGYXMXRZQJCK4UYSYOGF74PZS4GFALAFOZOOA, seg_pos={0 0}, upload=436739]: Long tail cancellation due to optimal threshold Dec 26 09:05:22.837183636 (497h36m22.708266681s) [seg_pos={0 0}, upload=436739]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:05:22.837302172 (497h36m22.708385207s) [upload=436739]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:05:22.837309886 (497h36m22.708392931s) [upload=436739]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:05:23.143746675 (497h36m23.01482972s) [upload=436739]: Done uploading segments... Dec 26 09:05:23.143766713 (497h36m23.014849748s) [upload=436739]: Upload finished. err: <nil> Duration: 5.765097827s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X......X.X...X.XX..X.XXX....................X....XXX..X....] duration: 9.490250444s trace: https://jaeger.dev.storj.io/trace/1bc7698723cdc676 FAIL (2025-12-26 09:08:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 09:05:20.203562675 (497h36m20.07464572s) [upload=436739]: Starting upload Dec 26 09:05:20.203657285 (497h36m20.07474032s) [upload=436739]: Done starting upload Dec 26 09:05:20.203740263 (497h36m20.074823308s) [upload=436739]: Uploading segments... Dec 26 09:05:20.203881842 (497h36m20.074964877s) [upload=436739]: Got next segment. Inline: false Dec 26 09:05:20.203915156 (497h36m20.074998201s) [upload=436739]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:05:20.203919946 (497h36m20.075002991s) [upload=436739]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:05:20.451528164 (497h36m20.322611209s) [seg_pos={0 0}, upload=436739]: Begin upload segment... Dec 26 09:05:20.451579712 (497h36m20.322687094s) [seg_pos={0 0}, upload=436739]: Done begin upload segment. Dec 26 09:05:20.451681246 (497h36m20.322764281s) [noise=true , address=145.79.227.45:4230 , piece_id=F6V27RD3FSPA5VEHR3XWQSN65OACRMLQ7BKEKOV433K4UK57F6JA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.451705141 (497h36m20.322788176s) [noise=true , address=45.91.116.11:4117 , piece_id=6P623RZ47UIK4UCUBWRHED7RC45GPU6UDL5OKPA43E4364CP3YAQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.452346592 (497h36m20.323429637s) [noise=true , address=145.79.227.49:4338 , piece_id=F4TGKH3AMO7BXVRCDQ4OTLTJAJZB4LHCF2FQASPUJO2G442B7ENA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.452878154 (497h36m20.323961199s) [noise=true , address=145.79.227.46:4603 , piece_id=7QYHIKK22SFK6Z245HTN6ANBL2LJQ2MGBUZXPA4SAIMO5UYG4BNQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.453292944 (497h36m20.324375989s) [noise=true , address=145.79.227.42:4122 , piece_id=MUW743I4E66WWNEANZPPAJQHHP7F245LYOJK5ELDV3MYOPJLIJKQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.453598226 (497h36m20.324681271s) [noise=true , address=145.79.227.37:4565 , piece_id=RYFRLEO4NB5YV4Y3AS6VRB5VRH5FBHEJEGN6IEMSROTHUWQDWNGQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.453911482 (497h36m20.324994527s) [noise=true , address=45.91.116.8:4047 , piece_id=GZZZACVBAPUKOPYFGYXMXRZQJCK4UYSYOGF74PZS4GFALAFOZOOA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.454291346 (497h36m20.325374381s) [noise=true , address=145.79.227.35:4483 , piece_id=XY3WYAZZFYPAE6OORXCHVGLIXF74PK4YK6L2GU2BEV4NHSZYRPLQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.454619029 (497h36m20.325702064s) [noise=true , address=145.79.227.48:4303 , piece_id=I5C36PABM3IO7Z7LNTVFZGY2XRBQZXS3A7JAXYQ3GIYV6RQ3WKTQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.455510106 (497h36m20.326593151s) [noise=true , address=145.79.227.3:4149 , piece_id=EHGSRXFW6G2D5SH7AAHSGOYMWPJW45KIXPSCGY726CL5V7MAWIWQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.456091063 (497h36m20.327174108s) [noise=true , address=45.91.116.12:4225 , piece_id=J7R5WRDM6HJD2W5K5URZBWVIBL2MP2GOU7ZZ7TXBS3LN4ZHJ42UA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.456776608 (497h36m20.327859643s) [noise=true , address=145.79.227.13:4804 , piece_id=CVUQPTEPXVZULIREN377ONZIQ665NMGBEQCEOUEVZHTEIVGUVRLQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.457207779 (497h36m20.328290824s) [noise=true , address=45.91.116.6:4974 , piece_id=H76F5IVAIO7Q2GDIINQGO25XNROXWNPZ7RSJ242YZPENAZ6DRKLA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.458453440 (497h36m20.329536476s) [noise=true , address=45.91.116.5:4299 , piece_id=FN5HENWXRAUGQZ3DLCSTZJFMICEIECOS4WXT7ZYS25IA46YYXLEA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.463328213 (497h36m20.334411258s) [noise=true , address=145.79.227.50:4375 , piece_id=F2T7S2AXBCZVCSIBV673JB65PTH2KOJ3BGPBT6DHUEM76QDYCHAQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.465622081 (497h36m20.336705126s) [noise=true , address=145.79.227.19:4078 , piece_id=LJ5RXJFEABFIIK7TV5BHHWIA7CDQHPDBRCRXHMN2CVP65ZP3YPCA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:22.163012587 (497h36m22.034095632s) [upload=436739]: Next returned nil segment Dec 26 09:05:22.163041332 (497h36m22.034124377s) [upload=436739]: Waiting for error group managing segments... Dec 26 09:05:22.453559579 (497h36m22.324642624s) [noise=true , address=145.79.227.35:4483 , piece_id=XY3WYAZZFYPAE6OORXCHVGLIXF74PK4YK6L2GU2BEV4NHSZYRPLQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.459896496 (497h36m22.330979531s) [noise=true , address=145.79.227.46:4603 , piece_id=7QYHIKK22SFK6Z245HTN6ANBL2LJQ2MGBUZXPA4SAIMO5UYG4BNQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.464918438 (497h36m22.336001483s) [noise=true , address=145.79.227.49:4338 , piece_id=F4TGKH3AMO7BXVRCDQ4OTLTJAJZB4LHCF2FQASPUJO2G442B7ENA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.466566056 (497h36m22.337649091s) [noise=true , address=145.79.227.37:4565 , piece_id=RYFRLEO4NB5YV4Y3AS6VRB5VRH5FBHEJEGN6IEMSROTHUWQDWNGQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.483914400 (497h36m22.354997435s) [noise=true , address=145.79.227.3:4149 , piece_id=EHGSRXFW6G2D5SH7AAHSGOYMWPJW45KIXPSCGY726CL5V7MAWIWQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.484448046 (497h36m22.355531081s) [noise=true , address=145.79.227.42:4122 , piece_id=MUW743I4E66WWNEANZPPAJQHHP7F245LYOJK5ELDV3MYOPJLIJKQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.485048360 (497h36m22.356131395s) [noise=true , address=145.79.227.48:4303 , piece_id=I5C36PABM3IO7Z7LNTVFZGY2XRBQZXS3A7JAXYQ3GIYV6RQ3WKTQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.485844586 (497h36m22.356927621s) [noise=true , address=145.79.227.13:4804 , piece_id=CVUQPTEPXVZULIREN377ONZIQ665NMGBEQCEOUEVZHTEIVGUVRLQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.491593703 (497h36m22.362676748s) [noise=true , address=145.79.227.45:4230 , piece_id=F6V27RD3FSPA5VEHR3XWQSN65OACRMLQ7BKEKOV433K4UK57F6JA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.494334231 (497h36m22.365417266s) [noise=true , address=145.79.227.19:4078 , piece_id=LJ5RXJFEABFIIK7TV5BHHWIA7CDQHPDBRCRXHMN2CVP65ZP3YPCA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.497173928 (497h36m22.368256973s) [noise=true , address=145.79.227.50:4375 , piece_id=F2T7S2AXBCZVCSIBV673JB65PTH2KOJ3BGPBT6DHUEM76QDYCHAQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.826633215 (497h36m22.697716261s) [noise=true , address=45.91.116.12:4225 , piece_id=J7R5WRDM6HJD2W5K5URZBWVIBL2MP2GOU7ZZ7TXBS3LN4ZHJ42UA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.835764464 (497h36m22.706847499s) [noise=true , address=45.91.116.11:4117 , piece_id=6P623RZ47UIK4UCUBWRHED7RC45GPU6UDL5OKPA43E4364CP3YAQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.835908578 (497h36m22.706991623s) [seg_pos={0 0}, upload=436739]: Segment reached optimal threshold of 13 pieces. Dec 26 09:05:22.836284153 (497h36m22.707367188s) [noise=true , address=45.91.116.5:4299 , piece_id=FN5HENWXRAUGQZ3DLCSTZJFMICEIECOS4WXT7ZYS25IA46YYXLEA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 09:05:22.836295585 (497h36m22.70737862s) [noise=true , address=45.91.116.5:4299 , piece_id=FN5HENWXRAUGQZ3DLCSTZJFMICEIECOS4WXT7ZYS25IA46YYXLEA, seg_pos={0 0}, upload=436739]: Long tail cancellation due to optimal threshold Dec 26 09:05:22.836830483 (497h36m22.707913528s) [noise=true , address=45.91.116.6:4974 , piece_id=H76F5IVAIO7Q2GDIINQGO25XNROXWNPZ7RSJ242YZPENAZ6DRKLA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:49846->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:49846->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->45.91.116.6:4974: use of closed network connection Dec 26 09:05:22.836840893 (497h36m22.707923938s) [noise=true , address=45.91.116.6:4974 , piece_id=H76F5IVAIO7Q2GDIINQGO25XNROXWNPZ7RSJ242YZPENAZ6DRKLA, seg_pos={0 0}, upload=436739]: Long tail cancellation due to optimal threshold Dec 26 09:05:22.837148850 (497h36m22.708231885s) [noise=true , address=45.91.116.8:4047 , piece_id=GZZZACVBAPUKOPYFGYXMXRZQJCK4UYSYOGF74PZS4GFALAFOZOOA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 09:05:22.837159640 (497h36m22.708242675s) [noise=true , address=45.91.116.8:4047 , piece_id=GZZZACVBAPUKOPYFGYXMXRZQJCK4UYSYOGF74PZS4GFALAFOZOOA, seg_pos={0 0}, upload=436739]: Long tail cancellation due to optimal threshold Dec 26 09:05:22.837183636 (497h36m22.708266681s) [seg_pos={0 0}, upload=436739]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:05:22.837302172 (497h36m22.708385207s) [upload=436739]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:05:22.837309886 (497h36m22.708392931s) [upload=436739]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:05:23.143746675 (497h36m23.01482972s) [upload=436739]: Done uploading segments... Dec 26 09:05:23.143766713 (497h36m23.014849748s) [upload=436739]: Upload finished. err: <nil> Duration: 9.490281943s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 24.63023018s trace: https://jaeger.dev.storj.io/trace/6820217d73ecf2a FAIL (2025-12-26 09:07:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments... Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces. Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments... Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil> Duration: 24.630257122s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X......X.X...X.XX..X.XXX....................X....XXX..X...] duration: 6.046214881s trace: https://jaeger.dev.storj.io/trace/1bc7698723cdc676 FAIL (2025-12-26 09:07:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 09:05:20.203562675 (497h36m20.07464572s) [upload=436739]: Starting upload Dec 26 09:05:20.203657285 (497h36m20.07474032s) [upload=436739]: Done starting upload Dec 26 09:05:20.203740263 (497h36m20.074823308s) [upload=436739]: Uploading segments... Dec 26 09:05:20.203881842 (497h36m20.074964877s) [upload=436739]: Got next segment. Inline: false Dec 26 09:05:20.203915156 (497h36m20.074998201s) [upload=436739]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:05:20.203919946 (497h36m20.075002991s) [upload=436739]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:05:20.451528164 (497h36m20.322611209s) [seg_pos={0 0}, upload=436739]: Begin upload segment... Dec 26 09:05:20.451579712 (497h36m20.322687094s) [seg_pos={0 0}, upload=436739]: Done begin upload segment. Dec 26 09:05:20.451681246 (497h36m20.322764281s) [noise=true , address=145.79.227.45:4230 , piece_id=F6V27RD3FSPA5VEHR3XWQSN65OACRMLQ7BKEKOV433K4UK57F6JA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.451705141 (497h36m20.322788176s) [noise=true , address=45.91.116.11:4117 , piece_id=6P623RZ47UIK4UCUBWRHED7RC45GPU6UDL5OKPA43E4364CP3YAQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.452346592 (497h36m20.323429637s) [noise=true , address=145.79.227.49:4338 , piece_id=F4TGKH3AMO7BXVRCDQ4OTLTJAJZB4LHCF2FQASPUJO2G442B7ENA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.452878154 (497h36m20.323961199s) [noise=true , address=145.79.227.46:4603 , piece_id=7QYHIKK22SFK6Z245HTN6ANBL2LJQ2MGBUZXPA4SAIMO5UYG4BNQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.453292944 (497h36m20.324375989s) [noise=true , address=145.79.227.42:4122 , piece_id=MUW743I4E66WWNEANZPPAJQHHP7F245LYOJK5ELDV3MYOPJLIJKQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.453598226 (497h36m20.324681271s) [noise=true , address=145.79.227.37:4565 , piece_id=RYFRLEO4NB5YV4Y3AS6VRB5VRH5FBHEJEGN6IEMSROTHUWQDWNGQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.453911482 (497h36m20.324994527s) [noise=true , address=45.91.116.8:4047 , piece_id=GZZZACVBAPUKOPYFGYXMXRZQJCK4UYSYOGF74PZS4GFALAFOZOOA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.454291346 (497h36m20.325374381s) [noise=true , address=145.79.227.35:4483 , piece_id=XY3WYAZZFYPAE6OORXCHVGLIXF74PK4YK6L2GU2BEV4NHSZYRPLQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.454619029 (497h36m20.325702064s) [noise=true , address=145.79.227.48:4303 , piece_id=I5C36PABM3IO7Z7LNTVFZGY2XRBQZXS3A7JAXYQ3GIYV6RQ3WKTQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.455510106 (497h36m20.326593151s) [noise=true , address=145.79.227.3:4149 , piece_id=EHGSRXFW6G2D5SH7AAHSGOYMWPJW45KIXPSCGY726CL5V7MAWIWQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.456091063 (497h36m20.327174108s) [noise=true , address=45.91.116.12:4225 , piece_id=J7R5WRDM6HJD2W5K5URZBWVIBL2MP2GOU7ZZ7TXBS3LN4ZHJ42UA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.456776608 (497h36m20.327859643s) [noise=true , address=145.79.227.13:4804 , piece_id=CVUQPTEPXVZULIREN377ONZIQ665NMGBEQCEOUEVZHTEIVGUVRLQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.457207779 (497h36m20.328290824s) [noise=true , address=45.91.116.6:4974 , piece_id=H76F5IVAIO7Q2GDIINQGO25XNROXWNPZ7RSJ242YZPENAZ6DRKLA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.458453440 (497h36m20.329536476s) [noise=true , address=45.91.116.5:4299 , piece_id=FN5HENWXRAUGQZ3DLCSTZJFMICEIECOS4WXT7ZYS25IA46YYXLEA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.463328213 (497h36m20.334411258s) [noise=true , address=145.79.227.50:4375 , piece_id=F2T7S2AXBCZVCSIBV673JB65PTH2KOJ3BGPBT6DHUEM76QDYCHAQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.465622081 (497h36m20.336705126s) [noise=true , address=145.79.227.19:4078 , piece_id=LJ5RXJFEABFIIK7TV5BHHWIA7CDQHPDBRCRXHMN2CVP65ZP3YPCA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:22.163012587 (497h36m22.034095632s) [upload=436739]: Next returned nil segment Dec 26 09:05:22.163041332 (497h36m22.034124377s) [upload=436739]: Waiting for error group managing segments... Dec 26 09:05:22.453559579 (497h36m22.324642624s) [noise=true , address=145.79.227.35:4483 , piece_id=XY3WYAZZFYPAE6OORXCHVGLIXF74PK4YK6L2GU2BEV4NHSZYRPLQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.459896496 (497h36m22.330979531s) [noise=true , address=145.79.227.46:4603 , piece_id=7QYHIKK22SFK6Z245HTN6ANBL2LJQ2MGBUZXPA4SAIMO5UYG4BNQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.464918438 (497h36m22.336001483s) [noise=true , address=145.79.227.49:4338 , piece_id=F4TGKH3AMO7BXVRCDQ4OTLTJAJZB4LHCF2FQASPUJO2G442B7ENA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.466566056 (497h36m22.337649091s) [noise=true , address=145.79.227.37:4565 , piece_id=RYFRLEO4NB5YV4Y3AS6VRB5VRH5FBHEJEGN6IEMSROTHUWQDWNGQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.483914400 (497h36m22.354997435s) [noise=true , address=145.79.227.3:4149 , piece_id=EHGSRXFW6G2D5SH7AAHSGOYMWPJW45KIXPSCGY726CL5V7MAWIWQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.484448046 (497h36m22.355531081s) [noise=true , address=145.79.227.42:4122 , piece_id=MUW743I4E66WWNEANZPPAJQHHP7F245LYOJK5ELDV3MYOPJLIJKQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.485048360 (497h36m22.356131395s) [noise=true , address=145.79.227.48:4303 , piece_id=I5C36PABM3IO7Z7LNTVFZGY2XRBQZXS3A7JAXYQ3GIYV6RQ3WKTQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.485844586 (497h36m22.356927621s) [noise=true , address=145.79.227.13:4804 , piece_id=CVUQPTEPXVZULIREN377ONZIQ665NMGBEQCEOUEVZHTEIVGUVRLQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.491593703 (497h36m22.362676748s) [noise=true , address=145.79.227.45:4230 , piece_id=F6V27RD3FSPA5VEHR3XWQSN65OACRMLQ7BKEKOV433K4UK57F6JA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.494334231 (497h36m22.365417266s) [noise=true , address=145.79.227.19:4078 , piece_id=LJ5RXJFEABFIIK7TV5BHHWIA7CDQHPDBRCRXHMN2CVP65ZP3YPCA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.497173928 (497h36m22.368256973s) [noise=true , address=145.79.227.50:4375 , piece_id=F2T7S2AXBCZVCSIBV673JB65PTH2KOJ3BGPBT6DHUEM76QDYCHAQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.826633215 (497h36m22.697716261s) [noise=true , address=45.91.116.12:4225 , piece_id=J7R5WRDM6HJD2W5K5URZBWVIBL2MP2GOU7ZZ7TXBS3LN4ZHJ42UA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.835764464 (497h36m22.706847499s) [noise=true , address=45.91.116.11:4117 , piece_id=6P623RZ47UIK4UCUBWRHED7RC45GPU6UDL5OKPA43E4364CP3YAQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.835908578 (497h36m22.706991623s) [seg_pos={0 0}, upload=436739]: Segment reached optimal threshold of 13 pieces. Dec 26 09:05:22.836284153 (497h36m22.707367188s) [noise=true , address=45.91.116.5:4299 , piece_id=FN5HENWXRAUGQZ3DLCSTZJFMICEIECOS4WXT7ZYS25IA46YYXLEA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 09:05:22.836295585 (497h36m22.70737862s) [noise=true , address=45.91.116.5:4299 , piece_id=FN5HENWXRAUGQZ3DLCSTZJFMICEIECOS4WXT7ZYS25IA46YYXLEA, seg_pos={0 0}, upload=436739]: Long tail cancellation due to optimal threshold Dec 26 09:05:22.836830483 (497h36m22.707913528s) [noise=true , address=45.91.116.6:4974 , piece_id=H76F5IVAIO7Q2GDIINQGO25XNROXWNPZ7RSJ242YZPENAZ6DRKLA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:49846->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:49846->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->45.91.116.6:4974: use of closed network connection Dec 26 09:05:22.836840893 (497h36m22.707923938s) [noise=true , address=45.91.116.6:4974 , piece_id=H76F5IVAIO7Q2GDIINQGO25XNROXWNPZ7RSJ242YZPENAZ6DRKLA, seg_pos={0 0}, upload=436739]: Long tail cancellation due to optimal threshold Dec 26 09:05:22.837148850 (497h36m22.708231885s) [noise=true , address=45.91.116.8:4047 , piece_id=GZZZACVBAPUKOPYFGYXMXRZQJCK4UYSYOGF74PZS4GFALAFOZOOA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 09:05:22.837159640 (497h36m22.708242675s) [noise=true , address=45.91.116.8:4047 , piece_id=GZZZACVBAPUKOPYFGYXMXRZQJCK4UYSYOGF74PZS4GFALAFOZOOA, seg_pos={0 0}, upload=436739]: Long tail cancellation due to optimal threshold Dec 26 09:05:22.837183636 (497h36m22.708266681s) [seg_pos={0 0}, upload=436739]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:05:22.837302172 (497h36m22.708385207s) [upload=436739]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:05:22.837309886 (497h36m22.708392931s) [upload=436739]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:05:23.143746675 (497h36m23.01482972s) [upload=436739]: Done uploading segments... Dec 26 09:05:23.143766713 (497h36m23.014849748s) [upload=436739]: Upload finished. err: <nil> Duration: 6.046243526s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X......X.X...X.XX..X.XXX....................X....XXX..X..] duration: 25.616042175s trace: https://jaeger.dev.storj.io/trace/1bc7698723cdc676 FAIL (2025-12-26 09:07:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 09:05:20.203562675 (497h36m20.07464572s) [upload=436739]: Starting upload Dec 26 09:05:20.203657285 (497h36m20.07474032s) [upload=436739]: Done starting upload Dec 26 09:05:20.203740263 (497h36m20.074823308s) [upload=436739]: Uploading segments... Dec 26 09:05:20.203881842 (497h36m20.074964877s) [upload=436739]: Got next segment. Inline: false Dec 26 09:05:20.203915156 (497h36m20.074998201s) [upload=436739]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:05:20.203919946 (497h36m20.075002991s) [upload=436739]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:05:20.451528164 (497h36m20.322611209s) [seg_pos={0 0}, upload=436739]: Begin upload segment... Dec 26 09:05:20.451579712 (497h36m20.322687094s) [seg_pos={0 0}, upload=436739]: Done begin upload segment. Dec 26 09:05:20.451681246 (497h36m20.322764281s) [noise=true , address=145.79.227.45:4230 , piece_id=F6V27RD3FSPA5VEHR3XWQSN65OACRMLQ7BKEKOV433K4UK57F6JA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.451705141 (497h36m20.322788176s) [noise=true , address=45.91.116.11:4117 , piece_id=6P623RZ47UIK4UCUBWRHED7RC45GPU6UDL5OKPA43E4364CP3YAQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.452346592 (497h36m20.323429637s) [noise=true , address=145.79.227.49:4338 , piece_id=F4TGKH3AMO7BXVRCDQ4OTLTJAJZB4LHCF2FQASPUJO2G442B7ENA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.452878154 (497h36m20.323961199s) [noise=true , address=145.79.227.46:4603 , piece_id=7QYHIKK22SFK6Z245HTN6ANBL2LJQ2MGBUZXPA4SAIMO5UYG4BNQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.453292944 (497h36m20.324375989s) [noise=true , address=145.79.227.42:4122 , piece_id=MUW743I4E66WWNEANZPPAJQHHP7F245LYOJK5ELDV3MYOPJLIJKQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.453598226 (497h36m20.324681271s) [noise=true , address=145.79.227.37:4565 , piece_id=RYFRLEO4NB5YV4Y3AS6VRB5VRH5FBHEJEGN6IEMSROTHUWQDWNGQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.453911482 (497h36m20.324994527s) [noise=true , address=45.91.116.8:4047 , piece_id=GZZZACVBAPUKOPYFGYXMXRZQJCK4UYSYOGF74PZS4GFALAFOZOOA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.454291346 (497h36m20.325374381s) [noise=true , address=145.79.227.35:4483 , piece_id=XY3WYAZZFYPAE6OORXCHVGLIXF74PK4YK6L2GU2BEV4NHSZYRPLQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.454619029 (497h36m20.325702064s) [noise=true , address=145.79.227.48:4303 , piece_id=I5C36PABM3IO7Z7LNTVFZGY2XRBQZXS3A7JAXYQ3GIYV6RQ3WKTQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.455510106 (497h36m20.326593151s) [noise=true , address=145.79.227.3:4149 , piece_id=EHGSRXFW6G2D5SH7AAHSGOYMWPJW45KIXPSCGY726CL5V7MAWIWQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.456091063 (497h36m20.327174108s) [noise=true , address=45.91.116.12:4225 , piece_id=J7R5WRDM6HJD2W5K5URZBWVIBL2MP2GOU7ZZ7TXBS3LN4ZHJ42UA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.456776608 (497h36m20.327859643s) [noise=true , address=145.79.227.13:4804 , piece_id=CVUQPTEPXVZULIREN377ONZIQ665NMGBEQCEOUEVZHTEIVGUVRLQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.457207779 (497h36m20.328290824s) [noise=true , address=45.91.116.6:4974 , piece_id=H76F5IVAIO7Q2GDIINQGO25XNROXWNPZ7RSJ242YZPENAZ6DRKLA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.458453440 (497h36m20.329536476s) [noise=true , address=45.91.116.5:4299 , piece_id=FN5HENWXRAUGQZ3DLCSTZJFMICEIECOS4WXT7ZYS25IA46YYXLEA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.463328213 (497h36m20.334411258s) [noise=true , address=145.79.227.50:4375 , piece_id=F2T7S2AXBCZVCSIBV673JB65PTH2KOJ3BGPBT6DHUEM76QDYCHAQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.465622081 (497h36m20.336705126s) [noise=true , address=145.79.227.19:4078 , piece_id=LJ5RXJFEABFIIK7TV5BHHWIA7CDQHPDBRCRXHMN2CVP65ZP3YPCA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:22.163012587 (497h36m22.034095632s) [upload=436739]: Next returned nil segment Dec 26 09:05:22.163041332 (497h36m22.034124377s) [upload=436739]: Waiting for error group managing segments... Dec 26 09:05:22.453559579 (497h36m22.324642624s) [noise=true , address=145.79.227.35:4483 , piece_id=XY3WYAZZFYPAE6OORXCHVGLIXF74PK4YK6L2GU2BEV4NHSZYRPLQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.459896496 (497h36m22.330979531s) [noise=true , address=145.79.227.46:4603 , piece_id=7QYHIKK22SFK6Z245HTN6ANBL2LJQ2MGBUZXPA4SAIMO5UYG4BNQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.464918438 (497h36m22.336001483s) [noise=true , address=145.79.227.49:4338 , piece_id=F4TGKH3AMO7BXVRCDQ4OTLTJAJZB4LHCF2FQASPUJO2G442B7ENA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.466566056 (497h36m22.337649091s) [noise=true , address=145.79.227.37:4565 , piece_id=RYFRLEO4NB5YV4Y3AS6VRB5VRH5FBHEJEGN6IEMSROTHUWQDWNGQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.483914400 (497h36m22.354997435s) [noise=true , address=145.79.227.3:4149 , piece_id=EHGSRXFW6G2D5SH7AAHSGOYMWPJW45KIXPSCGY726CL5V7MAWIWQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.484448046 (497h36m22.355531081s) [noise=true , address=145.79.227.42:4122 , piece_id=MUW743I4E66WWNEANZPPAJQHHP7F245LYOJK5ELDV3MYOPJLIJKQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.485048360 (497h36m22.356131395s) [noise=true , address=145.79.227.48:4303 , piece_id=I5C36PABM3IO7Z7LNTVFZGY2XRBQZXS3A7JAXYQ3GIYV6RQ3WKTQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.485844586 (497h36m22.356927621s) [noise=true , address=145.79.227.13:4804 , piece_id=CVUQPTEPXVZULIREN377ONZIQ665NMGBEQCEOUEVZHTEIVGUVRLQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.491593703 (497h36m22.362676748s) [noise=true , address=145.79.227.45:4230 , piece_id=F6V27RD3FSPA5VEHR3XWQSN65OACRMLQ7BKEKOV433K4UK57F6JA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.494334231 (497h36m22.365417266s) [noise=true , address=145.79.227.19:4078 , piece_id=LJ5RXJFEABFIIK7TV5BHHWIA7CDQHPDBRCRXHMN2CVP65ZP3YPCA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.497173928 (497h36m22.368256973s) [noise=true , address=145.79.227.50:4375 , piece_id=F2T7S2AXBCZVCSIBV673JB65PTH2KOJ3BGPBT6DHUEM76QDYCHAQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.826633215 (497h36m22.697716261s) [noise=true , address=45.91.116.12:4225 , piece_id=J7R5WRDM6HJD2W5K5URZBWVIBL2MP2GOU7ZZ7TXBS3LN4ZHJ42UA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.835764464 (497h36m22.706847499s) [noise=true , address=45.91.116.11:4117 , piece_id=6P623RZ47UIK4UCUBWRHED7RC45GPU6UDL5OKPA43E4364CP3YAQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.835908578 (497h36m22.706991623s) [seg_pos={0 0}, upload=436739]: Segment reached optimal threshold of 13 pieces. Dec 26 09:05:22.836284153 (497h36m22.707367188s) [noise=true , address=45.91.116.5:4299 , piece_id=FN5HENWXRAUGQZ3DLCSTZJFMICEIECOS4WXT7ZYS25IA46YYXLEA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 09:05:22.836295585 (497h36m22.70737862s) [noise=true , address=45.91.116.5:4299 , piece_id=FN5HENWXRAUGQZ3DLCSTZJFMICEIECOS4WXT7ZYS25IA46YYXLEA, seg_pos={0 0}, upload=436739]: Long tail cancellation due to optimal threshold Dec 26 09:05:22.836830483 (497h36m22.707913528s) [noise=true , address=45.91.116.6:4974 , piece_id=H76F5IVAIO7Q2GDIINQGO25XNROXWNPZ7RSJ242YZPENAZ6DRKLA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:49846->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:49846->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->45.91.116.6:4974: use of closed network connection Dec 26 09:05:22.836840893 (497h36m22.707923938s) [noise=true , address=45.91.116.6:4974 , piece_id=H76F5IVAIO7Q2GDIINQGO25XNROXWNPZ7RSJ242YZPENAZ6DRKLA, seg_pos={0 0}, upload=436739]: Long tail cancellation due to optimal threshold Dec 26 09:05:22.837148850 (497h36m22.708231885s) [noise=true , address=45.91.116.8:4047 , piece_id=GZZZACVBAPUKOPYFGYXMXRZQJCK4UYSYOGF74PZS4GFALAFOZOOA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 09:05:22.837159640 (497h36m22.708242675s) [noise=true , address=45.91.116.8:4047 , piece_id=GZZZACVBAPUKOPYFGYXMXRZQJCK4UYSYOGF74PZS4GFALAFOZOOA, seg_pos={0 0}, upload=436739]: Long tail cancellation due to optimal threshold Dec 26 09:05:22.837183636 (497h36m22.708266681s) [seg_pos={0 0}, upload=436739]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:05:22.837302172 (497h36m22.708385207s) [upload=436739]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:05:22.837309886 (497h36m22.708392931s) [upload=436739]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:05:23.143746675 (497h36m23.01482972s) [upload=436739]: Done uploading segments... Dec 26 09:05:23.143766713 (497h36m23.014849748s) [upload=436739]: Upload finished. err: <nil> Duration: 25.616067222s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 15.120710486s trace: https://jaeger.dev.storj.io/trace/6820217d73ecf2a FAIL (2025-12-26 09:07:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments... Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces. Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments... Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil> Duration: 15.120741284s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 21.410794587s trace: https://jaeger.dev.storj.io/trace/6820217d73ecf2a FAIL (2025-12-26 09:06:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments... Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces. Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments... Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil> Duration: 21.410816008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X......X.X...X.XX..X.XXX....................X....XXX..X.] duration: 5.94271956s trace: https://jaeger.dev.storj.io/trace/1bc7698723cdc676 FAIL (2025-12-26 09:06:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 09:05:20.203562675 (497h36m20.07464572s) [upload=436739]: Starting upload Dec 26 09:05:20.203657285 (497h36m20.07474032s) [upload=436739]: Done starting upload Dec 26 09:05:20.203740263 (497h36m20.074823308s) [upload=436739]: Uploading segments... Dec 26 09:05:20.203881842 (497h36m20.074964877s) [upload=436739]: Got next segment. Inline: false Dec 26 09:05:20.203915156 (497h36m20.074998201s) [upload=436739]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 09:05:20.203919946 (497h36m20.075002991s) [upload=436739]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 09:05:20.451528164 (497h36m20.322611209s) [seg_pos={0 0}, upload=436739]: Begin upload segment... Dec 26 09:05:20.451579712 (497h36m20.322687094s) [seg_pos={0 0}, upload=436739]: Done begin upload segment. Dec 26 09:05:20.451681246 (497h36m20.322764281s) [noise=true , address=145.79.227.45:4230 , piece_id=F6V27RD3FSPA5VEHR3XWQSN65OACRMLQ7BKEKOV433K4UK57F6JA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.451705141 (497h36m20.322788176s) [noise=true , address=45.91.116.11:4117 , piece_id=6P623RZ47UIK4UCUBWRHED7RC45GPU6UDL5OKPA43E4364CP3YAQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.452346592 (497h36m20.323429637s) [noise=true , address=145.79.227.49:4338 , piece_id=F4TGKH3AMO7BXVRCDQ4OTLTJAJZB4LHCF2FQASPUJO2G442B7ENA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.452878154 (497h36m20.323961199s) [noise=true , address=145.79.227.46:4603 , piece_id=7QYHIKK22SFK6Z245HTN6ANBL2LJQ2MGBUZXPA4SAIMO5UYG4BNQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.453292944 (497h36m20.324375989s) [noise=true , address=145.79.227.42:4122 , piece_id=MUW743I4E66WWNEANZPPAJQHHP7F245LYOJK5ELDV3MYOPJLIJKQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.453598226 (497h36m20.324681271s) [noise=true , address=145.79.227.37:4565 , piece_id=RYFRLEO4NB5YV4Y3AS6VRB5VRH5FBHEJEGN6IEMSROTHUWQDWNGQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.453911482 (497h36m20.324994527s) [noise=true , address=45.91.116.8:4047 , piece_id=GZZZACVBAPUKOPYFGYXMXRZQJCK4UYSYOGF74PZS4GFALAFOZOOA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.454291346 (497h36m20.325374381s) [noise=true , address=145.79.227.35:4483 , piece_id=XY3WYAZZFYPAE6OORXCHVGLIXF74PK4YK6L2GU2BEV4NHSZYRPLQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.454619029 (497h36m20.325702064s) [noise=true , address=145.79.227.48:4303 , piece_id=I5C36PABM3IO7Z7LNTVFZGY2XRBQZXS3A7JAXYQ3GIYV6RQ3WKTQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.455510106 (497h36m20.326593151s) [noise=true , address=145.79.227.3:4149 , piece_id=EHGSRXFW6G2D5SH7AAHSGOYMWPJW45KIXPSCGY726CL5V7MAWIWQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.456091063 (497h36m20.327174108s) [noise=true , address=45.91.116.12:4225 , piece_id=J7R5WRDM6HJD2W5K5URZBWVIBL2MP2GOU7ZZ7TXBS3LN4ZHJ42UA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.456776608 (497h36m20.327859643s) [noise=true , address=145.79.227.13:4804 , piece_id=CVUQPTEPXVZULIREN377ONZIQ665NMGBEQCEOUEVZHTEIVGUVRLQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.457207779 (497h36m20.328290824s) [noise=true , address=45.91.116.6:4974 , piece_id=H76F5IVAIO7Q2GDIINQGO25XNROXWNPZ7RSJ242YZPENAZ6DRKLA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.458453440 (497h36m20.329536476s) [noise=true , address=45.91.116.5:4299 , piece_id=FN5HENWXRAUGQZ3DLCSTZJFMICEIECOS4WXT7ZYS25IA46YYXLEA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.463328213 (497h36m20.334411258s) [noise=true , address=145.79.227.50:4375 , piece_id=F2T7S2AXBCZVCSIBV673JB65PTH2KOJ3BGPBT6DHUEM76QDYCHAQ, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:20.465622081 (497h36m20.336705126s) [noise=true , address=145.79.227.19:4078 , piece_id=LJ5RXJFEABFIIK7TV5BHHWIA7CDQHPDBRCRXHMN2CVP65ZP3YPCA, seg_pos={0 0}, upload=436739]: Uploading piece... Dec 26 09:05:22.163012587 (497h36m22.034095632s) [upload=436739]: Next returned nil segment Dec 26 09:05:22.163041332 (497h36m22.034124377s) [upload=436739]: Waiting for error group managing segments... Dec 26 09:05:22.453559579 (497h36m22.324642624s) [noise=true , address=145.79.227.35:4483 , piece_id=XY3WYAZZFYPAE6OORXCHVGLIXF74PK4YK6L2GU2BEV4NHSZYRPLQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.459896496 (497h36m22.330979531s) [noise=true , address=145.79.227.46:4603 , piece_id=7QYHIKK22SFK6Z245HTN6ANBL2LJQ2MGBUZXPA4SAIMO5UYG4BNQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.464918438 (497h36m22.336001483s) [noise=true , address=145.79.227.49:4338 , piece_id=F4TGKH3AMO7BXVRCDQ4OTLTJAJZB4LHCF2FQASPUJO2G442B7ENA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.466566056 (497h36m22.337649091s) [noise=true , address=145.79.227.37:4565 , piece_id=RYFRLEO4NB5YV4Y3AS6VRB5VRH5FBHEJEGN6IEMSROTHUWQDWNGQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.483914400 (497h36m22.354997435s) [noise=true , address=145.79.227.3:4149 , piece_id=EHGSRXFW6G2D5SH7AAHSGOYMWPJW45KIXPSCGY726CL5V7MAWIWQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.484448046 (497h36m22.355531081s) [noise=true , address=145.79.227.42:4122 , piece_id=MUW743I4E66WWNEANZPPAJQHHP7F245LYOJK5ELDV3MYOPJLIJKQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.485048360 (497h36m22.356131395s) [noise=true , address=145.79.227.48:4303 , piece_id=I5C36PABM3IO7Z7LNTVFZGY2XRBQZXS3A7JAXYQ3GIYV6RQ3WKTQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.485844586 (497h36m22.356927621s) [noise=true , address=145.79.227.13:4804 , piece_id=CVUQPTEPXVZULIREN377ONZIQ665NMGBEQCEOUEVZHTEIVGUVRLQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.491593703 (497h36m22.362676748s) [noise=true , address=145.79.227.45:4230 , piece_id=F6V27RD3FSPA5VEHR3XWQSN65OACRMLQ7BKEKOV433K4UK57F6JA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.494334231 (497h36m22.365417266s) [noise=true , address=145.79.227.19:4078 , piece_id=LJ5RXJFEABFIIK7TV5BHHWIA7CDQHPDBRCRXHMN2CVP65ZP3YPCA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.497173928 (497h36m22.368256973s) [noise=true , address=145.79.227.50:4375 , piece_id=F2T7S2AXBCZVCSIBV673JB65PTH2KOJ3BGPBT6DHUEM76QDYCHAQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.826633215 (497h36m22.697716261s) [noise=true , address=45.91.116.12:4225 , piece_id=J7R5WRDM6HJD2W5K5URZBWVIBL2MP2GOU7ZZ7TXBS3LN4ZHJ42UA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.835764464 (497h36m22.706847499s) [noise=true , address=45.91.116.11:4117 , piece_id=6P623RZ47UIK4UCUBWRHED7RC45GPU6UDL5OKPA43E4364CP3YAQ, seg_pos={0 0}, upload=436739]: Done uploading piece. err: <nil> Dec 26 09:05:22.835908578 (497h36m22.706991623s) [seg_pos={0 0}, upload=436739]: Segment reached optimal threshold of 13 pieces. Dec 26 09:05:22.836284153 (497h36m22.707367188s) [noise=true , address=45.91.116.5:4299 , piece_id=FN5HENWXRAUGQZ3DLCSTZJFMICEIECOS4WXT7ZYS25IA46YYXLEA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 09:05:22.836295585 (497h36m22.70737862s) [noise=true , address=45.91.116.5:4299 , piece_id=FN5HENWXRAUGQZ3DLCSTZJFMICEIECOS4WXT7ZYS25IA46YYXLEA, seg_pos={0 0}, upload=436739]: Long tail cancellation due to optimal threshold Dec 26 09:05:22.836830483 (497h36m22.707913528s) [noise=true , address=45.91.116.6:4974 , piece_id=H76F5IVAIO7Q2GDIINQGO25XNROXWNPZ7RSJ242YZPENAZ6DRKLA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:49846->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:49846->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49846->45.91.116.6:4974: use of closed network connection Dec 26 09:05:22.836840893 (497h36m22.707923938s) [noise=true , address=45.91.116.6:4974 , piece_id=H76F5IVAIO7Q2GDIINQGO25XNROXWNPZ7RSJ242YZPENAZ6DRKLA, seg_pos={0 0}, upload=436739]: Long tail cancellation due to optimal threshold Dec 26 09:05:22.837148850 (497h36m22.708231885s) [noise=true , address=45.91.116.8:4047 , piece_id=GZZZACVBAPUKOPYFGYXMXRZQJCK4UYSYOGF74PZS4GFALAFOZOOA, seg_pos={0 0}, upload=436739]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 09:05:22.837159640 (497h36m22.708242675s) [noise=true , address=45.91.116.8:4047 , piece_id=GZZZACVBAPUKOPYFGYXMXRZQJCK4UYSYOGF74PZS4GFALAFOZOOA, seg_pos={0 0}, upload=436739]: Long tail cancellation due to optimal threshold Dec 26 09:05:22.837183636 (497h36m22.708266681s) [seg_pos={0 0}, upload=436739]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 09:05:22.837302172 (497h36m22.708385207s) [upload=436739]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 09:05:22.837309886 (497h36m22.708392931s) [upload=436739]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 09:05:23.143746675 (497h36m23.01482972s) [upload=436739]: Done uploading segments... Dec 26 09:05:23.143766713 (497h36m23.014849748s) [upload=436739]: Upload finished. err: <nil> Duration: 5.94275122s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 47.266761696s trace: https://jaeger.dev.storj.io/trace/6820217d73ecf2a FAIL (2025-12-26 09:06:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ss=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705106031 (497h3m36.576189076s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705493759 (497h3m36.576576804s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705630279 (497h3m36.576713334s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.705923348 (497h3m36.577006393s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706055580 (497h3m36.577138615s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706362764 (497h3m36.577445819s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706507651 (497h3m36.577590686s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706728992 (497h3m36.577812027s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.706869399 (497h3m36.577952444s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707077395 (497h3m36.57816044s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707229524 (497h3m36.578312559s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707443251 (497h3m36.578526296s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707633815 (497h3m36.57871685s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.707813797 (497h3m36.578896832s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708022374 (497h3m36.579105439s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:36.708682712 (497h3m36.579765747s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Uploading piece... Dec 26 08:32:38.013251430 (497h3m37.884334475s) [upload=436256]: Next returned nil segment Dec 26 08:32:38.013277480 (497h3m37.884360525s) [upload=436256]: Waiting for error group managing segments... Dec 26 08:32:38.345076384 (497h3m38.216159419s) [noise=true , address=64.49.13.154:4930 , piece_id=IHQYIUPLXMAKL4S76Y6NAGDP2ZWFRQBOXXJJENZ6JRGBXWLKDCFQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593234781 (497h3m38.464317836s) [noise=true , address=145.79.227.20:4443 , piece_id=HAFVT75GMRHFRR5E5VD4AOJFHGRUS56MQX3HNHI6FQYYMZT4J4FQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.593901300 (497h3m38.464984335s) [noise=true , address=145.79.227.5:4241 , piece_id=ANDNWFXV5YWOH55WDLOMJXSKZVN45XSIYT2I4QB52DNO2PT3HYKQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.594508065 (497h3m38.4655911s) [noise=true , address=145.79.227.13:4822 , piece_id=HSHVLD6IYQOG3KQYCZGENM4ZYH7XCKGRAZLEL6N2WELIBI2LNHSA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.636070587 (497h3m38.507153653s) [noise=true , address=145.79.227.34:4909 , piece_id=TSBSOK42K3BBLLTUBV6UFWW33FK67FZHJZTKEQFAAQ7ITKY3RAEQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.750629848 (497h3m38.621712883s) [noise=true , address=77.90.31.11:4581 , piece_id=NQOGQYJLH7W7CXFLBDW3XUN6Y6YVQEWAFBMLNKNHKTU5VCV4V3GQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.814987737 (497h3m38.686070782s) [noise=true , address=45.91.116.4:4942 , piece_id=VDKHG53ATFYJDQYDBXKZ7SN2RNAAZBYGQVGAF34PCG4UYO7XMFGA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.847327759 (497h3m38.718410794s) [noise=true , address=77.90.31.15:4717 , piece_id=CVL5BLBIJ5DWSL7ZSEJ4I62RENOFPNWWNZCSE62CTMWGSS2XTE2Q, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.902089014 (497h3m38.773172049s) [noise=true , address=45.91.116.13:4174 , piece_id=RLKVQ3QJM4STACAOUTWA7STEIG4PITJRIB4YOI3GNTYMPA76IFHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:38.912841608 (497h3m38.783924644s) [noise=true , address=77.90.31.12:4607 , piece_id=TPFAMNN2BEYPSYRYOMDMSS77ZRB35HOJYWMJGTJEERRWGV5KQHHA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.070851326 (497h3m38.941934371s) [noise=true , address=77.90.31.16:4750 , piece_id=HIZWD4OHAZ6UKMODPFHABQRQW7ESEZPVMSD45RI5HGXPLHDJLJHQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.469018898 (497h3m39.340101933s) [noise=true , address=145.79.227.30:4775 , piece_id=OG65SPMQI3XHNJHUHKSJ4NZKGAB764OYFE7D5CBJCP3YRAGYANQA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.477948602 (497h3m39.349031637s) [noise=true , address=145.79.227.44:4226 , piece_id=XFE2F2254TY5CMO7J5O6S3RDKHOHSLQCUGDNHB37VKOXPM63GJEA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534878005 (497h3m39.40596105s) [noise=true , address=77.90.31.17:4788 , piece_id=CIK5XZI3QMG7LFEOKBPZGNLDOORNP6IQV3QYSHLIY5XSF3LLXBBQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: <nil> Dec 26 08:32:39.534902853 (497h3m39.405985898s) [seg_pos={0 0}, upload=436256]: Segment reached optimal threshold of 14 pieces. Dec 26 08:32:39.535131698 (497h3m39.406214743s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535140075 (497h3m39.40622311s) [noise=true , address=77.90.31.19:4870 , piece_id=ZHJNUA7P36RBWXQXMDLFRIHQPSZNOPGL67AQVDTDEPCRY2R4EMWQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535367978 (497h3m39.406451013s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535375542 (497h3m39.406458577s) [noise=true , address=77.90.31.18:4838 , piece_id=V5LDUMURFU4H43E2A5MTLWBI6ZJHVPYMUILBDV5N7ERELPLTIBAA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535565575 (497h3m39.40664861s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K7fQM1NVA1L7ukmDtyWh2DYWypUoj3GbreETZpPL6gXfXmZDD): protocol: expected piece hash; context canceled; EOF Dec 26 08:32:39.535606041 (497h3m39.406689077s) [noise=true , address=77.90.31.13:4655 , piece_id=B6QS3X24QQRPWA3O5ENXR32XJOJZBZZOVVDMSESJYKU3XBIPGDIQ, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535707996 (497h3m39.406791031s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: read tcp 188.166.225.170:35726->64.49.13.169:4705: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:32:39.535715059 (497h3m39.406798094s) [noise=true , address=64.49.13.169:4705 , piece_id=NG53QDRTA5IAV5W64W6MSZVNCWDUDHBB3DP4BF6G7DTF5DAF7ECA, seg_pos={0 0}, upload=436256]: Long tail cancellation due to optimal threshold Dec 26 08:32:39.535740878 (497h3m39.406823923s) [seg_pos={0 0}, upload=436256]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 08:32:39.535861157 (497h3m39.406944202s) [upload=436256]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:32:39.535867620 (497h3m39.406950655s) [upload=436256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:32:39.833497059 (497h3m39.704580164s) [upload=436256]: Done uploading segments... Dec 26 08:32:39.833522858 (497h3m39.704605903s) [upload=436256]: Upload finished. err: <nil> Duration: 47.266789838s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)