Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.....X...XX.........X.....X..X..X.X.X.....XXX.X...X.XXX..X] duration: 30.002542275s trace: https://jaeger.dev.storj.io/trace/22aa59a703ae0913 FAIL (2025-12-26 18:48:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: TJWOKR5XCMSX3BMZCRWINFCKPMKJIG4B3WUY75ORX33RTSF4PKFA) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: CORMEKQMZXHMHTAJ6TJ2EUFKEO3UGQXLKAFH6MOYHWHS6ZLZ52HA) context deadline exceeded ======LOGS====== 26 (507h18m40.559592371s) [upload=445165]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:47:40.937707343 (507h18m40.808790388s) [seg_pos={0 0}, upload=445165]: Begin upload segment... Dec 26 18:47:40.937757789 (507h18m40.808840824s) [seg_pos={0 0}, upload=445165]: Done begin upload segment. Dec 26 18:47:40.937787676 (507h18m40.808870711s) [noise=true , address=145.79.227.36:4518 , piece_id=KQLHU6UJ72MWRAAYLLUDODNPK2TX4S7SYKAETJ5RT4LV3NNKU2FQ, seg_pos={0 0}, upload=445165]: Uploading piece... Dec 26 18:47:40.937936259 (507h18m40.809019294s) [noise=true , address=77.90.31.21:4878 , piece_id=BTYCCT4LDGZL6JNNHGIDZI4LSGSRDQIVBV4IA5Z7LZ5CTFHTGMEA, seg_pos={0 0}, upload=445165]: Uploading piece... Dec 26 18:47:40.938395573 (507h18m40.809478608s) [noise=true , address=145.79.227.45:4229 , piece_id=T5A2DRKA3RTRBMNWPR5ZI4BRRM4BZU2EMWDJSS3PUTH2ZG4TD5KA, seg_pos={0 0}, upload=445165]: Uploading piece... Dec 26 18:47:40.938905895 (507h18m40.80998893s) [noise=true , address=145.79.227.42:4123 , piece_id=LYD4VZ4RT36P6TOZ3M6JPDL6RJI6YD6AGZYXRFFOHR2F5WT7FYQQ, seg_pos={0 0}, upload=445165]: Uploading piece... Dec 26 18:47:40.939369478 (507h18m40.810452513s) [noise=true , address=145.79.227.14:4839 , piece_id=5J3ZQCAXN4IC47MIJFJPUIUD6YWY6D7KU5VQ6DNQQNHAQAILCETQ, seg_pos={0 0}, upload=445165]: Uploading piece... Dec 26 18:47:40.939843681 (507h18m40.810926716s) [noise=true , address=77.90.31.12:4590 , piece_id=TJWOKR5XCMSX3BMZCRWINFCKPMKJIG4B3WUY75ORX33RTSF4PKFA, seg_pos={0 0}, upload=445165]: Uploading piece... Dec 26 18:47:40.940317593 (507h18m40.811400638s) [noise=true , address=77.90.31.17:4773 , piece_id=KH4EDP3C2MQHOT64NNWJX54Y7CA3YM3BJIPQ6FLY32AKEDEBKGLA, seg_pos={0 0}, upload=445165]: Uploading piece... Dec 26 18:47:40.941860201 (507h18m40.812943236s) [noise=true , address=145.79.227.8:4330 , piece_id=G5NPEHCOQEMNBLNAOYFECVTZO44532J5JP3HJXAN5Z4OKDC34MPQ, seg_pos={0 0}, upload=445165]: Uploading piece... Dec 26 18:47:40.942221428 (507h18m40.813304474s) [noise=true , address=145.79.227.21:4475 , piece_id=JOLNQJ7OHCIYJBBSVN4SSWWV5MJSBY67DQTWBBPTRR24EOCW5QUA, seg_pos={0 0}, upload=445165]: Uploading piece... Dec 26 18:47:40.943366880 (507h18m40.814449915s) [noise=true , address=77.90.31.14:4664 , piece_id=CZM4LOXKJPKVBT6336O2UKS22WYGCPNM7XPFCBLFU4IUSVOJUU2Q, seg_pos={0 0}, upload=445165]: Uploading piece... Dec 26 18:47:40.944510407 (507h18m40.815593452s) [noise=true , address=145.79.227.4:4185 , piece_id=HXKPN2WSNUGJBQ7EPFHMKXJY2TXC4ANVYQQ7ZAC6QU5B242W7UVQ, seg_pos={0 0}, upload=445165]: Uploading piece... Dec 26 18:47:40.945336830 (507h18m40.816419865s) [noise=true , address=145.79.227.39:4013 , piece_id=MVD3W5C655XSAQRTJUAGJ3F7C3JIMB76NSOVQMOPV4MXR2SVKK4A, seg_pos={0 0}, upload=445165]: Uploading piece... Dec 26 18:47:40.946165929 (507h18m40.817248964s) [noise=true , address=77.90.31.18:4808 , piece_id=CORMEKQMZXHMHTAJ6TJ2EUFKEO3UGQXLKAFH6MOYHWHS6ZLZ52HA, seg_pos={0 0}, upload=445165]: Uploading piece... Dec 26 18:47:40.949417451 (507h18m40.820500486s) [noise=true , address=145.79.227.13:4804 , piece_id=MTGL62AXRUXA2HZPULDXE7ZTRR323B3SLXWLRU4ZC577SDHG75WQ, seg_pos={0 0}, upload=445165]: Uploading piece... Dec 26 18:47:40.951401339 (507h18m40.822484374s) [noise=true , address=145.79.227.17:4971 , piece_id=56W3HIJUPTFGNQ2VRDCEL44X4MZ3BUGUDCLE2UPHETWYUZ2DDWQQ, seg_pos={0 0}, upload=445165]: Uploading piece... Dec 26 18:47:40.954079348 (507h18m40.825162383s) [noise=true , address=145.79.227.2:4115 , piece_id=MGY5257EB37L3T52XEC377F2RMSWBEJASRWKUAM5BPWVLMALXFPA, seg_pos={0 0}, upload=445165]: Uploading piece... Dec 26 18:47:42.658457011 (507h18m42.529540046s) [upload=445165]: Next returned nil segment Dec 26 18:47:42.658479184 (507h18m42.529562219s) [upload=445165]: Waiting for error group managing segments... Dec 26 18:47:42.945212514 (507h18m42.816295549s) [noise=true , address=145.79.227.36:4518 , piece_id=KQLHU6UJ72MWRAAYLLUDODNPK2TX4S7SYKAETJ5RT4LV3NNKU2FQ, seg_pos={0 0}, upload=445165]: Done uploading piece. err: <nil> Dec 26 18:47:42.950291786 (507h18m42.821374821s) [noise=true , address=145.79.227.42:4123 , piece_id=LYD4VZ4RT36P6TOZ3M6JPDL6RJI6YD6AGZYXRFFOHR2F5WT7FYQQ, seg_pos={0 0}, upload=445165]: Done uploading piece. err: <nil> Dec 26 18:47:42.950684794 (507h18m42.821767839s) [noise=true , address=145.79.227.21:4475 , piece_id=JOLNQJ7OHCIYJBBSVN4SSWWV5MJSBY67DQTWBBPTRR24EOCW5QUA, seg_pos={0 0}, upload=445165]: Done uploading piece. err: <nil> Dec 26 18:47:42.967120370 (507h18m42.838203415s) [noise=true , address=145.79.227.4:4185 , piece_id=HXKPN2WSNUGJBQ7EPFHMKXJY2TXC4ANVYQQ7ZAC6QU5B242W7UVQ, seg_pos={0 0}, upload=445165]: Done uploading piece. err: <nil> Dec 26 18:47:42.967250368 (507h18m42.838333403s) [noise=true , address=145.79.227.45:4229 , piece_id=T5A2DRKA3RTRBMNWPR5ZI4BRRM4BZU2EMWDJSS3PUTH2ZG4TD5KA, seg_pos={0 0}, upload=445165]: Done uploading piece. err: <nil> Dec 26 18:47:42.979075525 (507h18m42.85015857s) [noise=true , address=145.79.227.8:4330 , piece_id=G5NPEHCOQEMNBLNAOYFECVTZO44532J5JP3HJXAN5Z4OKDC34MPQ, seg_pos={0 0}, upload=445165]: Done uploading piece. err: <nil> Dec 26 18:47:42.982834062 (507h18m42.853917107s) [noise=true , address=145.79.227.39:4013 , piece_id=MVD3W5C655XSAQRTJUAGJ3F7C3JIMB76NSOVQMOPV4MXR2SVKK4A, seg_pos={0 0}, upload=445165]: Done uploading piece. err: <nil> Dec 26 18:47:43.047206329 (507h18m42.918289364s) [noise=true , address=145.79.227.13:4804 , piece_id=MTGL62AXRUXA2HZPULDXE7ZTRR323B3SLXWLRU4ZC577SDHG75WQ, seg_pos={0 0}, upload=445165]: Done uploading piece. err: <nil> Dec 26 18:47:43.156546832 (507h18m43.027629897s) [noise=true , address=77.90.31.17:4773 , piece_id=KH4EDP3C2MQHOT64NNWJX54Y7CA3YM3BJIPQ6FLY32AKEDEBKGLA, seg_pos={0 0}, upload=445165]: Done uploading piece. err: <nil> Dec 26 18:47:43.218011251 (507h18m43.089094296s) [noise=true , address=77.90.31.14:4664 , piece_id=CZM4LOXKJPKVBT6336O2UKS22WYGCPNM7XPFCBLFU4IUSVOJUU2Q, seg_pos={0 0}, upload=445165]: Done uploading piece. err: <nil> Dec 26 18:47:43.224105335 (507h18m43.09518837s) [noise=true , address=77.90.31.18:4808 , piece_id=CORMEKQMZXHMHTAJ6TJ2EUFKEO3UGQXLKAFH6MOYHWHS6ZLZ52HA, seg_pos={0 0}, upload=445165]: Done uploading piece. err: <nil> Dec 26 18:47:43.290478872 (507h18m43.161561927s) [noise=true , address=145.79.227.14:4839 , piece_id=5J3ZQCAXN4IC47MIJFJPUIUD6YWY6D7KU5VQ6DNQQNHAQAILCETQ, seg_pos={0 0}, upload=445165]: Done uploading piece. err: <nil> Dec 26 18:47:43.400598178 (507h18m43.271681223s) [noise=true , address=77.90.31.12:4590 , piece_id=TJWOKR5XCMSX3BMZCRWINFCKPMKJIG4B3WUY75ORX33RTSF4PKFA, seg_pos={0 0}, upload=445165]: Done uploading piece. err: <nil> Dec 26 18:47:43.400625911 (507h18m43.271708956s) [seg_pos={0 0}, upload=445165]: Segment reached optimal threshold of 13 pieces. Dec 26 18:47:43.400850648 (507h18m43.271933683s) [noise=true , address=77.90.31.21:4878 , piece_id=BTYCCT4LDGZL6JNNHGIDZI4LSGSRDQIVBV4IA5Z7LZ5CTFHTGMEA, seg_pos={0 0}, upload=445165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:55992->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:55992->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55992->77.90.31.21:4878: use of closed network connection Dec 26 18:47:43.400858344 (507h18m43.271941389s) [noise=true , address=77.90.31.21:4878 , piece_id=BTYCCT4LDGZL6JNNHGIDZI4LSGSRDQIVBV4IA5Z7LZ5CTFHTGMEA, seg_pos={0 0}, upload=445165]: Long tail cancellation due to optimal threshold Dec 26 18:47:43.401036122 (507h18m43.272119157s) [noise=true , address=145.79.227.2:4115 , piece_id=MGY5257EB37L3T52XEC377F2RMSWBEJASRWKUAM5BPWVLMALXFPA, seg_pos={0 0}, upload=445165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:57292->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.2:4115: use of closed network connection Dec 26 18:47:43.401060638 (507h18m43.272143673s) [noise=true , address=145.79.227.2:4115 , piece_id=MGY5257EB37L3T52XEC377F2RMSWBEJASRWKUAM5BPWVLMALXFPA, seg_pos={0 0}, upload=445165]: Long tail cancellation due to optimal threshold Dec 26 18:47:43.401178113 (507h18m43.272261158s) [noise=true , address=145.79.227.17:4971 , piece_id=56W3HIJUPTFGNQ2VRDCEL44X4MZ3BUGUDCLE2UPHETWYUZ2DDWQQ, seg_pos={0 0}, upload=445165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:41330->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:41330->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41330->145.79.227.17:4971: use of closed network connection Dec 26 18:47:43.401184625 (507h18m43.27226767s) [noise=true , address=145.79.227.17:4971 , piece_id=56W3HIJUPTFGNQ2VRDCEL44X4MZ3BUGUDCLE2UPHETWYUZ2DDWQQ, seg_pos={0 0}, upload=445165]: Long tail cancellation due to optimal threshold Dec 26 18:47:43.401205515 (507h18m43.27228856s) [seg_pos={0 0}, upload=445165]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:47:43.401324581 (507h18m43.272407616s) [upload=445165]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:47:43.401332576 (507h18m43.272415621s) [upload=445165]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:47:43.696982706 (507h18m43.568065741s) [upload=445165]: Done uploading segments... Dec 26 18:47:43.697002383 (507h18m43.568085418s) [upload=445165]: Upload finished. err: <nil> Duration:* 30.002562735s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X.....X...XX.........X.....X..X..X.X.X.....XXX.X...X.XXX..] duration: 7.066890999s trace: https://jaeger.dev.storj.io/trace/ec456c7503d5ec5 FAIL (2025-12-26 18:46:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 18:45:41.579605859 (507h16m41.450688904s) [upload=445137]: Starting upload Dec 26 18:45:41.579695739 (507h16m41.450778774s) [upload=445137]: Done starting upload Dec 26 18:45:41.579757537 (507h16m41.450840582s) [upload=445137]: Uploading segments... Dec 26 18:45:41.580064793 (507h16m41.451147838s) [upload=445137]: Got next segment. Inline: false Dec 26 18:45:41.580111641 (507h16m41.451194717s) [upload=445137]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:45:41.580119416 (507h16m41.451202461s) [upload=445137]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:45:41.827795265 (507h16m41.69887831s) [seg_pos={0 0}, upload=445137]: Begin upload segment... Dec 26 18:45:41.827850509 (507h16m41.698933554s) [seg_pos={0 0}, upload=445137]: Done begin upload segment. Dec 26 18:45:41.827891298 (507h16m41.698974333s) [noise=true , address=45.91.116.11:4118 , piece_id=JJPQ5WHLKDAPJ2LLBCDOMKJAFXG3QGSQGG3HDPM3AZMO4DPJDZKQ, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.827904002 (507h16m41.698987047s) [noise=true , address=145.79.227.34:4907 , piece_id=PJGDSMWEVY6YDFETET2ASVDDJ63WCCLMSZRGCJO25S55FZ4QKHXQ, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.828032286 (507h16m41.699115321s) [noise=true , address=145.79.227.35:4481 , piece_id=YK3MRKI46HCLDAWA5VKQLMQRX7ODBBXY7TNTOGBXCPMH3YTZCNPQ, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.828453739 (507h16m41.699536784s) [noise=true , address=145.79.227.11:4730 , piece_id=A5XWQRN25LX67V6CAFLHZG7Z66DSMOOPSW2FAUVJCRHPKEYSHH4A, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.828894237 (507h16m41.699977272s) [noise=true , address=145.79.227.36:4517 , piece_id=JB5PR6SYEPRSF6FP5UZQMZATPGI3R7E6A6Z3YKUMP5JJZ6RZDD6Q, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.829294449 (507h16m41.700377494s) [noise=true , address=145.79.227.8:4330 , piece_id=OLVVNHQY7FZ7TKDBQARTVMRNIMH4M523C5ULXWTGI2CCFFVBQMIA, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.829710622 (507h16m41.700793667s) [noise=true , address=45.91.116.5:4300 , piece_id=A5QG2OAYOYH5VPZATGPVQBYDG46HT64ADVYXVZXRIKMEPKLXXFCA, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.830206175 (507h16m41.70128921s) [noise=true , address=77.90.31.11:4554 , piece_id=ZKMMQQNBYUZS4BW57FDWZTVBFCUZ6XGEM52KDZ2YDG6Q27RQBPFA, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.830802020 (507h16m41.701885065s) [noise=true , address=145.79.227.3:4150 , piece_id=KBAEZDFGJ43WLCRRJNN6M4JB52Y5PHD6LNHTDFSHIMNOCC4IE2PQ, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.831627351 (507h16m41.702710387s) [noise=true , address=45.91.116.12:4225 , piece_id=3XRVYGS7SN7XTSSFZFBKGCKOHKJMVTWJZBWOGFHS6DXKAIDZ5T6Q, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.832688682 (507h16m41.703771717s) [noise=true , address=45.91.116.6:4973 , piece_id=BRVKML6DZQ7E4IDMGGQHSOCOVB5ZYIFVBX2OLA5HK4MNFIAOQULQ, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.834831172 (507h16m41.705914217s) [noise=true , address=145.79.227.47:4265 , piece_id=U4IZ4Q2YSNNHZPZZIWVQAA423KJSWH3Q4APKGCNPA26I4ZU2BKKA, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.836319807 (507h16m41.707402842s) [noise=true , address=45.91.116.9:4262 , piece_id=GXITUMJR4AAVFIY26OFJXN64FTN3FOT2QNNZT55VL2EFIFX4XO7Q, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.837955922 (507h16m41.709038957s) [noise=true , address=77.90.31.13:4627 , piece_id=YSLMYXQG6JJTRAPD7EGIPBRRYTZL55YEHPYWMMULESNBXH4RBSFA, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.838983378 (507h16m41.710066413s) [noise=true , address=145.79.227.39:4014 , piece_id=WDW5DO44EG5JWOSXQLBDD4H6SJW3VIGKZFDP4ZGFEYP3EKCK46PA, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.840030372 (507h16m41.711113417s) [noise=true , address=145.79.227.2:4115 , piece_id=YOYTN5V2YFVEDJXOYLTLYM7EBCCX347XIVZPZO4DCIMDCZNVQE4Q, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:42.957741309 (507h16m42.828824344s) [noise=true , address=77.90.31.11:4554 , piece_id=ZKMMQQNBYUZS4BW57FDWZTVBFCUZ6XGEM52KDZ2YDG6Q27RQBPFA, seg_pos={0 0}, upload=445137]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: storage node overloaded, request limit: 2000 Dec 26 18:45:43.541890262 (507h16m43.412973297s) [upload=445137]: Next returned nil segment Dec 26 18:45:43.541904358 (507h16m43.412987393s) [upload=445137]: Waiting for error group managing segments... Dec 26 18:45:43.834856861 (507h16m43.705939906s) [noise=true , address=145.79.227.36:4517 , piece_id=JB5PR6SYEPRSF6FP5UZQMZATPGI3R7E6A6Z3YKUMP5JJZ6RZDD6Q, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:43.841568171 (507h16m43.712651216s) [noise=true , address=145.79.227.3:4150 , piece_id=KBAEZDFGJ43WLCRRJNN6M4JB52Y5PHD6LNHTDFSHIMNOCC4IE2PQ, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:43.853658332 (507h16m43.724741367s) [noise=true , address=145.79.227.8:4330 , piece_id=OLVVNHQY7FZ7TKDBQARTVMRNIMH4M523C5ULXWTGI2CCFFVBQMIA, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:43.855140224 (507h16m43.726223269s) [noise=true , address=145.79.227.11:4730 , piece_id=A5XWQRN25LX67V6CAFLHZG7Z66DSMOOPSW2FAUVJCRHPKEYSHH4A, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:43.856498050 (507h16m43.727581085s) [noise=true , address=145.79.227.2:4115 , piece_id=YOYTN5V2YFVEDJXOYLTLYM7EBCCX347XIVZPZO4DCIMDCZNVQE4Q, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:43.863872753 (507h16m43.734955788s) [noise=true , address=145.79.227.35:4481 , piece_id=YK3MRKI46HCLDAWA5VKQLMQRX7ODBBXY7TNTOGBXCPMH3YTZCNPQ, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:43.864416237 (507h16m43.735499272s) [noise=true , address=145.79.227.39:4014 , piece_id=WDW5DO44EG5JWOSXQLBDD4H6SJW3VIGKZFDP4ZGFEYP3EKCK46PA, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:44.086581709 (507h16m43.957664744s) [noise=true , address=45.91.116.11:4118 , piece_id=JJPQ5WHLKDAPJ2LLBCDOMKJAFXG3QGSQGG3HDPM3AZMO4DPJDZKQ, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:44.101789137 (507h16m43.972872172s) [noise=true , address=45.91.116.6:4973 , piece_id=BRVKML6DZQ7E4IDMGGQHSOCOVB5ZYIFVBX2OLA5HK4MNFIAOQULQ, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:44.112845289 (507h16m43.983928334s) [noise=true , address=77.90.31.13:4627 , piece_id=YSLMYXQG6JJTRAPD7EGIPBRRYTZL55YEHPYWMMULESNBXH4RBSFA, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:44.113564689 (507h16m43.984647735s) [noise=true , address=45.91.116.5:4300 , piece_id=A5QG2OAYOYH5VPZATGPVQBYDG46HT64ADVYXVZXRIKMEPKLXXFCA, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:44.204784428 (507h16m44.075867463s) [noise=true , address=145.79.227.47:4265 , piece_id=U4IZ4Q2YSNNHZPZZIWVQAA423KJSWH3Q4APKGCNPA26I4ZU2BKKA, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:44.223719293 (507h16m44.094802328s) [noise=true , address=45.91.116.12:4225 , piece_id=3XRVYGS7SN7XTSSFZFBKGCKOHKJMVTWJZBWOGFHS6DXKAIDZ5T6Q, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:44.223737779 (507h16m44.094820814s) [seg_pos={0 0}, upload=445137]: Segment reached optimal threshold of 13 pieces. Dec 26 18:45:44.223898725 (507h16m44.09498176s) [noise=true , address=145.79.227.34:4907 , piece_id=PJGDSMWEVY6YDFETET2ASVDDJ63WCCLMSZRGCJO25S55FZ4QKHXQ, seg_pos={0 0}, upload=445137]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 18:45:44.223904506 (507h16m44.094987541s) [noise=true , address=145.79.227.34:4907 , piece_id=PJGDSMWEVY6YDFETET2ASVDDJ63WCCLMSZRGCJO25S55FZ4QKHXQ, seg_pos={0 0}, upload=445137]: Long tail cancellation due to optimal threshold Dec 26 18:45:44.224149403 (507h16m44.095232438s) [noise=true , address=45.91.116.9:4262 , piece_id=GXITUMJR4AAVFIY26OFJXN64FTN3FOT2QNNZT55VL2EFIFX4XO7Q, seg_pos={0 0}, upload=445137]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 18:45:44.224166435 (507h16m44.09524947s) [noise=true , address=45.91.116.9:4262 , piece_id=GXITUMJR4AAVFIY26OFJXN64FTN3FOT2QNNZT55VL2EFIFX4XO7Q, seg_pos={0 0}, upload=445137]: Long tail cancellation due to optimal threshold Dec 26 18:45:44.224188678 (507h16m44.095271713s) [seg_pos={0 0}, upload=445137]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:45:44.224284189 (507h16m44.095367224s) [upload=445137]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:45:44.224290752 (507h16m44.095373787s) [upload=445137]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:45:44.538313666 (507h16m44.409396711s) [upload=445137]: Done uploading segments... Dec 26 18:45:44.538340638 (507h16m44.409423673s) [upload=445137]: Upload finished. err: <nil> Duration: 7.066915746s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X..X.....X...XX.........X.....X..X..X.X.X.....XXX.X...X.XXX.] duration: 5.758195121s trace: https://jaeger.dev.storj.io/trace/ec456c7503d5ec5 FAIL (2025-12-26 18:46:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 18:45:41.579605859 (507h16m41.450688904s) [upload=445137]: Starting upload Dec 26 18:45:41.579695739 (507h16m41.450778774s) [upload=445137]: Done starting upload Dec 26 18:45:41.579757537 (507h16m41.450840582s) [upload=445137]: Uploading segments... Dec 26 18:45:41.580064793 (507h16m41.451147838s) [upload=445137]: Got next segment. Inline: false Dec 26 18:45:41.580111641 (507h16m41.451194717s) [upload=445137]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:45:41.580119416 (507h16m41.451202461s) [upload=445137]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:45:41.827795265 (507h16m41.69887831s) [seg_pos={0 0}, upload=445137]: Begin upload segment... Dec 26 18:45:41.827850509 (507h16m41.698933554s) [seg_pos={0 0}, upload=445137]: Done begin upload segment. Dec 26 18:45:41.827891298 (507h16m41.698974333s) [noise=true , address=45.91.116.11:4118 , piece_id=JJPQ5WHLKDAPJ2LLBCDOMKJAFXG3QGSQGG3HDPM3AZMO4DPJDZKQ, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.827904002 (507h16m41.698987047s) [noise=true , address=145.79.227.34:4907 , piece_id=PJGDSMWEVY6YDFETET2ASVDDJ63WCCLMSZRGCJO25S55FZ4QKHXQ, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.828032286 (507h16m41.699115321s) [noise=true , address=145.79.227.35:4481 , piece_id=YK3MRKI46HCLDAWA5VKQLMQRX7ODBBXY7TNTOGBXCPMH3YTZCNPQ, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.828453739 (507h16m41.699536784s) [noise=true , address=145.79.227.11:4730 , piece_id=A5XWQRN25LX67V6CAFLHZG7Z66DSMOOPSW2FAUVJCRHPKEYSHH4A, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.828894237 (507h16m41.699977272s) [noise=true , address=145.79.227.36:4517 , piece_id=JB5PR6SYEPRSF6FP5UZQMZATPGI3R7E6A6Z3YKUMP5JJZ6RZDD6Q, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.829294449 (507h16m41.700377494s) [noise=true , address=145.79.227.8:4330 , piece_id=OLVVNHQY7FZ7TKDBQARTVMRNIMH4M523C5ULXWTGI2CCFFVBQMIA, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.829710622 (507h16m41.700793667s) [noise=true , address=45.91.116.5:4300 , piece_id=A5QG2OAYOYH5VPZATGPVQBYDG46HT64ADVYXVZXRIKMEPKLXXFCA, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.830206175 (507h16m41.70128921s) [noise=true , address=77.90.31.11:4554 , piece_id=ZKMMQQNBYUZS4BW57FDWZTVBFCUZ6XGEM52KDZ2YDG6Q27RQBPFA, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.830802020 (507h16m41.701885065s) [noise=true , address=145.79.227.3:4150 , piece_id=KBAEZDFGJ43WLCRRJNN6M4JB52Y5PHD6LNHTDFSHIMNOCC4IE2PQ, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.831627351 (507h16m41.702710387s) [noise=true , address=45.91.116.12:4225 , piece_id=3XRVYGS7SN7XTSSFZFBKGCKOHKJMVTWJZBWOGFHS6DXKAIDZ5T6Q, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.832688682 (507h16m41.703771717s) [noise=true , address=45.91.116.6:4973 , piece_id=BRVKML6DZQ7E4IDMGGQHSOCOVB5ZYIFVBX2OLA5HK4MNFIAOQULQ, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.834831172 (507h16m41.705914217s) [noise=true , address=145.79.227.47:4265 , piece_id=U4IZ4Q2YSNNHZPZZIWVQAA423KJSWH3Q4APKGCNPA26I4ZU2BKKA, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.836319807 (507h16m41.707402842s) [noise=true , address=45.91.116.9:4262 , piece_id=GXITUMJR4AAVFIY26OFJXN64FTN3FOT2QNNZT55VL2EFIFX4XO7Q, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.837955922 (507h16m41.709038957s) [noise=true , address=77.90.31.13:4627 , piece_id=YSLMYXQG6JJTRAPD7EGIPBRRYTZL55YEHPYWMMULESNBXH4RBSFA, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.838983378 (507h16m41.710066413s) [noise=true , address=145.79.227.39:4014 , piece_id=WDW5DO44EG5JWOSXQLBDD4H6SJW3VIGKZFDP4ZGFEYP3EKCK46PA, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.840030372 (507h16m41.711113417s) [noise=true , address=145.79.227.2:4115 , piece_id=YOYTN5V2YFVEDJXOYLTLYM7EBCCX347XIVZPZO4DCIMDCZNVQE4Q, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:42.957741309 (507h16m42.828824344s) [noise=true , address=77.90.31.11:4554 , piece_id=ZKMMQQNBYUZS4BW57FDWZTVBFCUZ6XGEM52KDZ2YDG6Q27RQBPFA, seg_pos={0 0}, upload=445137]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: storage node overloaded, request limit: 2000 Dec 26 18:45:43.541890262 (507h16m43.412973297s) [upload=445137]: Next returned nil segment Dec 26 18:45:43.541904358 (507h16m43.412987393s) [upload=445137]: Waiting for error group managing segments... Dec 26 18:45:43.834856861 (507h16m43.705939906s) [noise=true , address=145.79.227.36:4517 , piece_id=JB5PR6SYEPRSF6FP5UZQMZATPGI3R7E6A6Z3YKUMP5JJZ6RZDD6Q, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:43.841568171 (507h16m43.712651216s) [noise=true , address=145.79.227.3:4150 , piece_id=KBAEZDFGJ43WLCRRJNN6M4JB52Y5PHD6LNHTDFSHIMNOCC4IE2PQ, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:43.853658332 (507h16m43.724741367s) [noise=true , address=145.79.227.8:4330 , piece_id=OLVVNHQY7FZ7TKDBQARTVMRNIMH4M523C5ULXWTGI2CCFFVBQMIA, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:43.855140224 (507h16m43.726223269s) [noise=true , address=145.79.227.11:4730 , piece_id=A5XWQRN25LX67V6CAFLHZG7Z66DSMOOPSW2FAUVJCRHPKEYSHH4A, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:43.856498050 (507h16m43.727581085s) [noise=true , address=145.79.227.2:4115 , piece_id=YOYTN5V2YFVEDJXOYLTLYM7EBCCX347XIVZPZO4DCIMDCZNVQE4Q, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:43.863872753 (507h16m43.734955788s) [noise=true , address=145.79.227.35:4481 , piece_id=YK3MRKI46HCLDAWA5VKQLMQRX7ODBBXY7TNTOGBXCPMH3YTZCNPQ, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:43.864416237 (507h16m43.735499272s) [noise=true , address=145.79.227.39:4014 , piece_id=WDW5DO44EG5JWOSXQLBDD4H6SJW3VIGKZFDP4ZGFEYP3EKCK46PA, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:44.086581709 (507h16m43.957664744s) [noise=true , address=45.91.116.11:4118 , piece_id=JJPQ5WHLKDAPJ2LLBCDOMKJAFXG3QGSQGG3HDPM3AZMO4DPJDZKQ, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:44.101789137 (507h16m43.972872172s) [noise=true , address=45.91.116.6:4973 , piece_id=BRVKML6DZQ7E4IDMGGQHSOCOVB5ZYIFVBX2OLA5HK4MNFIAOQULQ, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:44.112845289 (507h16m43.983928334s) [noise=true , address=77.90.31.13:4627 , piece_id=YSLMYXQG6JJTRAPD7EGIPBRRYTZL55YEHPYWMMULESNBXH4RBSFA, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:44.113564689 (507h16m43.984647735s) [noise=true , address=45.91.116.5:4300 , piece_id=A5QG2OAYOYH5VPZATGPVQBYDG46HT64ADVYXVZXRIKMEPKLXXFCA, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:44.204784428 (507h16m44.075867463s) [noise=true , address=145.79.227.47:4265 , piece_id=U4IZ4Q2YSNNHZPZZIWVQAA423KJSWH3Q4APKGCNPA26I4ZU2BKKA, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:44.223719293 (507h16m44.094802328s) [noise=true , address=45.91.116.12:4225 , piece_id=3XRVYGS7SN7XTSSFZFBKGCKOHKJMVTWJZBWOGFHS6DXKAIDZ5T6Q, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:44.223737779 (507h16m44.094820814s) [seg_pos={0 0}, upload=445137]: Segment reached optimal threshold of 13 pieces. Dec 26 18:45:44.223898725 (507h16m44.09498176s) [noise=true , address=145.79.227.34:4907 , piece_id=PJGDSMWEVY6YDFETET2ASVDDJ63WCCLMSZRGCJO25S55FZ4QKHXQ, seg_pos={0 0}, upload=445137]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 18:45:44.223904506 (507h16m44.094987541s) [noise=true , address=145.79.227.34:4907 , piece_id=PJGDSMWEVY6YDFETET2ASVDDJ63WCCLMSZRGCJO25S55FZ4QKHXQ, seg_pos={0 0}, upload=445137]: Long tail cancellation due to optimal threshold Dec 26 18:45:44.224149403 (507h16m44.095232438s) [noise=true , address=45.91.116.9:4262 , piece_id=GXITUMJR4AAVFIY26OFJXN64FTN3FOT2QNNZT55VL2EFIFX4XO7Q, seg_pos={0 0}, upload=445137]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 18:45:44.224166435 (507h16m44.09524947s) [noise=true , address=45.91.116.9:4262 , piece_id=GXITUMJR4AAVFIY26OFJXN64FTN3FOT2QNNZT55VL2EFIFX4XO7Q, seg_pos={0 0}, upload=445137]: Long tail cancellation due to optimal threshold Dec 26 18:45:44.224188678 (507h16m44.095271713s) [seg_pos={0 0}, upload=445137]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:45:44.224284189 (507h16m44.095367224s) [upload=445137]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:45:44.224290752 (507h16m44.095373787s) [upload=445137]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:45:44.538313666 (507h16m44.409396711s) [upload=445137]: Done uploading segments... Dec 26 18:45:44.538340638 (507h16m44.409423673s) [upload=445137]: Upload finished. err: <nil> Duration: 5.758225479s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X..X.....X...XX.........X.....X..X..X.X.X.....XXX.X...X.XXX] duration: 30.003658539s trace: https://jaeger.dev.storj.io/trace/ec456c7503d5ec5 FAIL (2025-12-26 18:46:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 18:45:41.579605859 (507h16m41.450688904s) [upload=445137]: Starting upload Dec 26 18:45:41.579695739 (507h16m41.450778774s) [upload=445137]: Done starting upload Dec 26 18:45:41.579757537 (507h16m41.450840582s) [upload=445137]: Uploading segments... Dec 26 18:45:41.580064793 (507h16m41.451147838s) [upload=445137]: Got next segment. Inline: false Dec 26 18:45:41.580111641 (507h16m41.451194717s) [upload=445137]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:45:41.580119416 (507h16m41.451202461s) [upload=445137]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:45:41.827795265 (507h16m41.69887831s) [seg_pos={0 0}, upload=445137]: Begin upload segment... Dec 26 18:45:41.827850509 (507h16m41.698933554s) [seg_pos={0 0}, upload=445137]: Done begin upload segment. Dec 26 18:45:41.827891298 (507h16m41.698974333s) [noise=true , address=45.91.116.11:4118 , piece_id=JJPQ5WHLKDAPJ2LLBCDOMKJAFXG3QGSQGG3HDPM3AZMO4DPJDZKQ, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.827904002 (507h16m41.698987047s) [noise=true , address=145.79.227.34:4907 , piece_id=PJGDSMWEVY6YDFETET2ASVDDJ63WCCLMSZRGCJO25S55FZ4QKHXQ, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.828032286 (507h16m41.699115321s) [noise=true , address=145.79.227.35:4481 , piece_id=YK3MRKI46HCLDAWA5VKQLMQRX7ODBBXY7TNTOGBXCPMH3YTZCNPQ, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.828453739 (507h16m41.699536784s) [noise=true , address=145.79.227.11:4730 , piece_id=A5XWQRN25LX67V6CAFLHZG7Z66DSMOOPSW2FAUVJCRHPKEYSHH4A, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.828894237 (507h16m41.699977272s) [noise=true , address=145.79.227.36:4517 , piece_id=JB5PR6SYEPRSF6FP5UZQMZATPGI3R7E6A6Z3YKUMP5JJZ6RZDD6Q, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.829294449 (507h16m41.700377494s) [noise=true , address=145.79.227.8:4330 , piece_id=OLVVNHQY7FZ7TKDBQARTVMRNIMH4M523C5ULXWTGI2CCFFVBQMIA, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.829710622 (507h16m41.700793667s) [noise=true , address=45.91.116.5:4300 , piece_id=A5QG2OAYOYH5VPZATGPVQBYDG46HT64ADVYXVZXRIKMEPKLXXFCA, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.830206175 (507h16m41.70128921s) [noise=true , address=77.90.31.11:4554 , piece_id=ZKMMQQNBYUZS4BW57FDWZTVBFCUZ6XGEM52KDZ2YDG6Q27RQBPFA, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.830802020 (507h16m41.701885065s) [noise=true , address=145.79.227.3:4150 , piece_id=KBAEZDFGJ43WLCRRJNN6M4JB52Y5PHD6LNHTDFSHIMNOCC4IE2PQ, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.831627351 (507h16m41.702710387s) [noise=true , address=45.91.116.12:4225 , piece_id=3XRVYGS7SN7XTSSFZFBKGCKOHKJMVTWJZBWOGFHS6DXKAIDZ5T6Q, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.832688682 (507h16m41.703771717s) [noise=true , address=45.91.116.6:4973 , piece_id=BRVKML6DZQ7E4IDMGGQHSOCOVB5ZYIFVBX2OLA5HK4MNFIAOQULQ, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.834831172 (507h16m41.705914217s) [noise=true , address=145.79.227.47:4265 , piece_id=U4IZ4Q2YSNNHZPZZIWVQAA423KJSWH3Q4APKGCNPA26I4ZU2BKKA, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.836319807 (507h16m41.707402842s) [noise=true , address=45.91.116.9:4262 , piece_id=GXITUMJR4AAVFIY26OFJXN64FTN3FOT2QNNZT55VL2EFIFX4XO7Q, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.837955922 (507h16m41.709038957s) [noise=true , address=77.90.31.13:4627 , piece_id=YSLMYXQG6JJTRAPD7EGIPBRRYTZL55YEHPYWMMULESNBXH4RBSFA, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.838983378 (507h16m41.710066413s) [noise=true , address=145.79.227.39:4014 , piece_id=WDW5DO44EG5JWOSXQLBDD4H6SJW3VIGKZFDP4ZGFEYP3EKCK46PA, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:41.840030372 (507h16m41.711113417s) [noise=true , address=145.79.227.2:4115 , piece_id=YOYTN5V2YFVEDJXOYLTLYM7EBCCX347XIVZPZO4DCIMDCZNVQE4Q, seg_pos={0 0}, upload=445137]: Uploading piece... Dec 26 18:45:42.957741309 (507h16m42.828824344s) [noise=true , address=77.90.31.11:4554 , piece_id=ZKMMQQNBYUZS4BW57FDWZTVBFCUZ6XGEM52KDZ2YDG6Q27RQBPFA, seg_pos={0 0}, upload=445137]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: storage node overloaded, request limit: 2000 Dec 26 18:45:43.541890262 (507h16m43.412973297s) [upload=445137]: Next returned nil segment Dec 26 18:45:43.541904358 (507h16m43.412987393s) [upload=445137]: Waiting for error group managing segments... Dec 26 18:45:43.834856861 (507h16m43.705939906s) [noise=true , address=145.79.227.36:4517 , piece_id=JB5PR6SYEPRSF6FP5UZQMZATPGI3R7E6A6Z3YKUMP5JJZ6RZDD6Q, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:43.841568171 (507h16m43.712651216s) [noise=true , address=145.79.227.3:4150 , piece_id=KBAEZDFGJ43WLCRRJNN6M4JB52Y5PHD6LNHTDFSHIMNOCC4IE2PQ, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:43.853658332 (507h16m43.724741367s) [noise=true , address=145.79.227.8:4330 , piece_id=OLVVNHQY7FZ7TKDBQARTVMRNIMH4M523C5ULXWTGI2CCFFVBQMIA, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:43.855140224 (507h16m43.726223269s) [noise=true , address=145.79.227.11:4730 , piece_id=A5XWQRN25LX67V6CAFLHZG7Z66DSMOOPSW2FAUVJCRHPKEYSHH4A, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:43.856498050 (507h16m43.727581085s) [noise=true , address=145.79.227.2:4115 , piece_id=YOYTN5V2YFVEDJXOYLTLYM7EBCCX347XIVZPZO4DCIMDCZNVQE4Q, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:43.863872753 (507h16m43.734955788s) [noise=true , address=145.79.227.35:4481 , piece_id=YK3MRKI46HCLDAWA5VKQLMQRX7ODBBXY7TNTOGBXCPMH3YTZCNPQ, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:43.864416237 (507h16m43.735499272s) [noise=true , address=145.79.227.39:4014 , piece_id=WDW5DO44EG5JWOSXQLBDD4H6SJW3VIGKZFDP4ZGFEYP3EKCK46PA, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:44.086581709 (507h16m43.957664744s) [noise=true , address=45.91.116.11:4118 , piece_id=JJPQ5WHLKDAPJ2LLBCDOMKJAFXG3QGSQGG3HDPM3AZMO4DPJDZKQ, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:44.101789137 (507h16m43.972872172s) [noise=true , address=45.91.116.6:4973 , piece_id=BRVKML6DZQ7E4IDMGGQHSOCOVB5ZYIFVBX2OLA5HK4MNFIAOQULQ, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:44.112845289 (507h16m43.983928334s) [noise=true , address=77.90.31.13:4627 , piece_id=YSLMYXQG6JJTRAPD7EGIPBRRYTZL55YEHPYWMMULESNBXH4RBSFA, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:44.113564689 (507h16m43.984647735s) [noise=true , address=45.91.116.5:4300 , piece_id=A5QG2OAYOYH5VPZATGPVQBYDG46HT64ADVYXVZXRIKMEPKLXXFCA, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:44.204784428 (507h16m44.075867463s) [noise=true , address=145.79.227.47:4265 , piece_id=U4IZ4Q2YSNNHZPZZIWVQAA423KJSWH3Q4APKGCNPA26I4ZU2BKKA, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:44.223719293 (507h16m44.094802328s) [noise=true , address=45.91.116.12:4225 , piece_id=3XRVYGS7SN7XTSSFZFBKGCKOHKJMVTWJZBWOGFHS6DXKAIDZ5T6Q, seg_pos={0 0}, upload=445137]: Done uploading piece. err: <nil> Dec 26 18:45:44.223737779 (507h16m44.094820814s) [seg_pos={0 0}, upload=445137]: Segment reached optimal threshold of 13 pieces. Dec 26 18:45:44.223898725 (507h16m44.09498176s) [noise=true , address=145.79.227.34:4907 , piece_id=PJGDSMWEVY6YDFETET2ASVDDJ63WCCLMSZRGCJO25S55FZ4QKHXQ, seg_pos={0 0}, upload=445137]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 18:45:44.223904506 (507h16m44.094987541s) [noise=true , address=145.79.227.34:4907 , piece_id=PJGDSMWEVY6YDFETET2ASVDDJ63WCCLMSZRGCJO25S55FZ4QKHXQ, seg_pos={0 0}, upload=445137]: Long tail cancellation due to optimal threshold Dec 26 18:45:44.224149403 (507h16m44.095232438s) [noise=true , address=45.91.116.9:4262 , piece_id=GXITUMJR4AAVFIY26OFJXN64FTN3FOT2QNNZT55VL2EFIFX4XO7Q, seg_pos={0 0}, upload=445137]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 18:45:44.224166435 (507h16m44.09524947s) [noise=true , address=45.91.116.9:4262 , piece_id=GXITUMJR4AAVFIY26OFJXN64FTN3FOT2QNNZT55VL2EFIFX4XO7Q, seg_pos={0 0}, upload=445137]: Long tail cancellation due to optimal threshold Dec 26 18:45:44.224188678 (507h16m44.095271713s) [seg_pos={0 0}, upload=445137]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:45:44.224284189 (507h16m44.095367224s) [upload=445137]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:45:44.224290752 (507h16m44.095373787s) [upload=445137]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:45:44.538313666 (507h16m44.409396711s) [upload=445137]: Done uploading segments... Dec 26 18:45:44.538340638 (507h16m44.409423673s) [upload=445137]: Upload finished. err: <nil> Duration: 30.003679208s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X..X.....X...XX.........X.....X..X..X.X.X.....XXX.X...X.XX] duration: 30.003639874s trace: https://jaeger.dev.storj.io/trace/1041f94cd05c1f0b FAIL (2025-12-26 18:45:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: MIXHNRJ2OOOMQKYTYTBHFVQFXZL3UOOA7675OSTETTP2ROEEGQ5A) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: EMUZGFKBUJM2NAPTKGYVPWDYH5HUISR7ASXKVKM4HGNU4T4BQLVQ) context deadline exceeded ======LOGS====== Dec 26 18:45:07.902650418 (507h16m7.773733463s) [upload=445128]: Starting upload Dec 26 18:45:07.902866630 (507h16m7.773949675s) [upload=445128]: Done starting upload Dec 26 18:45:07.902918037 (507h16m7.774001072s) [upload=445128]: Uploading segments... Dec 26 18:45:07.903171841 (507h16m7.774254886s) [upload=445128]: Got next segment. Inline: false Dec 26 18:45:07.903204984 (507h16m7.774288019s) [upload=445128]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:45:07.903210254 (507h16m7.774293289s) [upload=445128]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:45:08.155238440 (507h16m8.026321485s) [seg_pos={0 0}, upload=445128]: Begin upload segment... Dec 26 18:45:08.155283566 (507h16m8.026366601s) [seg_pos={0 0}, upload=445128]: Done begin upload segment. Dec 26 18:45:08.155298754 (507h16m8.026381789s) [noise=true , address=145.79.227.49:4337 , piece_id=SBHSN6N5AE72AJRBRHGSNI4DVEB7VJFIBYZCUK6WRFAWH2WMKUXA, seg_pos={0 0}, upload=445128]: Uploading piece... Dec 26 18:45:08.155314715 (507h16m8.02639775s) [noise=true , address=45.91.116.7:4011 , piece_id=IZVR55VUQRPB55HENXCNEGUAKEHONDFAHJWDMGVFLIGLX5QSHGDA, seg_pos={0 0}, upload=445128]: Uploading piece... Dec 26 18:45:08.155962710 (507h16m8.027045745s) [noise=true , address=145.79.227.32:4836 , piece_id=UTLJ3FIPBA3ZP4MDHMHLVQPPD3K2542T5W2UOMCAPEHF7SD4OXWA, seg_pos={0 0}, upload=445128]: Uploading piece... Dec 26 18:45:08.156430640 (507h16m8.027513675s) [noise=true , address=145.79.227.5:4221 , piece_id=SRWJYWUK3MZ7EYZO5YE66MAYMYZLF4KYAFXC6ZYMT2MJ4PTOW5ZQ, seg_pos={0 0}, upload=445128]: Uploading piece... Dec 26 18:45:08.156778693 (507h16m8.027861738s) [noise=true , address=145.79.227.17:4972 , piece_id=PJX7TQ5V4A52VQOVNH4AHLULWV7XO2GCZR2RZKQH7WTDB7GLI5XQ, seg_pos={0 0}, upload=445128]: Uploading piece... Dec 26 18:45:08.157062593 (507h16m8.028145628s) [noise=true , address=145.79.227.41:4086 , piece_id=7YYBJ4TBKVRURF5VVDQVGS7HKF2W6OGMF3TEK7NKY3N7XLMNWYMQ, seg_pos={0 0}, upload=445128]: Uploading piece... Dec 26 18:45:08.157385057 (507h16m8.028468102s) [noise=true , address=145.79.227.13:4804 , piece_id=PIOCOQHFHPYYRNPL7MYIMOTIO5KK2EWUCQ2F6DRREHFCFANDINIA, seg_pos={0 0}, upload=445128]: Uploading piece... Dec 26 18:45:08.157769158 (507h16m8.028852204s) [noise=true , address=77.90.31.21:4878 , piece_id=YD44VJLKPPJGQS5XOYYXB2T3FHFVYPDTYEDCCTEJOQX3WJJLAJVA, seg_pos={0 0}, upload=445128]: Uploading piece... Dec 26 18:45:08.158166135 (507h16m8.02924917s) [noise=true , address=145.79.227.8:4330 , piece_id=R3LOXDPQGHGCWVOUDSEYD4OTD6UFG4KGYQZPOOYE4LFM6FS26XLA, seg_pos={0 0}, upload=445128]: Uploading piece... Dec 26 18:45:08.159161480 (507h16m8.030244515s) [noise=true , address=145.79.227.37:4566 , piece_id=COJU5VZXE6HC42YFYEGO6PXLW2NE67HXZAAL7OH43XDGZAZ5M5AQ, seg_pos={0 0}, upload=445128]: Uploading piece... Dec 26 18:45:08.159433147 (507h16m8.030516182s) [noise=true , address=45.91.116.14:4190 , piece_id=CPUPJJH3BVDGHKOCRPXYUHVYSS3TLSXMEHI2MLUCKE4G5ESWS56Q, seg_pos={0 0}, upload=445128]: Uploading piece... Dec 26 18:45:08.160239963 (507h16m8.031322998s) [noise=true , address=45.91.116.9:4261 , piece_id=MIXHNRJ2OOOMQKYTYTBHFVQFXZL3UOOA7675OSTETTP2ROEEGQ5A, seg_pos={0 0}, upload=445128]: Uploading piece... Dec 26 18:45:08.160496252 (507h16m8.031579287s) [noise=true , address=45.91.116.6:4975 , piece_id=PI5PF5PDHKRCB2Z3MW5RRN2ODGTJEUWDKJHN2G27N3W35FZCLNSQ, seg_pos={0 0}, upload=445128]: Uploading piece... Dec 26 18:45:08.161306745 (507h16m8.03238978s) [noise=true , address=45.91.116.10:4084 , piece_id=BPXGNCZSMIOXPVCV7EHQOTJKODFCWF66HIVYEBRBCE3KBPYGZNDQ, seg_pos={0 0}, upload=445128]: Uploading piece... Dec 26 18:45:08.161550038 (507h16m8.032633083s) [noise=true , address=45.91.116.5:4299 , piece_id=EMUZGFKBUJM2NAPTKGYVPWDYH5HUISR7ASXKVKM4HGNU4T4BQLVQ, seg_pos={0 0}, upload=445128]: Uploading piece... Dec 26 18:45:08.162479929 (507h16m8.033562974s) [noise=true , address=45.91.116.8:4045 , piece_id=2AHSIJVKPNATJT7QAUIT7V7OMX76MGRWSGTAAYZPPLRAB2L47ZLA, seg_pos={0 0}, upload=445128]: Uploading piece... Dec 26 18:45:09.867589756 (507h16m9.738672791s) [upload=445128]: Next returned nil segment Dec 26 18:45:09.867602672 (507h16m9.738685707s) [upload=445128]: Waiting for error group managing segments... Dec 26 18:45:10.158357279 (507h16m10.029440324s) [noise=true , address=145.79.227.49:4337 , piece_id=SBHSN6N5AE72AJRBRHGSNI4DVEB7VJFIBYZCUK6WRFAWH2WMKUXA, seg_pos={0 0}, upload=445128]: Done uploading piece. err: <nil> Dec 26 18:45:10.165766286 (507h16m10.036849331s) [noise=true , address=145.79.227.37:4566 , piece_id=COJU5VZXE6HC42YFYEGO6PXLW2NE67HXZAAL7OH43XDGZAZ5M5AQ, seg_pos={0 0}, upload=445128]: Done uploading piece. err: <nil> Dec 26 18:45:10.182481706 (507h16m10.053564751s) [noise=true , address=145.79.227.32:4836 , piece_id=UTLJ3FIPBA3ZP4MDHMHLVQPPD3K2542T5W2UOMCAPEHF7SD4OXWA, seg_pos={0 0}, upload=445128]: Done uploading piece. err: <nil> Dec 26 18:45:10.188384656 (507h16m10.059467701s) [noise=true , address=145.79.227.8:4330 , piece_id=R3LOXDPQGHGCWVOUDSEYD4OTD6UFG4KGYQZPOOYE4LFM6FS26XLA, seg_pos={0 0}, upload=445128]: Done uploading piece. err: <nil> Dec 26 18:45:10.189639996 (507h16m10.060723031s) [noise=true , address=145.79.227.41:4086 , piece_id=7YYBJ4TBKVRURF5VVDQVGS7HKF2W6OGMF3TEK7NKY3N7XLMNWYMQ, seg_pos={0 0}, upload=445128]: Done uploading piece. err: <nil> Dec 26 18:45:10.237034143 (507h16m10.108117188s) [noise=true , address=145.79.227.13:4804 , piece_id=PIOCOQHFHPYYRNPL7MYIMOTIO5KK2EWUCQ2F6DRREHFCFANDINIA, seg_pos={0 0}, upload=445128]: Done uploading piece. err: <nil> Dec 26 18:45:10.281325022 (507h16m10.152408067s) [noise=true , address=145.79.227.17:4972 , piece_id=PJX7TQ5V4A52VQOVNH4AHLULWV7XO2GCZR2RZKQH7WTDB7GLI5XQ, seg_pos={0 0}, upload=445128]: Done uploading piece. err: <nil> Dec 26 18:45:10.399053357 (507h16m10.270136402s) [noise=true , address=77.90.31.21:4878 , piece_id=YD44VJLKPPJGQS5XOYYXB2T3FHFVYPDTYEDCCTEJOQX3WJJLAJVA, seg_pos={0 0}, upload=445128]: Done uploading piece. err: <nil> Dec 26 18:45:10.402149342 (507h16m10.273232387s) [noise=true , address=45.91.116.7:4011 , piece_id=IZVR55VUQRPB55HENXCNEGUAKEHONDFAHJWDMGVFLIGLX5QSHGDA, seg_pos={0 0}, upload=445128]: Done uploading piece. err: <nil> Dec 26 18:45:10.421703717 (507h16m10.292786752s) [noise=true , address=45.91.116.9:4261 , piece_id=MIXHNRJ2OOOMQKYTYTBHFVQFXZL3UOOA7675OSTETTP2ROEEGQ5A, seg_pos={0 0}, upload=445128]: Done uploading piece. err: <nil> Dec 26 18:45:10.422902159 (507h16m10.293985204s) [noise=true , address=45.91.116.8:4045 , piece_id=2AHSIJVKPNATJT7QAUIT7V7OMX76MGRWSGTAAYZPPLRAB2L47ZLA, seg_pos={0 0}, upload=445128]: Done uploading piece. err: <nil> Dec 26 18:45:10.470054776 (507h16m10.341137821s) [noise=true , address=45.91.116.5:4299 , piece_id=EMUZGFKBUJM2NAPTKGYVPWDYH5HUISR7ASXKVKM4HGNU4T4BQLVQ, seg_pos={0 0}, upload=445128]: Done uploading piece. err: <nil> Dec 26 18:45:10.472014427 (507h16m10.343097482s) [noise=true , address=45.91.116.14:4190 , piece_id=CPUPJJH3BVDGHKOCRPXYUHVYSS3TLSXMEHI2MLUCKE4G5ESWS56Q, seg_pos={0 0}, upload=445128]: Done uploading piece. err: <nil> Dec 26 18:45:10.472041990 (507h16m10.343125045s) [seg_pos={0 0}, upload=445128]: Segment reached optimal threshold of 13 pieces. Dec 26 18:45:10.472292036 (507h16m10.343375101s) [noise=true , address=145.79.227.5:4221 , piece_id=SRWJYWUK3MZ7EYZO5YE66MAYMYZLF4KYAFXC6ZYMT2MJ4PTOW5ZQ, seg_pos={0 0}, upload=445128]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:35460->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:35460->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35460->145.79.227.5:4221: use of closed network connection Dec 26 18:45:10.472302375 (507h16m10.34338543s) [noise=true , address=145.79.227.5:4221 , piece_id=SRWJYWUK3MZ7EYZO5YE66MAYMYZLF4KYAFXC6ZYMT2MJ4PTOW5ZQ, seg_pos={0 0}, upload=445128]: Long tail cancellation due to optimal threshold Dec 26 18:45:10.472391906 (507h16m10.343474941s) [noise=true , address=45.91.116.10:4084 , piece_id=BPXGNCZSMIOXPVCV7EHQOTJKODFCWF66HIVYEBRBCE3KBPYGZNDQ, seg_pos={0 0}, upload=445128]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 18:45:10.472401204 (507h16m10.343484239s) [noise=true , address=45.91.116.10:4084 , piece_id=BPXGNCZSMIOXPVCV7EHQOTJKODFCWF66HIVYEBRBCE3KBPYGZNDQ, seg_pos={0 0}, upload=445128]: Long tail cancellation due to optimal threshold Dec 26 18:45:10.472560337 (507h16m10.343643372s) [noise=true , address=45.91.116.6:4975 , piece_id=PI5PF5PDHKRCB2Z3MW5RRN2ODGTJEUWDKJHN2G27N3W35FZCLNSQ, seg_pos={0 0}, upload=445128]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 26 18:45:10.472566609 (507h16m10.343649654s) [noise=true , address=45.91.116.6:4975 , piece_id=PI5PF5PDHKRCB2Z3MW5RRN2ODGTJEUWDKJHN2G27N3W35FZCLNSQ, seg_pos={0 0}, upload=445128]: Long tail cancellation due to optimal threshold Dec 26 18:45:10.472603990 (507h16m10.343687025s) [seg_pos={0 0}, upload=445128]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:45:10.472698009 (507h16m10.343781044s) [upload=445128]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:45:10.472703520 (507h16m10.343786565s) [upload=445128]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:45:10.776195201 (507h16m10.647278236s) [upload=445128]: Done uploading segments... Dec 26 18:45:10.776215930 (507h16m10.647298975s) [upload=445128]: Upload finished. err: <nil> Duration: 30.003657998s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X..X.....X...XX.........X.....X..X..X.X.X.....XXX.X...X.X] duration: 30.003110366s trace: https://jaeger.dev.storj.io/trace/63ff9aa686a1af2d FAIL (2025-12-26 18:44:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: OHUVWQXKB6UGIEH2NUKA2LVGJXSAX5ZEJOLEZTNOLIAH46TNYKXQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: P77CSSZZIQRER6OPNGQYJRRWSL4VIMP6BDEPDYFVSWNNEPP3UKSA) context deadline exceeded ======LOGS====== Dec 26 18:44:15.208886104 (507h15m15.079969149s) [upload=445115]: Starting upload Dec 26 18:44:15.208965374 (507h15m15.08004842s) [upload=445115]: Done starting upload Dec 26 18:44:15.209064724 (507h15m15.080147789s) [upload=445115]: Uploading segments... Dec 26 18:44:15.209289623 (507h15m15.080372658s) [upload=445115]: Got next segment. Inline: false Dec 26 18:44:15.209331071 (507h15m15.080414116s) [upload=445115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:44:15.209337644 (507h15m15.080420679s) [upload=445115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:44:15.457743202 (507h15m15.328826237s) [seg_pos={0 0}, upload=445115]: Begin upload segment... Dec 26 18:44:15.457790662 (507h15m15.328873707s) [seg_pos={0 0}, upload=445115]: Done begin upload segment. Dec 26 18:44:15.457849164 (507h15m15.328932199s) [noise=true , address=145.79.227.8:4330 , piece_id=M6HZ5ZBRMEBCEBMD7DW5BCYG6KNFCCGKIQP5IL7SFAZXVXF7BVWA, seg_pos={0 0}, upload=445115]: Uploading piece... Dec 26 18:44:15.457926440 (507h15m15.329009486s) [noise=true , address=145.79.227.21:4475 , piece_id=ROTI4PVGFTUBZVUHQFEFUFLZYHNMLIVVTR7TXBL2W7ITPUSJXJAQ, seg_pos={0 0}, upload=445115]: Uploading piece... Dec 26 18:44:15.458449486 (507h15m15.329532551s) [noise=true , address=145.79.227.49:4338 , piece_id=SMZZN6IYOSJ3V3JAR63YF2RJBHYXPAEQU2EEV6ROYEZ2N2KL534A, seg_pos={0 0}, upload=445115]: Uploading piece... Dec 26 18:44:15.459058947 (507h15m15.330141982s) [noise=true , address=145.79.227.6:4259 , piece_id=SIOOJXKRKCWWMXZRQHBC4WUZBWK2ARDBPEACTQG4FESGRMW7DI3A, seg_pos={0 0}, upload=445115]: Uploading piece... Dec 26 18:44:15.459475000 (507h15m15.330558045s) [noise=true , address=77.90.31.14:4662 , piece_id=P77CSSZZIQRER6OPNGQYJRRWSL4VIMP6BDEPDYFVSWNNEPP3UKSA, seg_pos={0 0}, upload=445115]: Uploading piece... Dec 26 18:44:15.459614335 (507h15m15.33069738s) [noise=true , address=145.79.227.52:4445 , piece_id=E2WSGI7OID2NIOD72LJAA3AN5XYIKJOZLVOXVHUNIT2YWMPYG2BQ, seg_pos={0 0}, upload=445115]: Uploading piece... Dec 26 18:44:15.459792123 (507h15m15.330875168s) [noise=true , address=145.79.227.3:4150 , piece_id=TUIMTKTOGBYQBOZRK5WHJEKA2JXVAP3SQFZOEFWXACASJJETSVIQ, seg_pos={0 0}, upload=445115]: Uploading piece... Dec 26 18:44:15.460034044 (507h15m15.331117089s) [noise=true , address=77.90.31.19:4842 , piece_id=BM7G4QDJ4GJ3YFEUAYTCTAHO7EJSMNF64SOWE4KEPGH37ONVLW5Q, seg_pos={0 0}, upload=445115]: Uploading piece... Dec 26 18:44:15.460054823 (507h15m15.331137858s) [noise=true , address=145.79.227.48:4303 , piece_id=66K3YA5R4QMC53I7NTIYSGF656RSFXM3SY7S66VK2CQ4Z62OCJ5A, seg_pos={0 0}, upload=445115]: Uploading piece... Dec 26 18:44:15.460329667 (507h15m15.331412712s) [noise=true , address=145.79.227.31:4799 , piece_id=FTLSI5GCJ26UTK4HHFS4IYGRLI7GWC2L5JNHAXY5FC5LU3J7HGPA, seg_pos={0 0}, upload=445115]: Uploading piece... Dec 26 18:44:15.460457100 (507h15m15.331540145s) [noise=true , address=77.90.31.18:4805 , piece_id=UJSGSOWKTLG6J6HNF5EYOL336BIUPGJDDMV7HUS2LWFDPAL6VFBQ, seg_pos={0 0}, upload=445115]: Uploading piece... Dec 26 18:44:15.460613256 (507h15m15.331696322s) [noise=true , address=145.79.227.38:4977 , piece_id=A2MQ3642IHFBNEUT7EKGUJOKSU6G3RBSN2MJKGXKBCCI2PZ2IAXA, seg_pos={0 0}, upload=445115]: Uploading piece... Dec 26 18:44:15.460918458 (507h15m15.332001493s) [noise=true , address=145.79.227.51:4409 , piece_id=WGQOQPPWBHJWQQM6ZFVFZ3YLEMZIJHZFO3NKII2CQGORGBMZDSXQ, seg_pos={0 0}, upload=445115]: Uploading piece... Dec 26 18:44:15.460955308 (507h15m15.332038353s) [noise=true , address=145.79.227.36:4517 , piece_id=AIBEZQ7VTXEA4HVMKGA3NPL7C3UB5Z72YCDEFPLTHBBTKJWTQRIA, seg_pos={0 0}, upload=445115]: Uploading piece... Dec 26 18:44:15.461299383 (507h15m15.332382418s) [noise=true , address=77.90.31.17:4773 , piece_id=3CCFYBXY633AFRHUKE2L6CLDT7H2FKJJ2AZDCOTKWTRZLTAG4MYQ, seg_pos={0 0}, upload=445115]: Uploading piece... Dec 26 18:44:15.461837098 (507h15m15.332920133s) [noise=true , address=45.91.116.6:4973 , piece_id=OHUVWQXKB6UGIEH2NUKA2LVGJXSAX5ZEJOLEZTNOLIAH46TNYKXQ, seg_pos={0 0}, upload=445115]: Uploading piece... Dec 26 18:44:16.633101520 (507h15m16.504184575s) [noise=true , address=77.90.31.18:4805 , piece_id=UJSGSOWKTLG6J6HNF5EYOL336BIUPGJDDMV7HUS2LWFDPAL6VFBQ, seg_pos={0 0}, upload=445115]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 18:44:17.174012709 (507h15m17.045095744s) [upload=445115]: Next returned nil segment Dec 26 18:44:17.174033810 (507h15m17.045116855s) [upload=445115]: Waiting for error group managing segments... Dec 26 18:44:17.460598208 (507h15m17.331681243s) [noise=true , address=145.79.227.38:4977 , piece_id=A2MQ3642IHFBNEUT7EKGUJOKSU6G3RBSN2MJKGXKBCCI2PZ2IAXA, seg_pos={0 0}, upload=445115]: Done uploading piece. err: <nil> Dec 26 18:44:17.461129670 (507h15m17.332212715s) [noise=true , address=145.79.227.36:4517 , piece_id=AIBEZQ7VTXEA4HVMKGA3NPL7C3UB5Z72YCDEFPLTHBBTKJWTQRIA, seg_pos={0 0}, upload=445115]: Done uploading piece. err: <nil> Dec 26 18:44:17.462491123 (507h15m17.333574158s) [noise=true , address=145.79.227.8:4330 , piece_id=M6HZ5ZBRMEBCEBMD7DW5BCYG6KNFCCGKIQP5IL7SFAZXVXF7BVWA, seg_pos={0 0}, upload=445115]: Done uploading piece. err: <nil> Dec 26 18:44:17.463556631 (507h15m17.334639676s) [noise=true , address=145.79.227.21:4475 , piece_id=ROTI4PVGFTUBZVUHQFEFUFLZYHNMLIVVTR7TXBL2W7ITPUSJXJAQ, seg_pos={0 0}, upload=445115]: Done uploading piece. err: <nil> Dec 26 18:44:17.463975359 (507h15m17.335058404s) [noise=true , address=145.79.227.49:4338 , piece_id=SMZZN6IYOSJ3V3JAR63YF2RJBHYXPAEQU2EEV6ROYEZ2N2KL534A, seg_pos={0 0}, upload=445115]: Done uploading piece. err: <nil> Dec 26 18:44:17.464824676 (507h15m17.335907722s) [noise=true , address=145.79.227.52:4445 , piece_id=E2WSGI7OID2NIOD72LJAA3AN5XYIKJOZLVOXVHUNIT2YWMPYG2BQ, seg_pos={0 0}, upload=445115]: Done uploading piece. err: <nil> Dec 26 18:44:17.468356231 (507h15m17.339439266s) [noise=true , address=145.79.227.31:4799 , piece_id=FTLSI5GCJ26UTK4HHFS4IYGRLI7GWC2L5JNHAXY5FC5LU3J7HGPA, seg_pos={0 0}, upload=445115]: Done uploading piece. err: <nil> Dec 26 18:44:17.469056394 (507h15m17.340139429s) [noise=true , address=145.79.227.3:4150 , piece_id=TUIMTKTOGBYQBOZRK5WHJEKA2JXVAP3SQFZOEFWXACASJJETSVIQ, seg_pos={0 0}, upload=445115]: Done uploading piece. err: <nil> Dec 26 18:44:17.488705580 (507h15m17.359788625s) [noise=true , address=145.79.227.48:4303 , piece_id=66K3YA5R4QMC53I7NTIYSGF656RSFXM3SY7S66VK2CQ4Z62OCJ5A, seg_pos={0 0}, upload=445115]: Done uploading piece. err: <nil> Dec 26 18:44:17.514482253 (507h15m17.385565288s) [noise=true , address=145.79.227.51:4409 , piece_id=WGQOQPPWBHJWQQM6ZFVFZ3YLEMZIJHZFO3NKII2CQGORGBMZDSXQ, seg_pos={0 0}, upload=445115]: Done uploading piece. err: <nil> Dec 26 18:44:18.320957483 (507h15m18.192040528s) [noise=true , address=77.90.31.14:4662 , piece_id=P77CSSZZIQRER6OPNGQYJRRWSL4VIMP6BDEPDYFVSWNNEPP3UKSA, seg_pos={0 0}, upload=445115]: Done uploading piece. err: <nil> Dec 26 18:44:18.761002355 (507h15m18.632085391s) [noise=true , address=45.91.116.6:4973 , piece_id=OHUVWQXKB6UGIEH2NUKA2LVGJXSAX5ZEJOLEZTNOLIAH46TNYKXQ, seg_pos={0 0}, upload=445115]: Done uploading piece. err: <nil> Dec 26 18:44:18.782304008 (507h15m18.653387043s) [noise=true , address=77.90.31.19:4842 , piece_id=BM7G4QDJ4GJ3YFEUAYTCTAHO7EJSMNF64SOWE4KEPGH37ONVLW5Q, seg_pos={0 0}, upload=445115]: Done uploading piece. err: <nil> Dec 26 18:44:18.782323554 (507h15m18.653406589s) [seg_pos={0 0}, upload=445115]: Segment reached optimal threshold of 13 pieces. Dec 26 18:44:18.782545858 (507h15m18.653628903s) [noise=true , address=145.79.227.6:4259 , piece_id=SIOOJXKRKCWWMXZRQHBC4WUZBWK2ARDBPEACTQG4FESGRMW7DI3A, seg_pos={0 0}, upload=445115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:60562->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:60562->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60562->145.79.227.6:4259: use of closed network connection Dec 26 18:44:18.782554164 (507h15m18.653637199s) [noise=true , address=145.79.227.6:4259 , piece_id=SIOOJXKRKCWWMXZRQHBC4WUZBWK2ARDBPEACTQG4FESGRMW7DI3A, seg_pos={0 0}, upload=445115]: Long tail cancellation due to optimal threshold Dec 26 18:44:18.783064977 (507h15m18.654148012s) [noise=true , address=77.90.31.17:4773 , piece_id=3CCFYBXY633AFRHUKE2L6CLDT7H2FKJJ2AZDCOTKWTRZLTAG4MYQ, seg_pos={0 0}, upload=445115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 18:44:18.783082840 (507h15m18.654165875s) [noise=true , address=77.90.31.17:4773 , piece_id=3CCFYBXY633AFRHUKE2L6CLDT7H2FKJJ2AZDCOTKWTRZLTAG4MYQ, seg_pos={0 0}, upload=445115]: Long tail cancellation due to optimal threshold Dec 26 18:44:18.783101506 (507h15m18.654184551s) [seg_pos={0 0}, upload=445115]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:44:18.783177491 (507h15m18.654260526s) [upload=445115]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:44:18.783182250 (507h15m18.654265285s) [upload=445115]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:44:19.082971351 (507h15m18.954054396s) [upload=445115]: Done uploading segments... Dec 26 18:44:19.083007299 (507h15m18.954090334s) [upload=445115]: Upload finished. err: <nil> Duration: 30.003135775s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X..X.....X...XX.........X.....X..X..X.X.X.....XXX.X...X.] duration: 28.119702827s trace: https://jaeger.dev.storj.io/trace/5119b890e6f248c1 FAIL (2025-12-26 18:44:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: LKWNKUYPZMYEJXHN3UMHRAV5IPRGXGXXCAWPUBIYHUP6MY7OSOZA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: YRUMG65DWEHD5OZQK4QURE25GZLIF2IL4V7BW3QVKPHBUKOX4F6Q) context deadline exceeded ======LOGS====== Dec 26 18:42:32.313619830 (507h13m32.184702865s) [upload=445087]: Starting upload Dec 26 18:42:32.313692559 (507h13m32.184775614s) [upload=445087]: Done starting upload Dec 26 18:42:32.313837354 (507h13m32.184920389s) [upload=445087]: Uploading segments... Dec 26 18:42:32.314011366 (507h13m32.185094411s) [upload=445087]: Got next segment. Inline: false Dec 26 18:42:32.314072893 (507h13m32.185155928s) [upload=445087]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:42:32.314080016 (507h13m32.185163061s) [upload=445087]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:42:32.572703461 (507h13m32.443786506s) [seg_pos={0 0}, upload=445087]: Begin upload segment... Dec 26 18:42:32.572763405 (507h13m32.44384645s) [seg_pos={0 0}, upload=445087]: Done begin upload segment. Dec 26 18:42:32.572801848 (507h13m32.443884893s) [noise=true , address=77.90.31.10:4518 , piece_id=HSRHICABEFAAMK2OCMDYJWCDEASF4E6C5TUX4HA4CSRP4J75IZ4Q, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.572811055 (507h13m32.4438941s) [noise=true , address=145.79.227.21:4474 , piece_id=UYJZBUJNNRG5IPC7X4F4BNQ5IOGUNYMWBZPPQLRD245XR7WLC2KQ, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.573612732 (507h13m32.444695767s) [noise=true , address=45.91.116.9:4261 , piece_id=WX7CNGE2BH6UHZPOWE4TSM7FUSWX7DEGT37UUZUAK3FDJYWCMD3Q, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.573704898 (507h13m32.444787943s) [noise=true , address=145.79.227.39:4013 , piece_id=PZDC3XSCTXAOMWM2HXEWERWHMUQHJY5WV6S3P33T5Z6XT72MGXWA, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.574063701 (507h13m32.445146736s) [noise=true , address=145.79.227.46:4602 , piece_id=47WQSQ6BO4P5UU5XELBPBLLAKQM3LOZIXO5ZD7OYWMSASEJ2DLCQ, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.574396515 (507h13m32.44547955s) [noise=true , address=145.79.227.51:4410 , piece_id=DYEDU7XYQIL7VTA2KWUBGXKK4K6IMUOEQL7NSLB4P3YJPMFQEBSQ, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.574773282 (507h13m32.445856317s) [noise=true , address=45.91.116.7:4010 , piece_id=MLU2QLRZIDGLA6E7VU7OFCEH6FPDMSHKTQKWICEBB5YN6S7XOMOQ, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.575116515 (507h13m32.44619956s) [noise=true , address=145.79.227.14:4839 , piece_id=2Q4RS3TX63BQLQYJYFWUOYCDRZYM3VJ3QV62WONLK4MMRPKUVCAA, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.575454449 (507h13m32.446537494s) [noise=true , address=145.79.227.47:4265 , piece_id=V4LSML2UQXNAXBV5P44H2T4OA27Z2LJAHLCQQLD2L5FXPUXFAW4Q, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.575804325 (507h13m32.44688736s) [noise=true , address=145.79.227.34:4905 , piece_id=4SZ5DCUJYPJERESB3GRUID6GKVROU7UQGCEG272YJTELNFD4NECQ, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.576127600 (507h13m32.447210635s) [noise=true , address=145.79.227.42:4122 , piece_id=NORSJGGYYBNABLNLGIFINK7JVICMI24X7YCVJWZIYBEC63WZYU5Q, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.578128852 (507h13m32.449211887s) [noise=true , address=145.79.227.43:4158 , piece_id=7ZZUR7VDEPNRNOWO5ASJHNH22UTDXFNTYZOM32GBS7PCHUGQNRUA, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.578475281 (507h13m32.449558326s) [noise=true , address=45.91.116.14:4190 , piece_id=LKWNKUYPZMYEJXHN3UMHRAV5IPRGXGXXCAWPUBIYHUP6MY7OSOZA, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.578830978 (507h13m32.449914024s) [noise=true , address=145.79.227.50:4373 , piece_id=5UN53UUJKNM5ZCZ7NWVYGZT2O7HTHE4YGW6AQ7KIHH4QIVNHDNMQ, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.579160416 (507h13m32.450243451s) [noise=true , address=45.91.116.8:4046 , piece_id=YRUMG65DWEHD5OZQK4QURE25GZLIF2IL4V7BW3QVKPHBUKOX4F6Q, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.579463863 (507h13m32.450546898s) [noise=true , address=145.79.227.48:4303 , piece_id=XYJBMN2CAQFUDCFXE7QAD3KNNGZNK2SNUBMGNIJL7OO3SUBRIV6A, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:34.215891879 (507h13m34.086974914s) [upload=445087]: Next returned nil segment Dec 26 18:42:34.215950430 (507h13m34.087033465s) [upload=445087]: Waiting for error group managing segments... Dec 26 18:42:34.562976419 (507h13m34.434059454s) [noise=true , address=145.79.227.51:4410 , piece_id=DYEDU7XYQIL7VTA2KWUBGXKK4K6IMUOEQL7NSLB4P3YJPMFQEBSQ, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:34.581924499 (507h13m34.453007544s) [noise=true , address=145.79.227.46:4602 , piece_id=47WQSQ6BO4P5UU5XELBPBLLAKQM3LOZIXO5ZD7OYWMSASEJ2DLCQ, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:34.588214426 (507h13m34.459297461s) [noise=true , address=145.79.227.50:4373 , piece_id=5UN53UUJKNM5ZCZ7NWVYGZT2O7HTHE4YGW6AQ7KIHH4QIVNHDNMQ, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:34.601773324 (507h13m34.472856359s) [noise=true , address=145.79.227.14:4839 , piece_id=2Q4RS3TX63BQLQYJYFWUOYCDRZYM3VJ3QV62WONLK4MMRPKUVCAA, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:34.604062253 (507h13m34.475145318s) [noise=true , address=145.79.227.42:4122 , piece_id=NORSJGGYYBNABLNLGIFINK7JVICMI24X7YCVJWZIYBEC63WZYU5Q, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:34.604135532 (507h13m34.475218568s) [noise=true , address=145.79.227.48:4303 , piece_id=XYJBMN2CAQFUDCFXE7QAD3KNNGZNK2SNUBMGNIJL7OO3SUBRIV6A, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:34.611816759 (507h13m34.482899794s) [noise=true , address=145.79.227.21:4474 , piece_id=UYJZBUJNNRG5IPC7X4F4BNQ5IOGUNYMWBZPPQLRD245XR7WLC2KQ, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:34.620146640 (507h13m34.491229676s) [noise=true , address=145.79.227.39:4013 , piece_id=PZDC3XSCTXAOMWM2HXEWERWHMUQHJY5WV6S3P33T5Z6XT72MGXWA, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:34.863106418 (507h13m34.734189463s) [noise=true , address=45.91.116.8:4046 , piece_id=YRUMG65DWEHD5OZQK4QURE25GZLIF2IL4V7BW3QVKPHBUKOX4F6Q, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:34.912668363 (507h13m34.783751398s) [noise=true , address=45.91.116.9:4261 , piece_id=WX7CNGE2BH6UHZPOWE4TSM7FUSWX7DEGT37UUZUAK3FDJYWCMD3Q, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:34.979565106 (507h13m34.850648141s) [noise=true , address=145.79.227.47:4265 , piece_id=V4LSML2UQXNAXBV5P44H2T4OA27Z2LJAHLCQQLD2L5FXPUXFAW4Q, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:35.482188419 (507h13m35.353271464s) [noise=true , address=45.91.116.14:4190 , piece_id=LKWNKUYPZMYEJXHN3UMHRAV5IPRGXGXXCAWPUBIYHUP6MY7OSOZA, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:35.561573893 (507h13m35.432656928s) [noise=true , address=45.91.116.7:4010 , piece_id=MLU2QLRZIDGLA6E7VU7OFCEH6FPDMSHKTQKWICEBB5YN6S7XOMOQ, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:35.561593680 (507h13m35.432676725s) [seg_pos={0 0}, upload=445087]: Segment reached optimal threshold of 13 pieces. Dec 26 18:42:35.561770657 (507h13m35.432853692s) [noise=true , address=145.79.227.34:4905 , piece_id=4SZ5DCUJYPJERESB3GRUID6GKVROU7UQGCEG272YJTELNFD4NECQ, seg_pos={0 0}, upload=445087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 26 18:42:35.561777390 (507h13m35.432860425s) [noise=true , address=145.79.227.34:4905 , piece_id=4SZ5DCUJYPJERESB3GRUID6GKVROU7UQGCEG272YJTELNFD4NECQ, seg_pos={0 0}, upload=445087]: Long tail cancellation due to optimal threshold Dec 26 18:42:35.561826344 (507h13m35.432909379s) [noise=true , address=145.79.227.43:4158 , piece_id=7ZZUR7VDEPNRNOWO5ASJHNH22UTDXFNTYZOM32GBS7PCHUGQNRUA, seg_pos={0 0}, upload=445087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 26 18:42:35.561850741 (507h13m35.432933776s) [noise=true , address=145.79.227.43:4158 , piece_id=7ZZUR7VDEPNRNOWO5ASJHNH22UTDXFNTYZOM32GBS7PCHUGQNRUA, seg_pos={0 0}, upload=445087]: Long tail cancellation due to optimal threshold Dec 26 18:42:35.561971009 (507h13m35.433054054s) [noise=true , address=77.90.31.10:4518 , piece_id=HSRHICABEFAAMK2OCMDYJWCDEASF4E6C5TUX4HA4CSRP4J75IZ4Q, seg_pos={0 0}, upload=445087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 26 18:42:35.561977311 (507h13m35.433060346s) [noise=true , address=77.90.31.10:4518 , piece_id=HSRHICABEFAAMK2OCMDYJWCDEASF4E6C5TUX4HA4CSRP4J75IZ4Q, seg_pos={0 0}, upload=445087]: Long tail cancellation due to optimal threshold Dec 26 18:42:35.561990777 (507h13m35.433073812s) [seg_pos={0 0}, upload=445087]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:42:35.562054688 (507h13m35.433137723s) [upload=445087]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:42:35.562059328 (507h13m35.433142363s) [upload=445087]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:42:35.852075588 (507h13m35.723158624s) [upload=445087]: Done uploading segments... Dec 26 18:42:35.852096749 (507h13m35.723179784s) [upload=445087]: Upload finished. err: <nil> Duration: 28.119732464s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.....X..X.....X...XX.........X.....X..X..X.X.X.....XXX.X...X] duration: 30.00430522s trace: https://jaeger.dev.storj.io/trace/5119b890e6f248c1 FAIL (2025-12-26 18:43:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: LKWNKUYPZMYEJXHN3UMHRAV5IPRGXGXXCAWPUBIYHUP6MY7OSOZA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: YRUMG65DWEHD5OZQK4QURE25GZLIF2IL4V7BW3QVKPHBUKOX4F6Q) context deadline exceeded ======LOGS====== Dec 26 18:42:32.313619830 (507h13m32.184702865s) [upload=445087]: Starting upload Dec 26 18:42:32.313692559 (507h13m32.184775614s) [upload=445087]: Done starting upload Dec 26 18:42:32.313837354 (507h13m32.184920389s) [upload=445087]: Uploading segments... Dec 26 18:42:32.314011366 (507h13m32.185094411s) [upload=445087]: Got next segment. Inline: false Dec 26 18:42:32.314072893 (507h13m32.185155928s) [upload=445087]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:42:32.314080016 (507h13m32.185163061s) [upload=445087]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:42:32.572703461 (507h13m32.443786506s) [seg_pos={0 0}, upload=445087]: Begin upload segment... Dec 26 18:42:32.572763405 (507h13m32.44384645s) [seg_pos={0 0}, upload=445087]: Done begin upload segment. Dec 26 18:42:32.572801848 (507h13m32.443884893s) [noise=true , address=77.90.31.10:4518 , piece_id=HSRHICABEFAAMK2OCMDYJWCDEASF4E6C5TUX4HA4CSRP4J75IZ4Q, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.572811055 (507h13m32.4438941s) [noise=true , address=145.79.227.21:4474 , piece_id=UYJZBUJNNRG5IPC7X4F4BNQ5IOGUNYMWBZPPQLRD245XR7WLC2KQ, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.573612732 (507h13m32.444695767s) [noise=true , address=45.91.116.9:4261 , piece_id=WX7CNGE2BH6UHZPOWE4TSM7FUSWX7DEGT37UUZUAK3FDJYWCMD3Q, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.573704898 (507h13m32.444787943s) [noise=true , address=145.79.227.39:4013 , piece_id=PZDC3XSCTXAOMWM2HXEWERWHMUQHJY5WV6S3P33T5Z6XT72MGXWA, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.574063701 (507h13m32.445146736s) [noise=true , address=145.79.227.46:4602 , piece_id=47WQSQ6BO4P5UU5XELBPBLLAKQM3LOZIXO5ZD7OYWMSASEJ2DLCQ, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.574396515 (507h13m32.44547955s) [noise=true , address=145.79.227.51:4410 , piece_id=DYEDU7XYQIL7VTA2KWUBGXKK4K6IMUOEQL7NSLB4P3YJPMFQEBSQ, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.574773282 (507h13m32.445856317s) [noise=true , address=45.91.116.7:4010 , piece_id=MLU2QLRZIDGLA6E7VU7OFCEH6FPDMSHKTQKWICEBB5YN6S7XOMOQ, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.575116515 (507h13m32.44619956s) [noise=true , address=145.79.227.14:4839 , piece_id=2Q4RS3TX63BQLQYJYFWUOYCDRZYM3VJ3QV62WONLK4MMRPKUVCAA, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.575454449 (507h13m32.446537494s) [noise=true , address=145.79.227.47:4265 , piece_id=V4LSML2UQXNAXBV5P44H2T4OA27Z2LJAHLCQQLD2L5FXPUXFAW4Q, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.575804325 (507h13m32.44688736s) [noise=true , address=145.79.227.34:4905 , piece_id=4SZ5DCUJYPJERESB3GRUID6GKVROU7UQGCEG272YJTELNFD4NECQ, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.576127600 (507h13m32.447210635s) [noise=true , address=145.79.227.42:4122 , piece_id=NORSJGGYYBNABLNLGIFINK7JVICMI24X7YCVJWZIYBEC63WZYU5Q, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.578128852 (507h13m32.449211887s) [noise=true , address=145.79.227.43:4158 , piece_id=7ZZUR7VDEPNRNOWO5ASJHNH22UTDXFNTYZOM32GBS7PCHUGQNRUA, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.578475281 (507h13m32.449558326s) [noise=true , address=45.91.116.14:4190 , piece_id=LKWNKUYPZMYEJXHN3UMHRAV5IPRGXGXXCAWPUBIYHUP6MY7OSOZA, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.578830978 (507h13m32.449914024s) [noise=true , address=145.79.227.50:4373 , piece_id=5UN53UUJKNM5ZCZ7NWVYGZT2O7HTHE4YGW6AQ7KIHH4QIVNHDNMQ, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.579160416 (507h13m32.450243451s) [noise=true , address=45.91.116.8:4046 , piece_id=YRUMG65DWEHD5OZQK4QURE25GZLIF2IL4V7BW3QVKPHBUKOX4F6Q, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:32.579463863 (507h13m32.450546898s) [noise=true , address=145.79.227.48:4303 , piece_id=XYJBMN2CAQFUDCFXE7QAD3KNNGZNK2SNUBMGNIJL7OO3SUBRIV6A, seg_pos={0 0}, upload=445087]: Uploading piece... Dec 26 18:42:34.215891879 (507h13m34.086974914s) [upload=445087]: Next returned nil segment Dec 26 18:42:34.215950430 (507h13m34.087033465s) [upload=445087]: Waiting for error group managing segments... Dec 26 18:42:34.562976419 (507h13m34.434059454s) [noise=true , address=145.79.227.51:4410 , piece_id=DYEDU7XYQIL7VTA2KWUBGXKK4K6IMUOEQL7NSLB4P3YJPMFQEBSQ, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:34.581924499 (507h13m34.453007544s) [noise=true , address=145.79.227.46:4602 , piece_id=47WQSQ6BO4P5UU5XELBPBLLAKQM3LOZIXO5ZD7OYWMSASEJ2DLCQ, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:34.588214426 (507h13m34.459297461s) [noise=true , address=145.79.227.50:4373 , piece_id=5UN53UUJKNM5ZCZ7NWVYGZT2O7HTHE4YGW6AQ7KIHH4QIVNHDNMQ, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:34.601773324 (507h13m34.472856359s) [noise=true , address=145.79.227.14:4839 , piece_id=2Q4RS3TX63BQLQYJYFWUOYCDRZYM3VJ3QV62WONLK4MMRPKUVCAA, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:34.604062253 (507h13m34.475145318s) [noise=true , address=145.79.227.42:4122 , piece_id=NORSJGGYYBNABLNLGIFINK7JVICMI24X7YCVJWZIYBEC63WZYU5Q, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:34.604135532 (507h13m34.475218568s) [noise=true , address=145.79.227.48:4303 , piece_id=XYJBMN2CAQFUDCFXE7QAD3KNNGZNK2SNUBMGNIJL7OO3SUBRIV6A, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:34.611816759 (507h13m34.482899794s) [noise=true , address=145.79.227.21:4474 , piece_id=UYJZBUJNNRG5IPC7X4F4BNQ5IOGUNYMWBZPPQLRD245XR7WLC2KQ, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:34.620146640 (507h13m34.491229676s) [noise=true , address=145.79.227.39:4013 , piece_id=PZDC3XSCTXAOMWM2HXEWERWHMUQHJY5WV6S3P33T5Z6XT72MGXWA, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:34.863106418 (507h13m34.734189463s) [noise=true , address=45.91.116.8:4046 , piece_id=YRUMG65DWEHD5OZQK4QURE25GZLIF2IL4V7BW3QVKPHBUKOX4F6Q, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:34.912668363 (507h13m34.783751398s) [noise=true , address=45.91.116.9:4261 , piece_id=WX7CNGE2BH6UHZPOWE4TSM7FUSWX7DEGT37UUZUAK3FDJYWCMD3Q, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:34.979565106 (507h13m34.850648141s) [noise=true , address=145.79.227.47:4265 , piece_id=V4LSML2UQXNAXBV5P44H2T4OA27Z2LJAHLCQQLD2L5FXPUXFAW4Q, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:35.482188419 (507h13m35.353271464s) [noise=true , address=45.91.116.14:4190 , piece_id=LKWNKUYPZMYEJXHN3UMHRAV5IPRGXGXXCAWPUBIYHUP6MY7OSOZA, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:35.561573893 (507h13m35.432656928s) [noise=true , address=45.91.116.7:4010 , piece_id=MLU2QLRZIDGLA6E7VU7OFCEH6FPDMSHKTQKWICEBB5YN6S7XOMOQ, seg_pos={0 0}, upload=445087]: Done uploading piece. err: <nil> Dec 26 18:42:35.561593680 (507h13m35.432676725s) [seg_pos={0 0}, upload=445087]: Segment reached optimal threshold of 13 pieces. Dec 26 18:42:35.561770657 (507h13m35.432853692s) [noise=true , address=145.79.227.34:4905 , piece_id=4SZ5DCUJYPJERESB3GRUID6GKVROU7UQGCEG272YJTELNFD4NECQ, seg_pos={0 0}, upload=445087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 26 18:42:35.561777390 (507h13m35.432860425s) [noise=true , address=145.79.227.34:4905 , piece_id=4SZ5DCUJYPJERESB3GRUID6GKVROU7UQGCEG272YJTELNFD4NECQ, seg_pos={0 0}, upload=445087]: Long tail cancellation due to optimal threshold Dec 26 18:42:35.561826344 (507h13m35.432909379s) [noise=true , address=145.79.227.43:4158 , piece_id=7ZZUR7VDEPNRNOWO5ASJHNH22UTDXFNTYZOM32GBS7PCHUGQNRUA, seg_pos={0 0}, upload=445087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 26 18:42:35.561850741 (507h13m35.432933776s) [noise=true , address=145.79.227.43:4158 , piece_id=7ZZUR7VDEPNRNOWO5ASJHNH22UTDXFNTYZOM32GBS7PCHUGQNRUA, seg_pos={0 0}, upload=445087]: Long tail cancellation due to optimal threshold Dec 26 18:42:35.561971009 (507h13m35.433054054s) [noise=true , address=77.90.31.10:4518 , piece_id=HSRHICABEFAAMK2OCMDYJWCDEASF4E6C5TUX4HA4CSRP4J75IZ4Q, seg_pos={0 0}, upload=445087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 26 18:42:35.561977311 (507h13m35.433060346s) [noise=true , address=77.90.31.10:4518 , piece_id=HSRHICABEFAAMK2OCMDYJWCDEASF4E6C5TUX4HA4CSRP4J75IZ4Q, seg_pos={0 0}, upload=445087]: Long tail cancellation due to optimal threshold Dec 26 18:42:35.561990777 (507h13m35.433073812s) [seg_pos={0 0}, upload=445087]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:42:35.562054688 (507h13m35.433137723s) [upload=445087]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:42:35.562059328 (507h13m35.433142363s) [upload=445087]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:42:35.852075588 (507h13m35.723158624s) [upload=445087]: Done uploading segments... Dec 26 18:42:35.852096749 (507h13m35.723179784s) [upload=445087]: Upload finished. err: <nil> Duration: 30.004329317s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [......X..X.....X...XX.........X.....X..X..X.X.X.....XXX.X...] duration: 6.327410939s trace: https://jaeger.dev.storj.io/trace/4d0231720e50e740 FAIL (2025-12-26 18:41:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 18:38:31.157821160 (507h9m31.028904205s) [upload=445034]: Starting upload Dec 26 18:38:31.157877146 (507h9m31.028960191s) [upload=445034]: Done starting upload Dec 26 18:38:31.157942992 (507h9m31.029026037s) [upload=445034]: Uploading segments... Dec 26 18:38:31.158059945 (507h9m31.02914298s) [upload=445034]: Got next segment. Inline: false Dec 26 18:38:31.158089972 (507h9m31.029173018s) [upload=445034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:38:31.158094451 (507h9m31.029177486s) [upload=445034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:38:31.404855186 (507h9m31.275938231s) [seg_pos={0 0}, upload=445034]: Begin upload segment... Dec 26 18:38:31.404909629 (507h9m31.275992664s) [seg_pos={0 0}, upload=445034]: Done begin upload segment. Dec 26 18:38:31.404939226 (507h9m31.276022261s) [noise=true , address=45.91.116.8:4048 , piece_id=DUK7YWADLEYMV2YVL3R6FE6MKVNI3EFECHIXONPRU3K2JIZBP3VA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.405917509 (507h9m31.277000554s) [noise=true , address=145.79.227.36:4519 , piece_id=CYT4FFVVOTUIKTHMRM5VIJ4O5QR6AAB3MMZGQJ2DHLHV7BSHXC3Q, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.406590080 (507h9m31.277673115s) [noise=true , address=145.79.227.42:4121 , piece_id=4U7MBYKX2BMTHU3FPXNWQONZLPAH5I5REX4MD2IGPEK6DXG6COAA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.407173159 (507h9m31.278256195s) [noise=true , address=145.79.227.46:4601 , piece_id=JS6TZ4ILXJ6ERH3HWMTSDNWRSPCTP25ND7UIFO7B7CDHHHAQKE4Q, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.407546741 (507h9m31.278629786s) [noise=true , address=45.91.116.11:4120 , piece_id=PYWGMEJ3SJ56CE4WW6N22XQUK5ICC2LJC5ICEHDNA7RP44SU5ZJA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.407938116 (507h9m31.279021151s) [noise=true , address=145.79.227.52:4445 , piece_id=ZPIMEROEBTNLTCAQF42GIRND2LZPUFECG5QP2GPJUUHBG47BFTOA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.408758859 (507h9m31.279841894s) [noise=true , address=145.79.227.32:4836 , piece_id=2R6WOS7NHLSL3IECRA4DTPQCA2AIGJE4VZWR472QHUXBYGWED45Q, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.409240627 (507h9m31.280323662s) [noise=true , address=145.79.227.45:4229 , piece_id=YWGQ3PU6GNCTDD3ECFONV2GROLUZNRPCGHJN7UP7ZDJITO5N56FA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.409752571 (507h9m31.280835626s) [noise=true , address=77.90.31.12:4589 , piece_id=4IGKBXF3SOCISAXEAKCSWWZ66BLIQ7NQ5L4YYMITLMCFFDSK3DXA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.410632658 (507h9m31.281715693s) [noise=true , address=145.79.227.16:4909 , piece_id=4CQ3ZCDAGEWPQOM4VANX45TTWFOGKJRXWVAUNUR6FD4EZHZMK2SQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.410855772 (507h9m31.281938817s) [noise=true , address=45.91.116.12:4227 , piece_id=X6LTBYJNVPHRDRKFTYTDDY6N33YWIX3JCME6JLIDFKI52YKSRPLQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.411905451 (507h9m31.282988516s) [noise=true , address=77.90.31.13:4627 , piece_id=46RM6UI6UNSEYAYTYRE5RTI6TTJSMKZQVI2OOK2FVYBWM5T6X4JA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.412072970 (507h9m31.283156015s) [noise=true , address=145.79.227.29:4943 , piece_id=L5TE5QVCMG6IZ5FE7HOXRBWJMUAI3QA6D6JOCH52AEGIX7JU3XDQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.413211528 (507h9m31.284294563s) [noise=true , address=45.91.116.14:4190 , piece_id=RSD64GG2KX27FKVD5XXRA4I3TFEA76ZNBY5HYTPKPLYLPCKGVUKA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.414546780 (507h9m31.285629815s) [noise=true , address=145.79.227.2:4114 , piece_id=36QOULUKE6SBVZIKHJBIPB36OBVOFXAJZ6DVNJHTQVM4FIEMNFOA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.415591630 (507h9m31.286674665s) [noise=true , address=145.79.227.51:4409 , piece_id=2I5ZV3TP3NXSZS7KVH4JBSO5VBWQGKRPDS3MFRR2YLH3TLM33SSQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:32.593710866 (507h9m32.464793931s) [noise=true , address=77.90.31.12:4589 , piece_id=4IGKBXF3SOCISAXEAKCSWWZ66BLIQ7NQ5L4YYMITLMCFFDSK3DXA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 18:38:33.121407613 (507h9m32.992490648s) [upload=445034]: Next returned nil segment Dec 26 18:38:33.121426890 (507h9m32.992509925s) [upload=445034]: Waiting for error group managing segments... Dec 26 18:38:33.413463338 (507h9m33.284546373s) [noise=true , address=145.79.227.45:4229 , piece_id=YWGQ3PU6GNCTDD3ECFONV2GROLUZNRPCGHJN7UP7ZDJITO5N56FA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.414417395 (507h9m33.28550043s) [noise=true , address=145.79.227.46:4601 , piece_id=JS6TZ4ILXJ6ERH3HWMTSDNWRSPCTP25ND7UIFO7B7CDHHHAQKE4Q, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.423567389 (507h9m33.294650424s) [noise=true , address=145.79.227.32:4836 , piece_id=2R6WOS7NHLSL3IECRA4DTPQCA2AIGJE4VZWR472QHUXBYGWED45Q, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.432953442 (507h9m33.304036477s) [noise=true , address=145.79.227.52:4445 , piece_id=ZPIMEROEBTNLTCAQF42GIRND2LZPUFECG5QP2GPJUUHBG47BFTOA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.436006996 (507h9m33.307090031s) [noise=true , address=145.79.227.51:4409 , piece_id=2I5ZV3TP3NXSZS7KVH4JBSO5VBWQGKRPDS3MFRR2YLH3TLM33SSQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.436422918 (507h9m33.307505953s) [noise=true , address=145.79.227.36:4519 , piece_id=CYT4FFVVOTUIKTHMRM5VIJ4O5QR6AAB3MMZGQJ2DHLHV7BSHXC3Q, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.436849921 (507h9m33.307932966s) [noise=true , address=145.79.227.42:4121 , piece_id=4U7MBYKX2BMTHU3FPXNWQONZLPAH5I5REX4MD2IGPEK6DXG6COAA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.449704228 (507h9m33.320787263s) [noise=true , address=145.79.227.29:4943 , piece_id=L5TE5QVCMG6IZ5FE7HOXRBWJMUAI3QA6D6JOCH52AEGIX7JU3XDQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.460416667 (507h9m33.331499712s) [noise=true , address=145.79.227.16:4909 , piece_id=4CQ3ZCDAGEWPQOM4VANX45TTWFOGKJRXWVAUNUR6FD4EZHZMK2SQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.517087077 (507h9m33.388170112s) [noise=true , address=145.79.227.2:4114 , piece_id=36QOULUKE6SBVZIKHJBIPB36OBVOFXAJZ6DVNJHTQVM4FIEMNFOA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.600393006 (507h9m33.471476051s) [noise=true , address=77.90.31.13:4627 , piece_id=46RM6UI6UNSEYAYTYRE5RTI6TTJSMKZQVI2OOK2FVYBWM5T6X4JA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.662483417 (507h9m33.533566472s) [noise=true , address=45.91.116.14:4190 , piece_id=RSD64GG2KX27FKVD5XXRA4I3TFEA76ZNBY5HYTPKPLYLPCKGVUKA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.783839309 (507h9m33.654922344s) [noise=true , address=45.91.116.8:4048 , piece_id=DUK7YWADLEYMV2YVL3R6FE6MKVNI3EFECHIXONPRU3K2JIZBP3VA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.783857174 (507h9m33.654940209s) [seg_pos={0 0}, upload=445034]: Segment reached optimal threshold of 13 pieces. Dec 26 18:38:33.784044630 (507h9m33.655127665s) [noise=true , address=45.91.116.11:4120 , piece_id=PYWGMEJ3SJ56CE4WW6N22XQUK5ICC2LJC5ICEHDNA7RP44SU5ZJA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 18:38:33.784050812 (507h9m33.655133847s) [noise=true , address=45.91.116.11:4120 , piece_id=PYWGMEJ3SJ56CE4WW6N22XQUK5ICC2LJC5ICEHDNA7RP44SU5ZJA, seg_pos={0 0}, upload=445034]: Long tail cancellation due to optimal threshold Dec 26 18:38:33.784255181 (507h9m33.655338216s) [noise=true , address=45.91.116.12:4227 , piece_id=X6LTBYJNVPHRDRKFTYTDDY6N33YWIX3JCME6JLIDFKI52YKSRPLQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:51964->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51964->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51964->45.91.116.12:4227: use of closed network connection Dec 26 18:38:33.784269758 (507h9m33.655352794s) [noise=true , address=45.91.116.12:4227 , piece_id=X6LTBYJNVPHRDRKFTYTDDY6N33YWIX3JCME6JLIDFKI52YKSRPLQ, seg_pos={0 0}, upload=445034]: Long tail cancellation due to optimal threshold Dec 26 18:38:33.784299335 (507h9m33.65538237s) [seg_pos={0 0}, upload=445034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:38:33.784380079 (507h9m33.655463124s) [upload=445034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:38:33.784384908 (507h9m33.655467953s) [upload=445034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:38:34.099088118 (507h9m33.970171153s) [upload=445034]: Done uploading segments... Dec 26 18:38:34.099106454 (507h9m33.970189499s) [upload=445034]: Upload finished. err: <nil> Duration: 6.327437288s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.......X..X.....X...XX.........X.....X..X..X.X.X.....XXX.X..] duration: 16.362309542s trace: https://jaeger.dev.storj.io/trace/4d0231720e50e740 FAIL (2025-12-26 18:40:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 18:38:31.157821160 (507h9m31.028904205s) [upload=445034]: Starting upload Dec 26 18:38:31.157877146 (507h9m31.028960191s) [upload=445034]: Done starting upload Dec 26 18:38:31.157942992 (507h9m31.029026037s) [upload=445034]: Uploading segments... Dec 26 18:38:31.158059945 (507h9m31.02914298s) [upload=445034]: Got next segment. Inline: false Dec 26 18:38:31.158089972 (507h9m31.029173018s) [upload=445034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:38:31.158094451 (507h9m31.029177486s) [upload=445034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:38:31.404855186 (507h9m31.275938231s) [seg_pos={0 0}, upload=445034]: Begin upload segment... Dec 26 18:38:31.404909629 (507h9m31.275992664s) [seg_pos={0 0}, upload=445034]: Done begin upload segment. Dec 26 18:38:31.404939226 (507h9m31.276022261s) [noise=true , address=45.91.116.8:4048 , piece_id=DUK7YWADLEYMV2YVL3R6FE6MKVNI3EFECHIXONPRU3K2JIZBP3VA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.405917509 (507h9m31.277000554s) [noise=true , address=145.79.227.36:4519 , piece_id=CYT4FFVVOTUIKTHMRM5VIJ4O5QR6AAB3MMZGQJ2DHLHV7BSHXC3Q, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.406590080 (507h9m31.277673115s) [noise=true , address=145.79.227.42:4121 , piece_id=4U7MBYKX2BMTHU3FPXNWQONZLPAH5I5REX4MD2IGPEK6DXG6COAA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.407173159 (507h9m31.278256195s) [noise=true , address=145.79.227.46:4601 , piece_id=JS6TZ4ILXJ6ERH3HWMTSDNWRSPCTP25ND7UIFO7B7CDHHHAQKE4Q, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.407546741 (507h9m31.278629786s) [noise=true , address=45.91.116.11:4120 , piece_id=PYWGMEJ3SJ56CE4WW6N22XQUK5ICC2LJC5ICEHDNA7RP44SU5ZJA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.407938116 (507h9m31.279021151s) [noise=true , address=145.79.227.52:4445 , piece_id=ZPIMEROEBTNLTCAQF42GIRND2LZPUFECG5QP2GPJUUHBG47BFTOA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.408758859 (507h9m31.279841894s) [noise=true , address=145.79.227.32:4836 , piece_id=2R6WOS7NHLSL3IECRA4DTPQCA2AIGJE4VZWR472QHUXBYGWED45Q, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.409240627 (507h9m31.280323662s) [noise=true , address=145.79.227.45:4229 , piece_id=YWGQ3PU6GNCTDD3ECFONV2GROLUZNRPCGHJN7UP7ZDJITO5N56FA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.409752571 (507h9m31.280835626s) [noise=true , address=77.90.31.12:4589 , piece_id=4IGKBXF3SOCISAXEAKCSWWZ66BLIQ7NQ5L4YYMITLMCFFDSK3DXA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.410632658 (507h9m31.281715693s) [noise=true , address=145.79.227.16:4909 , piece_id=4CQ3ZCDAGEWPQOM4VANX45TTWFOGKJRXWVAUNUR6FD4EZHZMK2SQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.410855772 (507h9m31.281938817s) [noise=true , address=45.91.116.12:4227 , piece_id=X6LTBYJNVPHRDRKFTYTDDY6N33YWIX3JCME6JLIDFKI52YKSRPLQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.411905451 (507h9m31.282988516s) [noise=true , address=77.90.31.13:4627 , piece_id=46RM6UI6UNSEYAYTYRE5RTI6TTJSMKZQVI2OOK2FVYBWM5T6X4JA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.412072970 (507h9m31.283156015s) [noise=true , address=145.79.227.29:4943 , piece_id=L5TE5QVCMG6IZ5FE7HOXRBWJMUAI3QA6D6JOCH52AEGIX7JU3XDQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.413211528 (507h9m31.284294563s) [noise=true , address=45.91.116.14:4190 , piece_id=RSD64GG2KX27FKVD5XXRA4I3TFEA76ZNBY5HYTPKPLYLPCKGVUKA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.414546780 (507h9m31.285629815s) [noise=true , address=145.79.227.2:4114 , piece_id=36QOULUKE6SBVZIKHJBIPB36OBVOFXAJZ6DVNJHTQVM4FIEMNFOA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.415591630 (507h9m31.286674665s) [noise=true , address=145.79.227.51:4409 , piece_id=2I5ZV3TP3NXSZS7KVH4JBSO5VBWQGKRPDS3MFRR2YLH3TLM33SSQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:32.593710866 (507h9m32.464793931s) [noise=true , address=77.90.31.12:4589 , piece_id=4IGKBXF3SOCISAXEAKCSWWZ66BLIQ7NQ5L4YYMITLMCFFDSK3DXA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 18:38:33.121407613 (507h9m32.992490648s) [upload=445034]: Next returned nil segment Dec 26 18:38:33.121426890 (507h9m32.992509925s) [upload=445034]: Waiting for error group managing segments... Dec 26 18:38:33.413463338 (507h9m33.284546373s) [noise=true , address=145.79.227.45:4229 , piece_id=YWGQ3PU6GNCTDD3ECFONV2GROLUZNRPCGHJN7UP7ZDJITO5N56FA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.414417395 (507h9m33.28550043s) [noise=true , address=145.79.227.46:4601 , piece_id=JS6TZ4ILXJ6ERH3HWMTSDNWRSPCTP25ND7UIFO7B7CDHHHAQKE4Q, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.423567389 (507h9m33.294650424s) [noise=true , address=145.79.227.32:4836 , piece_id=2R6WOS7NHLSL3IECRA4DTPQCA2AIGJE4VZWR472QHUXBYGWED45Q, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.432953442 (507h9m33.304036477s) [noise=true , address=145.79.227.52:4445 , piece_id=ZPIMEROEBTNLTCAQF42GIRND2LZPUFECG5QP2GPJUUHBG47BFTOA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.436006996 (507h9m33.307090031s) [noise=true , address=145.79.227.51:4409 , piece_id=2I5ZV3TP3NXSZS7KVH4JBSO5VBWQGKRPDS3MFRR2YLH3TLM33SSQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.436422918 (507h9m33.307505953s) [noise=true , address=145.79.227.36:4519 , piece_id=CYT4FFVVOTUIKTHMRM5VIJ4O5QR6AAB3MMZGQJ2DHLHV7BSHXC3Q, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.436849921 (507h9m33.307932966s) [noise=true , address=145.79.227.42:4121 , piece_id=4U7MBYKX2BMTHU3FPXNWQONZLPAH5I5REX4MD2IGPEK6DXG6COAA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.449704228 (507h9m33.320787263s) [noise=true , address=145.79.227.29:4943 , piece_id=L5TE5QVCMG6IZ5FE7HOXRBWJMUAI3QA6D6JOCH52AEGIX7JU3XDQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.460416667 (507h9m33.331499712s) [noise=true , address=145.79.227.16:4909 , piece_id=4CQ3ZCDAGEWPQOM4VANX45TTWFOGKJRXWVAUNUR6FD4EZHZMK2SQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.517087077 (507h9m33.388170112s) [noise=true , address=145.79.227.2:4114 , piece_id=36QOULUKE6SBVZIKHJBIPB36OBVOFXAJZ6DVNJHTQVM4FIEMNFOA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.600393006 (507h9m33.471476051s) [noise=true , address=77.90.31.13:4627 , piece_id=46RM6UI6UNSEYAYTYRE5RTI6TTJSMKZQVI2OOK2FVYBWM5T6X4JA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.662483417 (507h9m33.533566472s) [noise=true , address=45.91.116.14:4190 , piece_id=RSD64GG2KX27FKVD5XXRA4I3TFEA76ZNBY5HYTPKPLYLPCKGVUKA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.783839309 (507h9m33.654922344s) [noise=true , address=45.91.116.8:4048 , piece_id=DUK7YWADLEYMV2YVL3R6FE6MKVNI3EFECHIXONPRU3K2JIZBP3VA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.783857174 (507h9m33.654940209s) [seg_pos={0 0}, upload=445034]: Segment reached optimal threshold of 13 pieces. Dec 26 18:38:33.784044630 (507h9m33.655127665s) [noise=true , address=45.91.116.11:4120 , piece_id=PYWGMEJ3SJ56CE4WW6N22XQUK5ICC2LJC5ICEHDNA7RP44SU5ZJA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 18:38:33.784050812 (507h9m33.655133847s) [noise=true , address=45.91.116.11:4120 , piece_id=PYWGMEJ3SJ56CE4WW6N22XQUK5ICC2LJC5ICEHDNA7RP44SU5ZJA, seg_pos={0 0}, upload=445034]: Long tail cancellation due to optimal threshold Dec 26 18:38:33.784255181 (507h9m33.655338216s) [noise=true , address=45.91.116.12:4227 , piece_id=X6LTBYJNVPHRDRKFTYTDDY6N33YWIX3JCME6JLIDFKI52YKSRPLQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:51964->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51964->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51964->45.91.116.12:4227: use of closed network connection Dec 26 18:38:33.784269758 (507h9m33.655352794s) [noise=true , address=45.91.116.12:4227 , piece_id=X6LTBYJNVPHRDRKFTYTDDY6N33YWIX3JCME6JLIDFKI52YKSRPLQ, seg_pos={0 0}, upload=445034]: Long tail cancellation due to optimal threshold Dec 26 18:38:33.784299335 (507h9m33.65538237s) [seg_pos={0 0}, upload=445034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:38:33.784380079 (507h9m33.655463124s) [upload=445034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:38:33.784384908 (507h9m33.655467953s) [upload=445034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:38:34.099088118 (507h9m33.970171153s) [upload=445034]: Done uploading segments... Dec 26 18:38:34.099106454 (507h9m33.970189499s) [upload=445034]: Upload finished. err: <nil> Duration: 16.362336364s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [........X..X.....X...XX.........X.....X..X..X.X.X.....XXX.X.] duration: 6.124315731s trace: https://jaeger.dev.storj.io/trace/4d0231720e50e740 FAIL (2025-12-26 18:39:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 18:38:31.157821160 (507h9m31.028904205s) [upload=445034]: Starting upload Dec 26 18:38:31.157877146 (507h9m31.028960191s) [upload=445034]: Done starting upload Dec 26 18:38:31.157942992 (507h9m31.029026037s) [upload=445034]: Uploading segments... Dec 26 18:38:31.158059945 (507h9m31.02914298s) [upload=445034]: Got next segment. Inline: false Dec 26 18:38:31.158089972 (507h9m31.029173018s) [upload=445034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:38:31.158094451 (507h9m31.029177486s) [upload=445034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:38:31.404855186 (507h9m31.275938231s) [seg_pos={0 0}, upload=445034]: Begin upload segment... Dec 26 18:38:31.404909629 (507h9m31.275992664s) [seg_pos={0 0}, upload=445034]: Done begin upload segment. Dec 26 18:38:31.404939226 (507h9m31.276022261s) [noise=true , address=45.91.116.8:4048 , piece_id=DUK7YWADLEYMV2YVL3R6FE6MKVNI3EFECHIXONPRU3K2JIZBP3VA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.405917509 (507h9m31.277000554s) [noise=true , address=145.79.227.36:4519 , piece_id=CYT4FFVVOTUIKTHMRM5VIJ4O5QR6AAB3MMZGQJ2DHLHV7BSHXC3Q, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.406590080 (507h9m31.277673115s) [noise=true , address=145.79.227.42:4121 , piece_id=4U7MBYKX2BMTHU3FPXNWQONZLPAH5I5REX4MD2IGPEK6DXG6COAA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.407173159 (507h9m31.278256195s) [noise=true , address=145.79.227.46:4601 , piece_id=JS6TZ4ILXJ6ERH3HWMTSDNWRSPCTP25ND7UIFO7B7CDHHHAQKE4Q, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.407546741 (507h9m31.278629786s) [noise=true , address=45.91.116.11:4120 , piece_id=PYWGMEJ3SJ56CE4WW6N22XQUK5ICC2LJC5ICEHDNA7RP44SU5ZJA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.407938116 (507h9m31.279021151s) [noise=true , address=145.79.227.52:4445 , piece_id=ZPIMEROEBTNLTCAQF42GIRND2LZPUFECG5QP2GPJUUHBG47BFTOA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.408758859 (507h9m31.279841894s) [noise=true , address=145.79.227.32:4836 , piece_id=2R6WOS7NHLSL3IECRA4DTPQCA2AIGJE4VZWR472QHUXBYGWED45Q, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.409240627 (507h9m31.280323662s) [noise=true , address=145.79.227.45:4229 , piece_id=YWGQ3PU6GNCTDD3ECFONV2GROLUZNRPCGHJN7UP7ZDJITO5N56FA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.409752571 (507h9m31.280835626s) [noise=true , address=77.90.31.12:4589 , piece_id=4IGKBXF3SOCISAXEAKCSWWZ66BLIQ7NQ5L4YYMITLMCFFDSK3DXA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.410632658 (507h9m31.281715693s) [noise=true , address=145.79.227.16:4909 , piece_id=4CQ3ZCDAGEWPQOM4VANX45TTWFOGKJRXWVAUNUR6FD4EZHZMK2SQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.410855772 (507h9m31.281938817s) [noise=true , address=45.91.116.12:4227 , piece_id=X6LTBYJNVPHRDRKFTYTDDY6N33YWIX3JCME6JLIDFKI52YKSRPLQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.411905451 (507h9m31.282988516s) [noise=true , address=77.90.31.13:4627 , piece_id=46RM6UI6UNSEYAYTYRE5RTI6TTJSMKZQVI2OOK2FVYBWM5T6X4JA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.412072970 (507h9m31.283156015s) [noise=true , address=145.79.227.29:4943 , piece_id=L5TE5QVCMG6IZ5FE7HOXRBWJMUAI3QA6D6JOCH52AEGIX7JU3XDQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.413211528 (507h9m31.284294563s) [noise=true , address=45.91.116.14:4190 , piece_id=RSD64GG2KX27FKVD5XXRA4I3TFEA76ZNBY5HYTPKPLYLPCKGVUKA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.414546780 (507h9m31.285629815s) [noise=true , address=145.79.227.2:4114 , piece_id=36QOULUKE6SBVZIKHJBIPB36OBVOFXAJZ6DVNJHTQVM4FIEMNFOA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.415591630 (507h9m31.286674665s) [noise=true , address=145.79.227.51:4409 , piece_id=2I5ZV3TP3NXSZS7KVH4JBSO5VBWQGKRPDS3MFRR2YLH3TLM33SSQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:32.593710866 (507h9m32.464793931s) [noise=true , address=77.90.31.12:4589 , piece_id=4IGKBXF3SOCISAXEAKCSWWZ66BLIQ7NQ5L4YYMITLMCFFDSK3DXA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 18:38:33.121407613 (507h9m32.992490648s) [upload=445034]: Next returned nil segment Dec 26 18:38:33.121426890 (507h9m32.992509925s) [upload=445034]: Waiting for error group managing segments... Dec 26 18:38:33.413463338 (507h9m33.284546373s) [noise=true , address=145.79.227.45:4229 , piece_id=YWGQ3PU6GNCTDD3ECFONV2GROLUZNRPCGHJN7UP7ZDJITO5N56FA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.414417395 (507h9m33.28550043s) [noise=true , address=145.79.227.46:4601 , piece_id=JS6TZ4ILXJ6ERH3HWMTSDNWRSPCTP25ND7UIFO7B7CDHHHAQKE4Q, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.423567389 (507h9m33.294650424s) [noise=true , address=145.79.227.32:4836 , piece_id=2R6WOS7NHLSL3IECRA4DTPQCA2AIGJE4VZWR472QHUXBYGWED45Q, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.432953442 (507h9m33.304036477s) [noise=true , address=145.79.227.52:4445 , piece_id=ZPIMEROEBTNLTCAQF42GIRND2LZPUFECG5QP2GPJUUHBG47BFTOA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.436006996 (507h9m33.307090031s) [noise=true , address=145.79.227.51:4409 , piece_id=2I5ZV3TP3NXSZS7KVH4JBSO5VBWQGKRPDS3MFRR2YLH3TLM33SSQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.436422918 (507h9m33.307505953s) [noise=true , address=145.79.227.36:4519 , piece_id=CYT4FFVVOTUIKTHMRM5VIJ4O5QR6AAB3MMZGQJ2DHLHV7BSHXC3Q, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.436849921 (507h9m33.307932966s) [noise=true , address=145.79.227.42:4121 , piece_id=4U7MBYKX2BMTHU3FPXNWQONZLPAH5I5REX4MD2IGPEK6DXG6COAA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.449704228 (507h9m33.320787263s) [noise=true , address=145.79.227.29:4943 , piece_id=L5TE5QVCMG6IZ5FE7HOXRBWJMUAI3QA6D6JOCH52AEGIX7JU3XDQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.460416667 (507h9m33.331499712s) [noise=true , address=145.79.227.16:4909 , piece_id=4CQ3ZCDAGEWPQOM4VANX45TTWFOGKJRXWVAUNUR6FD4EZHZMK2SQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.517087077 (507h9m33.388170112s) [noise=true , address=145.79.227.2:4114 , piece_id=36QOULUKE6SBVZIKHJBIPB36OBVOFXAJZ6DVNJHTQVM4FIEMNFOA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.600393006 (507h9m33.471476051s) [noise=true , address=77.90.31.13:4627 , piece_id=46RM6UI6UNSEYAYTYRE5RTI6TTJSMKZQVI2OOK2FVYBWM5T6X4JA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.662483417 (507h9m33.533566472s) [noise=true , address=45.91.116.14:4190 , piece_id=RSD64GG2KX27FKVD5XXRA4I3TFEA76ZNBY5HYTPKPLYLPCKGVUKA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.783839309 (507h9m33.654922344s) [noise=true , address=45.91.116.8:4048 , piece_id=DUK7YWADLEYMV2YVL3R6FE6MKVNI3EFECHIXONPRU3K2JIZBP3VA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.783857174 (507h9m33.654940209s) [seg_pos={0 0}, upload=445034]: Segment reached optimal threshold of 13 pieces. Dec 26 18:38:33.784044630 (507h9m33.655127665s) [noise=true , address=45.91.116.11:4120 , piece_id=PYWGMEJ3SJ56CE4WW6N22XQUK5ICC2LJC5ICEHDNA7RP44SU5ZJA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 18:38:33.784050812 (507h9m33.655133847s) [noise=true , address=45.91.116.11:4120 , piece_id=PYWGMEJ3SJ56CE4WW6N22XQUK5ICC2LJC5ICEHDNA7RP44SU5ZJA, seg_pos={0 0}, upload=445034]: Long tail cancellation due to optimal threshold Dec 26 18:38:33.784255181 (507h9m33.655338216s) [noise=true , address=45.91.116.12:4227 , piece_id=X6LTBYJNVPHRDRKFTYTDDY6N33YWIX3JCME6JLIDFKI52YKSRPLQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:51964->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51964->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51964->45.91.116.12:4227: use of closed network connection Dec 26 18:38:33.784269758 (507h9m33.655352794s) [noise=true , address=45.91.116.12:4227 , piece_id=X6LTBYJNVPHRDRKFTYTDDY6N33YWIX3JCME6JLIDFKI52YKSRPLQ, seg_pos={0 0}, upload=445034]: Long tail cancellation due to optimal threshold Dec 26 18:38:33.784299335 (507h9m33.65538237s) [seg_pos={0 0}, upload=445034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:38:33.784380079 (507h9m33.655463124s) [upload=445034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:38:33.784384908 (507h9m33.655467953s) [upload=445034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:38:34.099088118 (507h9m33.970171153s) [upload=445034]: Done uploading segments... Dec 26 18:38:34.099106454 (507h9m33.970189499s) [upload=445034]: Upload finished. err: <nil> Duration: 6.124332171s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.........X..X.....X...XX.........X.....X..X..X.X.X.....XXX.X] duration: 30.002530261s trace: https://jaeger.dev.storj.io/trace/4d0231720e50e740 FAIL (2025-12-26 18:39:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 18:38:31.157821160 (507h9m31.028904205s) [upload=445034]: Starting upload Dec 26 18:38:31.157877146 (507h9m31.028960191s) [upload=445034]: Done starting upload Dec 26 18:38:31.157942992 (507h9m31.029026037s) [upload=445034]: Uploading segments... Dec 26 18:38:31.158059945 (507h9m31.02914298s) [upload=445034]: Got next segment. Inline: false Dec 26 18:38:31.158089972 (507h9m31.029173018s) [upload=445034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:38:31.158094451 (507h9m31.029177486s) [upload=445034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:38:31.404855186 (507h9m31.275938231s) [seg_pos={0 0}, upload=445034]: Begin upload segment... Dec 26 18:38:31.404909629 (507h9m31.275992664s) [seg_pos={0 0}, upload=445034]: Done begin upload segment. Dec 26 18:38:31.404939226 (507h9m31.276022261s) [noise=true , address=45.91.116.8:4048 , piece_id=DUK7YWADLEYMV2YVL3R6FE6MKVNI3EFECHIXONPRU3K2JIZBP3VA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.405917509 (507h9m31.277000554s) [noise=true , address=145.79.227.36:4519 , piece_id=CYT4FFVVOTUIKTHMRM5VIJ4O5QR6AAB3MMZGQJ2DHLHV7BSHXC3Q, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.406590080 (507h9m31.277673115s) [noise=true , address=145.79.227.42:4121 , piece_id=4U7MBYKX2BMTHU3FPXNWQONZLPAH5I5REX4MD2IGPEK6DXG6COAA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.407173159 (507h9m31.278256195s) [noise=true , address=145.79.227.46:4601 , piece_id=JS6TZ4ILXJ6ERH3HWMTSDNWRSPCTP25ND7UIFO7B7CDHHHAQKE4Q, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.407546741 (507h9m31.278629786s) [noise=true , address=45.91.116.11:4120 , piece_id=PYWGMEJ3SJ56CE4WW6N22XQUK5ICC2LJC5ICEHDNA7RP44SU5ZJA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.407938116 (507h9m31.279021151s) [noise=true , address=145.79.227.52:4445 , piece_id=ZPIMEROEBTNLTCAQF42GIRND2LZPUFECG5QP2GPJUUHBG47BFTOA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.408758859 (507h9m31.279841894s) [noise=true , address=145.79.227.32:4836 , piece_id=2R6WOS7NHLSL3IECRA4DTPQCA2AIGJE4VZWR472QHUXBYGWED45Q, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.409240627 (507h9m31.280323662s) [noise=true , address=145.79.227.45:4229 , piece_id=YWGQ3PU6GNCTDD3ECFONV2GROLUZNRPCGHJN7UP7ZDJITO5N56FA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.409752571 (507h9m31.280835626s) [noise=true , address=77.90.31.12:4589 , piece_id=4IGKBXF3SOCISAXEAKCSWWZ66BLIQ7NQ5L4YYMITLMCFFDSK3DXA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.410632658 (507h9m31.281715693s) [noise=true , address=145.79.227.16:4909 , piece_id=4CQ3ZCDAGEWPQOM4VANX45TTWFOGKJRXWVAUNUR6FD4EZHZMK2SQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.410855772 (507h9m31.281938817s) [noise=true , address=45.91.116.12:4227 , piece_id=X6LTBYJNVPHRDRKFTYTDDY6N33YWIX3JCME6JLIDFKI52YKSRPLQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.411905451 (507h9m31.282988516s) [noise=true , address=77.90.31.13:4627 , piece_id=46RM6UI6UNSEYAYTYRE5RTI6TTJSMKZQVI2OOK2FVYBWM5T6X4JA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.412072970 (507h9m31.283156015s) [noise=true , address=145.79.227.29:4943 , piece_id=L5TE5QVCMG6IZ5FE7HOXRBWJMUAI3QA6D6JOCH52AEGIX7JU3XDQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.413211528 (507h9m31.284294563s) [noise=true , address=45.91.116.14:4190 , piece_id=RSD64GG2KX27FKVD5XXRA4I3TFEA76ZNBY5HYTPKPLYLPCKGVUKA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.414546780 (507h9m31.285629815s) [noise=true , address=145.79.227.2:4114 , piece_id=36QOULUKE6SBVZIKHJBIPB36OBVOFXAJZ6DVNJHTQVM4FIEMNFOA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.415591630 (507h9m31.286674665s) [noise=true , address=145.79.227.51:4409 , piece_id=2I5ZV3TP3NXSZS7KVH4JBSO5VBWQGKRPDS3MFRR2YLH3TLM33SSQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:32.593710866 (507h9m32.464793931s) [noise=true , address=77.90.31.12:4589 , piece_id=4IGKBXF3SOCISAXEAKCSWWZ66BLIQ7NQ5L4YYMITLMCFFDSK3DXA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 18:38:33.121407613 (507h9m32.992490648s) [upload=445034]: Next returned nil segment Dec 26 18:38:33.121426890 (507h9m32.992509925s) [upload=445034]: Waiting for error group managing segments... Dec 26 18:38:33.413463338 (507h9m33.284546373s) [noise=true , address=145.79.227.45:4229 , piece_id=YWGQ3PU6GNCTDD3ECFONV2GROLUZNRPCGHJN7UP7ZDJITO5N56FA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.414417395 (507h9m33.28550043s) [noise=true , address=145.79.227.46:4601 , piece_id=JS6TZ4ILXJ6ERH3HWMTSDNWRSPCTP25ND7UIFO7B7CDHHHAQKE4Q, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.423567389 (507h9m33.294650424s) [noise=true , address=145.79.227.32:4836 , piece_id=2R6WOS7NHLSL3IECRA4DTPQCA2AIGJE4VZWR472QHUXBYGWED45Q, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.432953442 (507h9m33.304036477s) [noise=true , address=145.79.227.52:4445 , piece_id=ZPIMEROEBTNLTCAQF42GIRND2LZPUFECG5QP2GPJUUHBG47BFTOA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.436006996 (507h9m33.307090031s) [noise=true , address=145.79.227.51:4409 , piece_id=2I5ZV3TP3NXSZS7KVH4JBSO5VBWQGKRPDS3MFRR2YLH3TLM33SSQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.436422918 (507h9m33.307505953s) [noise=true , address=145.79.227.36:4519 , piece_id=CYT4FFVVOTUIKTHMRM5VIJ4O5QR6AAB3MMZGQJ2DHLHV7BSHXC3Q, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.436849921 (507h9m33.307932966s) [noise=true , address=145.79.227.42:4121 , piece_id=4U7MBYKX2BMTHU3FPXNWQONZLPAH5I5REX4MD2IGPEK6DXG6COAA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.449704228 (507h9m33.320787263s) [noise=true , address=145.79.227.29:4943 , piece_id=L5TE5QVCMG6IZ5FE7HOXRBWJMUAI3QA6D6JOCH52AEGIX7JU3XDQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.460416667 (507h9m33.331499712s) [noise=true , address=145.79.227.16:4909 , piece_id=4CQ3ZCDAGEWPQOM4VANX45TTWFOGKJRXWVAUNUR6FD4EZHZMK2SQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.517087077 (507h9m33.388170112s) [noise=true , address=145.79.227.2:4114 , piece_id=36QOULUKE6SBVZIKHJBIPB36OBVOFXAJZ6DVNJHTQVM4FIEMNFOA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.600393006 (507h9m33.471476051s) [noise=true , address=77.90.31.13:4627 , piece_id=46RM6UI6UNSEYAYTYRE5RTI6TTJSMKZQVI2OOK2FVYBWM5T6X4JA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.662483417 (507h9m33.533566472s) [noise=true , address=45.91.116.14:4190 , piece_id=RSD64GG2KX27FKVD5XXRA4I3TFEA76ZNBY5HYTPKPLYLPCKGVUKA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.783839309 (507h9m33.654922344s) [noise=true , address=45.91.116.8:4048 , piece_id=DUK7YWADLEYMV2YVL3R6FE6MKVNI3EFECHIXONPRU3K2JIZBP3VA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.783857174 (507h9m33.654940209s) [seg_pos={0 0}, upload=445034]: Segment reached optimal threshold of 13 pieces. Dec 26 18:38:33.784044630 (507h9m33.655127665s) [noise=true , address=45.91.116.11:4120 , piece_id=PYWGMEJ3SJ56CE4WW6N22XQUK5ICC2LJC5ICEHDNA7RP44SU5ZJA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 18:38:33.784050812 (507h9m33.655133847s) [noise=true , address=45.91.116.11:4120 , piece_id=PYWGMEJ3SJ56CE4WW6N22XQUK5ICC2LJC5ICEHDNA7RP44SU5ZJA, seg_pos={0 0}, upload=445034]: Long tail cancellation due to optimal threshold Dec 26 18:38:33.784255181 (507h9m33.655338216s) [noise=true , address=45.91.116.12:4227 , piece_id=X6LTBYJNVPHRDRKFTYTDDY6N33YWIX3JCME6JLIDFKI52YKSRPLQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:51964->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51964->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51964->45.91.116.12:4227: use of closed network connection Dec 26 18:38:33.784269758 (507h9m33.655352794s) [noise=true , address=45.91.116.12:4227 , piece_id=X6LTBYJNVPHRDRKFTYTDDY6N33YWIX3JCME6JLIDFKI52YKSRPLQ, seg_pos={0 0}, upload=445034]: Long tail cancellation due to optimal threshold Dec 26 18:38:33.784299335 (507h9m33.65538237s) [seg_pos={0 0}, upload=445034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:38:33.784380079 (507h9m33.655463124s) [upload=445034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:38:33.784384908 (507h9m33.655467953s) [upload=445034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:38:34.099088118 (507h9m33.970171153s) [upload=445034]: Done uploading segments... Dec 26 18:38:34.099106454 (507h9m33.970189499s) [upload=445034]: Upload finished. err: <nil> Duration: 30.002552534s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.........X..X.....X...XX.........X.....X..X..X.X.X.....XXX.] duration: 7.210502173s trace: https://jaeger.dev.storj.io/trace/3b45fd7658be0262 FAIL (2025-12-26 18:38:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: NXLUQNJVFLGBOEPPCO2ZZOAECZAZT26SFU3KBFVBYUNZYLCAKRWA) context deadline exceeded; piecestore: (Node ID: 1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d, Piece ID: NK6JINBNQOSTK6W4CDVR5SEUJVCQOLJE2KYDLAHID2G63YQTQCMQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: PWCCOYLWEYFEXKCWUOKVRUN6HTENOWNKAV75WY2L4AY34N66V35Q) context deadline exceeded ======LOGS====== Dec 26 18:37:19.121626906 (507h8m18.992709951s) [upload=445016]: Starting upload Dec 26 18:37:19.121685578 (507h8m18.992768623s) [upload=445016]: Done starting upload Dec 26 18:37:19.121785598 (507h8m18.992868634s) [upload=445016]: Uploading segments... Dec 26 18:37:19.121920165 (507h8m18.99300321s) [upload=445016]: Got next segment. Inline: false Dec 26 18:37:19.121963387 (507h8m18.993046422s) [upload=445016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:37:19.121968106 (507h8m18.993051141s) [upload=445016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:37:19.372302056 (507h8m19.243385101s) [seg_pos={0 0}, upload=445016]: Begin upload segment... Dec 26 18:37:19.372356729 (507h8m19.243439764s) [seg_pos={0 0}, upload=445016]: Done begin upload segment. Dec 26 18:37:19.372392647 (507h8m19.243475682s) [noise=true , address=145.79.227.48:4303 , piece_id=ANWVQPQ4HARTRLRYO3X2FGXWBTN35YCMRQZXSK534FZOGNAZLNPQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.372401365 (507h8m19.24348441s) [noise=true , address=145.79.227.21:4475 , piece_id=IRI7GQFKTWPI2NRYCEM2PVDN46I6IFN4ONQOJV3SSMLUJXGCD7GQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.373123118 (507h8m19.244206163s) [noise=true , address=145.79.227.12:4767 , piece_id=T7A4IXELU3AYIXDV2W2EHFAFLN6XQC7UIT76F4OH3WYDPJ2RNSYQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.373405486 (507h8m19.244488521s) [noise=true , address=145.79.227.8:4330 , piece_id=PSXZGJ3UMIYBVRDKCXRPR63ZYBUKZ75C6O24CAHSQYZK7KO2LPXQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.373656364 (507h8m19.2447394s) [noise=true , address=145.79.227.14:4839 , piece_id=MAA4EEX7BSJKPCDQOGUO2QRT5O6ILJ3KFK7NOFSHWG357VAY737A, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.373679438 (507h8m19.244762493s) [noise=true , address=145.79.227.43:4158 , piece_id=4Z5DAXGBYWCZQH4PLYZDJL4OOMA7TTXAUXZ3HTSORMDAB7HHE3FQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.373951776 (507h8m19.245034812s) [noise=true , address=77.90.31.10:4518 , piece_id=XN6HWG2K6V4DMLR42H2BOPVC2BE6RGNBTNGUQNOC6FYAYD7LPQ2Q, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.374182646 (507h8m19.245265682s) [noise=true , address=145.79.227.2:4115 , piece_id=EU56CW4GG76TO67THVLBF6RZAT6D4GD6MPYSRDIDY44IBIHBBXNQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.374233853 (507h8m19.245316889s) [noise=true , address=45.91.116.5:4300 , piece_id=JMCPZRHD52SRLTLYJU3ACH37IBFMMHWEABV265SE5STCCOPHN4UA, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.374509018 (507h8m19.245592053s) [noise=true , address=145.79.227.46:4602 , piece_id=NK6JINBNQOSTK6W4CDVR5SEUJVCQOLJE2KYDLAHID2G63YQTQCMQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.374862831 (507h8m19.245945866s) [noise=true , address=145.79.227.41:4086 , piece_id=3IUK67FIMMKTR7NXHRSHWM7JCOXZFICZT6PHWXDKBUAWSJC23VMA, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.375174074 (507h8m19.246257119s) [noise=true , address=145.79.227.37:4565 , piece_id=Y22BCACQP6GKMNCFN7L2AWGJHXEBEQYGCW7P6UHQTN5S4SAMH4JQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.375501438 (507h8m19.246584483s) [noise=true , address=45.91.116.12:4227 , piece_id=PWCCOYLWEYFEXKCWUOKVRUN6HTENOWNKAV75WY2L4AY34N66V35Q, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.375841154 (507h8m19.246924199s) [noise=true , address=45.91.116.6:4976 , piece_id=NXLUQNJVFLGBOEPPCO2ZZOAECZAZT26SFU3KBFVBYUNZYLCAKRWA, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.376217330 (507h8m19.247300375s) [noise=true , address=145.79.227.51:4409 , piece_id=AKO6JWAXFOMWPQHM7SOVGX4HABVCBHYTSKCUGWNP6N2VLEBDIRJQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.376820149 (507h8m19.247903254s) [noise=true , address=77.90.31.16:4736 , piece_id=KO2NNFY4YDMRWD5BYBKHF7UUNAE2G5KICGDWDIHEJNFGUF7NIJBQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:21.089122453 (507h8m20.960205498s) [upload=445016]: Next returned nil segment Dec 26 18:37:21.089141359 (507h8m20.960224394s) [upload=445016]: Waiting for error group managing segments... Dec 26 18:37:21.363581731 (507h8m21.234664766s) [noise=true , address=145.79.227.43:4158 , piece_id=4Z5DAXGBYWCZQH4PLYZDJL4OOMA7TTXAUXZ3HTSORMDAB7HHE3FQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.368429613 (507h8m21.239512658s) [noise=true , address=145.79.227.21:4475 , piece_id=IRI7GQFKTWPI2NRYCEM2PVDN46I6IFN4ONQOJV3SSMLUJXGCD7GQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.370357303 (507h8m21.241440348s) [noise=true , address=145.79.227.37:4565 , piece_id=Y22BCACQP6GKMNCFN7L2AWGJHXEBEQYGCW7P6UHQTN5S4SAMH4JQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.379265586 (507h8m21.250348621s) [noise=true , address=145.79.227.48:4303 , piece_id=ANWVQPQ4HARTRLRYO3X2FGXWBTN35YCMRQZXSK534FZOGNAZLNPQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.394533419 (507h8m21.265616464s) [noise=true , address=145.79.227.8:4330 , piece_id=PSXZGJ3UMIYBVRDKCXRPR63ZYBUKZ75C6O24CAHSQYZK7KO2LPXQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.402674891 (507h8m21.273757966s) [noise=true , address=145.79.227.51:4409 , piece_id=AKO6JWAXFOMWPQHM7SOVGX4HABVCBHYTSKCUGWNP6N2VLEBDIRJQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.403498369 (507h8m21.274581414s) [noise=true , address=145.79.227.46:4602 , piece_id=NK6JINBNQOSTK6W4CDVR5SEUJVCQOLJE2KYDLAHID2G63YQTQCMQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.435802712 (507h8m21.306885747s) [noise=true , address=145.79.227.2:4115 , piece_id=EU56CW4GG76TO67THVLBF6RZAT6D4GD6MPYSRDIDY44IBIHBBXNQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.436628545 (507h8m21.30771158s) [noise=true , address=145.79.227.12:4767 , piece_id=T7A4IXELU3AYIXDV2W2EHFAFLN6XQC7UIT76F4OH3WYDPJ2RNSYQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.649643122 (507h8m21.520726167s) [noise=true , address=145.79.227.41:4086 , piece_id=3IUK67FIMMKTR7NXHRSHWM7JCOXZFICZT6PHWXDKBUAWSJC23VMA, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.739242897 (507h8m21.610325932s) [noise=true , address=45.91.116.12:4227 , piece_id=PWCCOYLWEYFEXKCWUOKVRUN6HTENOWNKAV75WY2L4AY34N66V35Q, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.979515096 (507h8m21.850598141s) [noise=true , address=77.90.31.10:4518 , piece_id=XN6HWG2K6V4DMLR42H2BOPVC2BE6RGNBTNGUQNOC6FYAYD7LPQ2Q, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:22.352800938 (507h8m22.223884013s) [noise=true , address=45.91.116.6:4976 , piece_id=NXLUQNJVFLGBOEPPCO2ZZOAECZAZT26SFU3KBFVBYUNZYLCAKRWA, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:22.352832238 (507h8m22.223915283s) [seg_pos={0 0}, upload=445016]: Segment reached optimal threshold of 13 pieces. Dec 26 18:37:22.353106590 (507h8m22.224189645s) [noise=true , address=145.79.227.14:4839 , piece_id=MAA4EEX7BSJKPCDQOGUO2QRT5O6ILJ3KFK7NOFSHWG357VAY737A, seg_pos={0 0}, upload=445016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:41048->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:41048->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41048->145.79.227.14:4839: use of closed network connection Dec 26 18:37:22.353116970 (507h8m22.224200025s) [noise=true , address=145.79.227.14:4839 , piece_id=MAA4EEX7BSJKPCDQOGUO2QRT5O6ILJ3KFK7NOFSHWG357VAY737A, seg_pos={0 0}, upload=445016]: Long tail cancellation due to optimal threshold Dec 26 18:37:22.353157867 (507h8m22.224240912s) [noise=true , address=45.91.116.5:4300 , piece_id=JMCPZRHD52SRLTLYJU3ACH37IBFMMHWEABV265SE5STCCOPHN4UA, seg_pos={0 0}, upload=445016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 26 18:37:22.353168328 (507h8m22.224251373s) [noise=true , address=45.91.116.5:4300 , piece_id=JMCPZRHD52SRLTLYJU3ACH37IBFMMHWEABV265SE5STCCOPHN4UA, seg_pos={0 0}, upload=445016]: Long tail cancellation due to optimal threshold Dec 26 18:37:22.353342600 (507h8m22.224425655s) [noise=true , address=77.90.31.16:4736 , piece_id=KO2NNFY4YDMRWD5BYBKHF7UUNAE2G5KICGDWDIHEJNFGUF7NIJBQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 18:37:22.353353791 (507h8m22.224436866s) [noise=true , address=77.90.31.16:4736 , piece_id=KO2NNFY4YDMRWD5BYBKHF7UUNAE2G5KICGDWDIHEJNFGUF7NIJBQ, seg_pos={0 0}, upload=445016]: Long tail cancellation due to optimal threshold Dec 26 18:37:22.353381794 (507h8m22.224464839s) [seg_pos={0 0}, upload=445016]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:37:22.353476835 (507h8m22.224559891s) [upload=445016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:37:22.353484730 (507h8m22.224567785s) [upload=445016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:37:22.655223604 (507h8m22.526306649s) [upload=445016]: Done uploading segments... Dec 26 18:37:22.655247399 (507h8m22.526330434s) [upload=445016]: Upload finished. err: <nil> Duration: 7.210585762s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XX.........X..X.....X...XX.........X.....X..X..X.X.X.....XXX] duration: 30.003966518s trace: https://jaeger.dev.storj.io/trace/3b45fd7658be0262 FAIL (2025-12-26 18:37:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: NXLUQNJVFLGBOEPPCO2ZZOAECZAZT26SFU3KBFVBYUNZYLCAKRWA) context deadline exceeded; piecestore: (Node ID: 1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d, Piece ID: NK6JINBNQOSTK6W4CDVR5SEUJVCQOLJE2KYDLAHID2G63YQTQCMQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: PWCCOYLWEYFEXKCWUOKVRUN6HTENOWNKAV75WY2L4AY34N66V35Q) context deadline exceeded ======LOGS====== Dec 26 18:37:19.121626906 (507h8m18.992709951s) [upload=445016]: Starting upload Dec 26 18:37:19.121685578 (507h8m18.992768623s) [upload=445016]: Done starting upload Dec 26 18:37:19.121785598 (507h8m18.992868634s) [upload=445016]: Uploading segments... Dec 26 18:37:19.121920165 (507h8m18.99300321s) [upload=445016]: Got next segment. Inline: false Dec 26 18:37:19.121963387 (507h8m18.993046422s) [upload=445016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:37:19.121968106 (507h8m18.993051141s) [upload=445016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:37:19.372302056 (507h8m19.243385101s) [seg_pos={0 0}, upload=445016]: Begin upload segment... Dec 26 18:37:19.372356729 (507h8m19.243439764s) [seg_pos={0 0}, upload=445016]: Done begin upload segment. Dec 26 18:37:19.372392647 (507h8m19.243475682s) [noise=true , address=145.79.227.48:4303 , piece_id=ANWVQPQ4HARTRLRYO3X2FGXWBTN35YCMRQZXSK534FZOGNAZLNPQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.372401365 (507h8m19.24348441s) [noise=true , address=145.79.227.21:4475 , piece_id=IRI7GQFKTWPI2NRYCEM2PVDN46I6IFN4ONQOJV3SSMLUJXGCD7GQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.373123118 (507h8m19.244206163s) [noise=true , address=145.79.227.12:4767 , piece_id=T7A4IXELU3AYIXDV2W2EHFAFLN6XQC7UIT76F4OH3WYDPJ2RNSYQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.373405486 (507h8m19.244488521s) [noise=true , address=145.79.227.8:4330 , piece_id=PSXZGJ3UMIYBVRDKCXRPR63ZYBUKZ75C6O24CAHSQYZK7KO2LPXQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.373656364 (507h8m19.2447394s) [noise=true , address=145.79.227.14:4839 , piece_id=MAA4EEX7BSJKPCDQOGUO2QRT5O6ILJ3KFK7NOFSHWG357VAY737A, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.373679438 (507h8m19.244762493s) [noise=true , address=145.79.227.43:4158 , piece_id=4Z5DAXGBYWCZQH4PLYZDJL4OOMA7TTXAUXZ3HTSORMDAB7HHE3FQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.373951776 (507h8m19.245034812s) [noise=true , address=77.90.31.10:4518 , piece_id=XN6HWG2K6V4DMLR42H2BOPVC2BE6RGNBTNGUQNOC6FYAYD7LPQ2Q, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.374182646 (507h8m19.245265682s) [noise=true , address=145.79.227.2:4115 , piece_id=EU56CW4GG76TO67THVLBF6RZAT6D4GD6MPYSRDIDY44IBIHBBXNQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.374233853 (507h8m19.245316889s) [noise=true , address=45.91.116.5:4300 , piece_id=JMCPZRHD52SRLTLYJU3ACH37IBFMMHWEABV265SE5STCCOPHN4UA, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.374509018 (507h8m19.245592053s) [noise=true , address=145.79.227.46:4602 , piece_id=NK6JINBNQOSTK6W4CDVR5SEUJVCQOLJE2KYDLAHID2G63YQTQCMQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.374862831 (507h8m19.245945866s) [noise=true , address=145.79.227.41:4086 , piece_id=3IUK67FIMMKTR7NXHRSHWM7JCOXZFICZT6PHWXDKBUAWSJC23VMA, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.375174074 (507h8m19.246257119s) [noise=true , address=145.79.227.37:4565 , piece_id=Y22BCACQP6GKMNCFN7L2AWGJHXEBEQYGCW7P6UHQTN5S4SAMH4JQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.375501438 (507h8m19.246584483s) [noise=true , address=45.91.116.12:4227 , piece_id=PWCCOYLWEYFEXKCWUOKVRUN6HTENOWNKAV75WY2L4AY34N66V35Q, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.375841154 (507h8m19.246924199s) [noise=true , address=45.91.116.6:4976 , piece_id=NXLUQNJVFLGBOEPPCO2ZZOAECZAZT26SFU3KBFVBYUNZYLCAKRWA, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.376217330 (507h8m19.247300375s) [noise=true , address=145.79.227.51:4409 , piece_id=AKO6JWAXFOMWPQHM7SOVGX4HABVCBHYTSKCUGWNP6N2VLEBDIRJQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.376820149 (507h8m19.247903254s) [noise=true , address=77.90.31.16:4736 , piece_id=KO2NNFY4YDMRWD5BYBKHF7UUNAE2G5KICGDWDIHEJNFGUF7NIJBQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:21.089122453 (507h8m20.960205498s) [upload=445016]: Next returned nil segment Dec 26 18:37:21.089141359 (507h8m20.960224394s) [upload=445016]: Waiting for error group managing segments... Dec 26 18:37:21.363581731 (507h8m21.234664766s) [noise=true , address=145.79.227.43:4158 , piece_id=4Z5DAXGBYWCZQH4PLYZDJL4OOMA7TTXAUXZ3HTSORMDAB7HHE3FQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.368429613 (507h8m21.239512658s) [noise=true , address=145.79.227.21:4475 , piece_id=IRI7GQFKTWPI2NRYCEM2PVDN46I6IFN4ONQOJV3SSMLUJXGCD7GQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.370357303 (507h8m21.241440348s) [noise=true , address=145.79.227.37:4565 , piece_id=Y22BCACQP6GKMNCFN7L2AWGJHXEBEQYGCW7P6UHQTN5S4SAMH4JQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.379265586 (507h8m21.250348621s) [noise=true , address=145.79.227.48:4303 , piece_id=ANWVQPQ4HARTRLRYO3X2FGXWBTN35YCMRQZXSK534FZOGNAZLNPQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.394533419 (507h8m21.265616464s) [noise=true , address=145.79.227.8:4330 , piece_id=PSXZGJ3UMIYBVRDKCXRPR63ZYBUKZ75C6O24CAHSQYZK7KO2LPXQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.402674891 (507h8m21.273757966s) [noise=true , address=145.79.227.51:4409 , piece_id=AKO6JWAXFOMWPQHM7SOVGX4HABVCBHYTSKCUGWNP6N2VLEBDIRJQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.403498369 (507h8m21.274581414s) [noise=true , address=145.79.227.46:4602 , piece_id=NK6JINBNQOSTK6W4CDVR5SEUJVCQOLJE2KYDLAHID2G63YQTQCMQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.435802712 (507h8m21.306885747s) [noise=true , address=145.79.227.2:4115 , piece_id=EU56CW4GG76TO67THVLBF6RZAT6D4GD6MPYSRDIDY44IBIHBBXNQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.436628545 (507h8m21.30771158s) [noise=true , address=145.79.227.12:4767 , piece_id=T7A4IXELU3AYIXDV2W2EHFAFLN6XQC7UIT76F4OH3WYDPJ2RNSYQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.649643122 (507h8m21.520726167s) [noise=true , address=145.79.227.41:4086 , piece_id=3IUK67FIMMKTR7NXHRSHWM7JCOXZFICZT6PHWXDKBUAWSJC23VMA, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.739242897 (507h8m21.610325932s) [noise=true , address=45.91.116.12:4227 , piece_id=PWCCOYLWEYFEXKCWUOKVRUN6HTENOWNKAV75WY2L4AY34N66V35Q, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.979515096 (507h8m21.850598141s) [noise=true , address=77.90.31.10:4518 , piece_id=XN6HWG2K6V4DMLR42H2BOPVC2BE6RGNBTNGUQNOC6FYAYD7LPQ2Q, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:22.352800938 (507h8m22.223884013s) [noise=true , address=45.91.116.6:4976 , piece_id=NXLUQNJVFLGBOEPPCO2ZZOAECZAZT26SFU3KBFVBYUNZYLCAKRWA, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:22.352832238 (507h8m22.223915283s) [seg_pos={0 0}, upload=445016]: Segment reached optimal threshold of 13 pieces. Dec 26 18:37:22.353106590 (507h8m22.224189645s) [noise=true , address=145.79.227.14:4839 , piece_id=MAA4EEX7BSJKPCDQOGUO2QRT5O6ILJ3KFK7NOFSHWG357VAY737A, seg_pos={0 0}, upload=445016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:41048->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:41048->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41048->145.79.227.14:4839: use of closed network connection Dec 26 18:37:22.353116970 (507h8m22.224200025s) [noise=true , address=145.79.227.14:4839 , piece_id=MAA4EEX7BSJKPCDQOGUO2QRT5O6ILJ3KFK7NOFSHWG357VAY737A, seg_pos={0 0}, upload=445016]: Long tail cancellation due to optimal threshold Dec 26 18:37:22.353157867 (507h8m22.224240912s) [noise=true , address=45.91.116.5:4300 , piece_id=JMCPZRHD52SRLTLYJU3ACH37IBFMMHWEABV265SE5STCCOPHN4UA, seg_pos={0 0}, upload=445016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 26 18:37:22.353168328 (507h8m22.224251373s) [noise=true , address=45.91.116.5:4300 , piece_id=JMCPZRHD52SRLTLYJU3ACH37IBFMMHWEABV265SE5STCCOPHN4UA, seg_pos={0 0}, upload=445016]: Long tail cancellation due to optimal threshold Dec 26 18:37:22.353342600 (507h8m22.224425655s) [noise=true , address=77.90.31.16:4736 , piece_id=KO2NNFY4YDMRWD5BYBKHF7UUNAE2G5KICGDWDIHEJNFGUF7NIJBQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 18:37:22.353353791 (507h8m22.224436866s) [noise=true , address=77.90.31.16:4736 , piece_id=KO2NNFY4YDMRWD5BYBKHF7UUNAE2G5KICGDWDIHEJNFGUF7NIJBQ, seg_pos={0 0}, upload=445016]: Long tail cancellation due to optimal threshold Dec 26 18:37:22.353381794 (507h8m22.224464839s) [seg_pos={0 0}, upload=445016]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:37:22.353476835 (507h8m22.224559891s) [upload=445016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:37:22.353484730 (507h8m22.224567785s) [upload=445016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:37:22.655223604 (507h8m22.526306649s) [upload=445016]: Done uploading segments... Dec 26 18:37:22.655247399 (507h8m22.526330434s) [upload=445016]: Upload finished. err: <nil> Duration: 30.00398931s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXX.........X..X.....X...XX.........X.....X..X..X.X.X.....XX] duration: 30.003867418s trace: https://jaeger.dev.storj.io/trace/60e16220d283e446 FAIL (2025-12-26 18:36:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: Q767FIFGU7H2Y4QFF2R55DH7FVZWG5MR5DVLZZVKLZWONVOCMOEQ) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: WFPLES7OZICSROZHQLMTFUZNGHK5JTVRM3GOAQB77H5V6TUKPAUQ) context deadline exceeded ======LOGS====== Dec 26 18:36:18.561272816 (507h7m18.432355861s) [upload=445001]: Starting upload Dec 26 18:36:18.561380051 (507h7m18.432463107s) [upload=445001]: Done starting upload Dec 26 18:36:18.561425408 (507h7m18.432508453s) [upload=445001]: Uploading segments... Dec 26 18:36:18.561658351 (507h7m18.432741396s) [upload=445001]: Got next segment. Inline: false Dec 26 18:36:18.561708096 (507h7m18.432791141s) [upload=445001]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:36:18.561714317 (507h7m18.432797362s) [upload=445001]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:36:18.815151536 (507h7m18.686234571s) [seg_pos={0 0}, upload=445001]: Begin upload segment... Dec 26 18:36:18.815218373 (507h7m18.686301419s) [seg_pos={0 0}, upload=445001]: Done begin upload segment. Dec 26 18:36:18.815399159 (507h7m18.686482204s) [noise=true , address=145.79.227.2:4115 , piece_id=2ORBM67MVBOGC5RSRX4ESFWV4A44L3D4ZOC3G54CGKRPAWRDVITQ, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.815463340 (507h7m18.686546375s) [noise=true , address=77.90.31.19:4844 , piece_id=Q767FIFGU7H2Y4QFF2R55DH7FVZWG5MR5DVLZZVKLZWONVOCMOEQ, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.816148455 (507h7m18.6872315s) [noise=true , address=145.79.227.36:4517 , piece_id=FDASGOQS6SQ6XMEBMVXUTEVDIXIFECJEVWROWI6Q3DGGNPWDSJZQ, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.816925925 (507h7m18.68800897s) [noise=true , address=145.79.227.8:4330 , piece_id=ZJQL3S44BP7J3Z5MTG6RM4727DIGBDPSFVEW7XSC3U3SHGXVQNJA, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.817498035 (507h7m18.68858107s) [noise=true , address=45.91.116.5:4297 , piece_id=VPHNE435FLAOQNQJAL76RZW5UBM7LHIFGPA34QZVWF467VJNYUDA, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.818185514 (507h7m18.689268549s) [noise=true , address=145.79.227.11:4729 , piece_id=S7EBFZ6JYWSJZSY24PQ5WKYV7W6RDPJV7SVM3WC6UZ4EXU2QE6AQ, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.818761961 (507h7m18.689845006s) [noise=true , address=45.91.116.13:4153 , piece_id=EQ4AHINMJQ7W2QTMF4BLT2TG7PCKBKXOVHOZVILDSOUYGW4INMDQ, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.819104263 (507h7m18.690187308s) [noise=true , address=45.91.116.14:4190 , piece_id=BDGLJ3SAP7ERSBFBQAOMUHD2IVPYUFMC6FPE6KDGVPM6HBGI2GKQ, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.819529813 (507h7m18.690612848s) [noise=true , address=45.91.116.12:4225 , piece_id=Z74ZZ5HSG7AU754ZZFHO4O4FBTW52MQZM3F6PCFB3XIVANL6S5OQ, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.819797193 (507h7m18.690880248s) [noise=true , address=145.79.227.47:4265 , piece_id=EVZ4VW2SDCYT6HRUSPLGWLSRUHXB4TUJJGPQCRQFSDW3LDJFWZ5Q, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.820127171 (507h7m18.691210216s) [noise=true , address=45.91.116.10:4084 , piece_id=WFPLES7OZICSROZHQLMTFUZNGHK5JTVRM3GOAQB77H5V6TUKPAUQ, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.820500091 (507h7m18.691583136s) [noise=true , address=145.79.227.4:4185 , piece_id=SHMWXWXU7FYXWBQAPTDLRRW3K42LJUZF5DOKSF2NFRHPYZ5SOEIA, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.820709660 (507h7m18.691792695s) [noise=true , address=45.91.116.4:4937 , piece_id=S26AQ32EUAN3J7WX7IBIHGPOW36XQ7Z3ZYYZZOA42KXNAU5U4WMQ, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.821017587 (507h7m18.692100632s) [noise=true , address=145.79.227.42:4122 , piece_id=VJWHP3KIMEOAGHNDYOGPHXSLYBNYVYDGJ5NYLK5C2JOLOVTATZPA, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.821316797 (507h7m18.692399832s) [noise=true , address=45.91.116.6:4975 , piece_id=K3FG3BBKEMLUGKBOGZVK4BMOM5IIQ7XYGO47OSN2W664PKZXPLKQ, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.821783866 (507h7m18.692866911s) [noise=true , address=145.79.227.48:4301 , piece_id=23TYBE2NUXCLCMGWFGCBWMEIJELOOUH22PFJ6ZNICINGDZINTR3A, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:20.527472836 (507h7m20.398555871s) [upload=445001]: Next returned nil segment Dec 26 18:36:20.527485421 (507h7m20.398568456s) [upload=445001]: Waiting for error group managing segments... Dec 26 18:36:20.820006592 (507h7m20.691089637s) [noise=true , address=145.79.227.11:4729 , piece_id=S7EBFZ6JYWSJZSY24PQ5WKYV7W6RDPJV7SVM3WC6UZ4EXU2QE6AQ, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:20.824369199 (507h7m20.695452234s) [noise=true , address=145.79.227.36:4517 , piece_id=FDASGOQS6SQ6XMEBMVXUTEVDIXIFECJEVWROWI6Q3DGGNPWDSJZQ, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:20.827186353 (507h7m20.698269399s) [noise=true , address=145.79.227.48:4301 , piece_id=23TYBE2NUXCLCMGWFGCBWMEIJELOOUH22PFJ6ZNICINGDZINTR3A, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:20.830331523 (507h7m20.701414558s) [noise=true , address=145.79.227.8:4330 , piece_id=ZJQL3S44BP7J3Z5MTG6RM4727DIGBDPSFVEW7XSC3U3SHGXVQNJA, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:20.833317318 (507h7m20.704400363s) [noise=true , address=145.79.227.42:4122 , piece_id=VJWHP3KIMEOAGHNDYOGPHXSLYBNYVYDGJ5NYLK5C2JOLOVTATZPA, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:20.834319486 (507h7m20.705402531s) [noise=true , address=145.79.227.47:4265 , piece_id=EVZ4VW2SDCYT6HRUSPLGWLSRUHXB4TUJJGPQCRQFSDW3LDJFWZ5Q, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:21.051595078 (507h7m20.922678123s) [noise=true , address=45.91.116.4:4937 , piece_id=S26AQ32EUAN3J7WX7IBIHGPOW36XQ7Z3ZYYZZOA42KXNAU5U4WMQ, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:21.111565442 (507h7m20.982648477s) [noise=true , address=45.91.116.5:4297 , piece_id=VPHNE435FLAOQNQJAL76RZW5UBM7LHIFGPA34QZVWF467VJNYUDA, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:21.136100971 (507h7m21.007184016s) [noise=true , address=45.91.116.13:4153 , piece_id=EQ4AHINMJQ7W2QTMF4BLT2TG7PCKBKXOVHOZVILDSOUYGW4INMDQ, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:21.137392211 (507h7m21.008475256s) [noise=true , address=77.90.31.19:4844 , piece_id=Q767FIFGU7H2Y4QFF2R55DH7FVZWG5MR5DVLZZVKLZWONVOCMOEQ, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:21.151964037 (507h7m21.023047073s) [noise=true , address=45.91.116.10:4084 , piece_id=WFPLES7OZICSROZHQLMTFUZNGHK5JTVRM3GOAQB77H5V6TUKPAUQ, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:21.206211344 (507h7m21.077294389s) [noise=true , address=45.91.116.14:4190 , piece_id=BDGLJ3SAP7ERSBFBQAOMUHD2IVPYUFMC6FPE6KDGVPM6HBGI2GKQ, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:21.240102849 (507h7m21.111185894s) [noise=true , address=45.91.116.6:4975 , piece_id=K3FG3BBKEMLUGKBOGZVK4BMOM5IIQ7XYGO47OSN2W664PKZXPLKQ, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:21.240129961 (507h7m21.111212996s) [seg_pos={0 0}, upload=445001]: Segment reached optimal threshold of 13 pieces. Dec 26 18:36:21.240350941 (507h7m21.111433976s) [noise=true , address=145.79.227.4:4185 , piece_id=SHMWXWXU7FYXWBQAPTDLRRW3K42LJUZF5DOKSF2NFRHPYZ5SOEIA, seg_pos={0 0}, upload=445001]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Dec 26 18:36:21.240360350 (507h7m21.111443385s) [noise=true , address=145.79.227.4:4185 , piece_id=SHMWXWXU7FYXWBQAPTDLRRW3K42LJUZF5DOKSF2NFRHPYZ5SOEIA, seg_pos={0 0}, upload=445001]: Long tail cancellation due to optimal threshold Dec 26 18:36:21.240562724 (507h7m21.11164577s) [noise=true , address=45.91.116.12:4225 , piece_id=Z74ZZ5HSG7AU754ZZFHO4O4FBTW52MQZM3F6PCFB3XIVANL6S5OQ, seg_pos={0 0}, upload=445001]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 18:36:21.240582062 (507h7m21.111665097s) [noise=true , address=45.91.116.12:4225 , piece_id=Z74ZZ5HSG7AU754ZZFHO4O4FBTW52MQZM3F6PCFB3XIVANL6S5OQ, seg_pos={0 0}, upload=445001]: Long tail cancellation due to optimal threshold Dec 26 18:36:21.240618391 (507h7m21.111701426s) [noise=true , address=145.79.227.2:4115 , piece_id=2ORBM67MVBOGC5RSRX4ESFWV4A44L3D4ZOC3G54CGKRPAWRDVITQ, seg_pos={0 0}, upload=445001]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:39560->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:39560->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39560->145.79.227.2:4115: use of closed network connection Dec 26 18:36:21.240625835 (507h7m21.11170889s) [noise=true , address=145.79.227.2:4115 , piece_id=2ORBM67MVBOGC5RSRX4ESFWV4A44L3D4ZOC3G54CGKRPAWRDVITQ, seg_pos={0 0}, upload=445001]: Long tail cancellation due to optimal threshold Dec 26 18:36:21.240643208 (507h7m21.111726254s) [seg_pos={0 0}, upload=445001]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:36:21.240716929 (507h7m21.111799974s) [upload=445001]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:36:21.240721447 (507h7m21.111804482s) [upload=445001]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:36:21.538036307 (507h7m21.409119352s) [upload=445001]: Done uploading segments... Dec 26 18:36:21.538059622 (507h7m21.409142657s) [upload=445001]: Upload finished. err: <nil> Duration: 30.003888998s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XXX.........X..X.....X...XX.........X.....X..X..X.X.X.....X] duration: 30.002338367s trace: https://jaeger.dev.storj.io/trace/691da54eaa5c6952 FAIL (2025-12-26 18:35:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 18:35:08.500462979 (507h6m8.371546024s) [upload=444981]: Starting upload Dec 26 18:35:08.500546970 (507h6m8.371630025s) [upload=444981]: Done starting upload Dec 26 18:35:08.500640638 (507h6m8.371723673s) [upload=444981]: Uploading segments... Dec 26 18:35:08.500788039 (507h6m8.371871074s) [upload=444981]: Got next segment. Inline: false Dec 26 18:35:08.500834036 (507h6m8.371917081s) [upload=444981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:35:08.500840549 (507h6m8.371923584s) [upload=444981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:35:08.751122910 (507h6m8.622205955s) [seg_pos={0 0}, upload=444981]: Begin upload segment... Dec 26 18:35:08.751289948 (507h6m8.622372983s) [seg_pos={0 0}, upload=444981]: Done begin upload segment. Dec 26 18:35:08.751332198 (507h6m8.622415233s) [noise=true , address=145.79.227.8:4330 , piece_id=KWHBAURRCQEXOCB4XADLOOUDZIKQ7KYVAKEHDBRG4V2YUGTM7VFQ, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.755222266 (507h6m8.626305321s) [noise=true , address=145.79.227.21:4474 , piece_id=LUUPX2QTVJNDJE7GU6TTWE5KI7KXVQODJN6245XXI2CDZOMOZQFA, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.759907798 (507h6m8.630990843s) [noise=true , address=145.79.227.38:4979 , piece_id=TTBTN7CTRJU26U3B6AQKYE2KXCUXXY45AHOMBPJIOM2OGA6EZLJA, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.760161210 (507h6m8.631244255s) [noise=true , address=77.90.31.13:4627 , piece_id=FAFKNLLQWAPM6EIU7LGCHWFP4I6MSJQJEVCQQSZJZFNGQND2YQ5Q, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.762002797 (507h6m8.633085852s) [noise=true , address=145.79.227.17:4972 , piece_id=USNUVEGUL246EVFO66TLRDVNIMH6OBN2MTPE5M7H73ERX2JDSHBA, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.762182179 (507h6m8.633265214s) [noise=true , address=145.79.227.33:4870 , piece_id=NGODTYLGR2ZYWAZNRDBG5IQ5UAZVNJ2ETOEHTMSEEW2FHB5UI7BA, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.763686132 (507h6m8.634769177s) [noise=true , address=145.79.227.48:4301 , piece_id=ARQNMS6TPFPZRJJXGA3XLTG4DV5VQEY67EVSQPLHRRIC2KHO5CSA, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.764391937 (507h6m8.635474982s) [noise=true , address=45.91.116.10:4084 , piece_id=YYBEWDUMQDXAL6K5NX3G7ZHATAXMDZJXK3SLQ456OPU7MSNV6WPQ, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.765191059 (507h6m8.636274104s) [noise=true , address=145.79.227.5:4222 , piece_id=Q5227ZHY2JHTRO6QS3SOY7SQEBWCEH7RVLRY6LLYT5WDYBY7P2PA, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.766238232 (507h6m8.637321278s) [noise=true , address=145.79.227.52:4445 , piece_id=LLDIZQQDYSWK55YABI2EVGESCTBFN76B7MF6ZV5B6WSZ7HZKMFWA, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.766917786 (507h6m8.638000831s) [noise=true , address=45.91.116.6:4973 , piece_id=KG24MG5HRPDIRSHJVCOSPFTLAW33FSZGDT723IOWBGYZ4XLWWCBQ, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.768061193 (507h6m8.639144238s) [noise=true , address=145.79.227.35:4483 , piece_id=6KKDTL7KU2QDX2IX5T6KMFWC7Z4G2EVK5R62BPEN3UFALTH2YYSQ, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.768335235 (507h6m8.63941828s) [noise=true , address=145.79.227.37:4565 , piece_id=KUZANZAJOAWF2WR5B4DASW7XJHRSWY3GBBJ2UKLE46J2ULROD2AQ, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.769230450 (507h6m8.640313485s) [noise=true , address=77.90.31.16:4735 , piece_id=G5DCHTYQ4YDVOIY3UHCEWMOPXOHZHCR3LCVAMJBTEYKPS5MAQLZA, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.773034584 (507h6m8.644117619s) [noise=true , address=77.90.31.14:4663 , piece_id=KYWCYO5ZLTD3VZDLX4WGUPOIY7HLSYWL3XHRT7L4DWA2TMFVKJ4A, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.774840763 (507h6m8.645923798s) [noise=true , address=145.79.227.36:4519 , piece_id=YH3E6JN7HFLAXE7QFOWSEJIF42HGNAAJVHNR46LVTK5MOQM3YG5A, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:10.475630375 (507h6m10.34671342s) [upload=444981]: Next returned nil segment Dec 26 18:35:10.475649411 (507h6m10.346732456s) [upload=444981]: Waiting for error group managing segments... Dec 26 18:35:10.774340622 (507h6m10.645423667s) [noise=true , address=145.79.227.48:4301 , piece_id=ARQNMS6TPFPZRJJXGA3XLTG4DV5VQEY67EVSQPLHRRIC2KHO5CSA, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:10.779162854 (507h6m10.650245909s) [noise=true , address=145.79.227.8:4330 , piece_id=KWHBAURRCQEXOCB4XADLOOUDZIKQ7KYVAKEHDBRG4V2YUGTM7VFQ, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:10.783109640 (507h6m10.654192685s) [noise=true , address=145.79.227.36:4519 , piece_id=YH3E6JN7HFLAXE7QFOWSEJIF42HGNAAJVHNR46LVTK5MOQM3YG5A, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:10.787047348 (507h6m10.658130383s) [noise=true , address=145.79.227.37:4565 , piece_id=KUZANZAJOAWF2WR5B4DASW7XJHRSWY3GBBJ2UKLE46J2ULROD2AQ, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:10.789705799 (507h6m10.660788834s) [noise=true , address=145.79.227.21:4474 , piece_id=LUUPX2QTVJNDJE7GU6TTWE5KI7KXVQODJN6245XXI2CDZOMOZQFA, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:10.799441458 (507h6m10.670524493s) [noise=true , address=145.79.227.33:4870 , piece_id=NGODTYLGR2ZYWAZNRDBG5IQ5UAZVNJ2ETOEHTMSEEW2FHB5UI7BA, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:10.800499753 (507h6m10.671582788s) [noise=true , address=145.79.227.52:4445 , piece_id=LLDIZQQDYSWK55YABI2EVGESCTBFN76B7MF6ZV5B6WSZ7HZKMFWA, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:10.809186094 (507h6m10.680269129s) [noise=true , address=145.79.227.17:4972 , piece_id=USNUVEGUL246EVFO66TLRDVNIMH6OBN2MTPE5M7H73ERX2JDSHBA, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:10.850078059 (507h6m10.721161104s) [noise=true , address=145.79.227.38:4979 , piece_id=TTBTN7CTRJU26U3B6AQKYE2KXCUXXY45AHOMBPJIOM2OGA6EZLJA, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:11.019478927 (507h6m10.890561962s) [noise=true , address=77.90.31.13:4627 , piece_id=FAFKNLLQWAPM6EIU7LGCHWFP4I6MSJQJEVCQQSZJZFNGQND2YQ5Q, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:11.064226665 (507h6m10.93530971s) [noise=true , address=45.91.116.6:4973 , piece_id=KG24MG5HRPDIRSHJVCOSPFTLAW33FSZGDT723IOWBGYZ4XLWWCBQ, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:11.083232706 (507h6m10.954315751s) [noise=true , address=77.90.31.14:4663 , piece_id=KYWCYO5ZLTD3VZDLX4WGUPOIY7HLSYWL3XHRT7L4DWA2TMFVKJ4A, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:11.144338862 (507h6m11.015421908s) [noise=true , address=45.91.116.10:4084 , piece_id=YYBEWDUMQDXAL6K5NX3G7ZHATAXMDZJXK3SLQ456OPU7MSNV6WPQ, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:11.144365794 (507h6m11.015448839s) [seg_pos={0 0}, upload=444981]: Segment reached optimal threshold of 13 pieces. Dec 26 18:35:11.144626611 (507h6m11.015709656s) [noise=true , address=145.79.227.35:4483 , piece_id=6KKDTL7KU2QDX2IX5T6KMFWC7Z4G2EVK5R62BPEN3UFALTH2YYSQ, seg_pos={0 0}, upload=444981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 26 18:35:11.144636810 (507h6m11.015719855s) [noise=true , address=145.79.227.35:4483 , piece_id=6KKDTL7KU2QDX2IX5T6KMFWC7Z4G2EVK5R62BPEN3UFALTH2YYSQ, seg_pos={0 0}, upload=444981]: Long tail cancellation due to optimal threshold Dec 26 18:35:11.144691906 (507h6m11.015774941s) [noise=true , address=77.90.31.16:4735 , piece_id=G5DCHTYQ4YDVOIY3UHCEWMOPXOHZHCR3LCVAMJBTEYKPS5MAQLZA, seg_pos={0 0}, upload=444981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:38232->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:38232->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38232->77.90.31.16:4735: use of closed network connection Dec 26 18:35:11.144702576 (507h6m11.015785621s) [noise=true , address=77.90.31.16:4735 , piece_id=G5DCHTYQ4YDVOIY3UHCEWMOPXOHZHCR3LCVAMJBTEYKPS5MAQLZA, seg_pos={0 0}, upload=444981]: Long tail cancellation due to optimal threshold Dec 26 18:35:11.144902136 (507h6m11.015985181s) [noise=true , address=145.79.227.5:4222 , piece_id=Q5227ZHY2JHTRO6QS3SOY7SQEBWCEH7RVLRY6LLYT5WDYBY7P2PA, seg_pos={0 0}, upload=444981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Dec 26 18:35:11.144912606 (507h6m11.015995661s) [noise=true , address=145.79.227.5:4222 , piece_id=Q5227ZHY2JHTRO6QS3SOY7SQEBWCEH7RVLRY6LLYT5WDYBY7P2PA, seg_pos={0 0}, upload=444981]: Long tail cancellation due to optimal threshold Dec 26 18:35:11.144939287 (507h6m11.016022332s) [seg_pos={0 0}, upload=444981]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:35:11.145035599 (507h6m11.016118664s) [upload=444981]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:35:11.145045589 (507h6m11.016128634s) [upload=444981]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:35:11.433280097 (507h6m11.304363142s) [upload=444981]: Done uploading segments... Dec 26 18:35:11.433303552 (507h6m11.304386587s) [upload=444981]: Upload finished. err: <nil> Duration: 30.002359448s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.XXX.........X..X.....X...XX.........X.....X..X..X.X.X.....] duration: 6.886396358s trace: https://jaeger.dev.storj.io/trace/b06b9ed50506f0d FAIL (2025-12-26 18:34:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA) context deadline exceeded ======LOGS====== Dec 26 18:30:10.267967389 (507h1m10.139050425s) [upload=444902]: Starting upload Dec 26 18:30:10.268102978 (507h1m10.139186023s) [upload=444902]: Done starting upload Dec 26 18:30:10.268175255 (507h1m10.1392583s) [upload=444902]: Uploading segments... Dec 26 18:30:10.268312386 (507h1m10.139395432s) [upload=444902]: Got next segment. Inline: false Dec 26 18:30:10.268383913 (507h1m10.139466958s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:30:10.268390245 (507h1m10.139473291s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:30:10.519136000 (507h1m10.390219055s) [seg_pos={0 0}, upload=444902]: Begin upload segment... Dec 26 18:30:10.519192507 (507h1m10.390275552s) [seg_pos={0 0}, upload=444902]: Done begin upload segment. Dec 26 18:30:10.519226561 (507h1m10.390309607s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.519235028 (507h1m10.390318073s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520217739 (507h1m10.391300774s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520400928 (507h1m10.391483973s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520811620 (507h1m10.391894665s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521203566 (507h1m10.392286611s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521573962 (507h1m10.392657007s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521992619 (507h1m10.393075664s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522392450 (507h1m10.393475495s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522960172 (507h1m10.394043207s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.523354963 (507h1m10.394438008s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525289748 (507h1m10.396372783s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525792434 (507h1m10.396875479s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526279041 (507h1m10.397362086s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526812617 (507h1m10.397895652s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.527245061 (507h1m10.398328096s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:12.144404335 (507h1m12.01548738s) [upload=444902]: Next returned nil segment Dec 26 18:30:12.144486131 (507h1m12.015569196s) [upload=444902]: Waiting for error group managing segments... Dec 26 18:30:12.527130614 (507h1m12.398213659s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.530644123 (507h1m12.401727168s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.546942449 (507h1m12.418025494s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.559542481 (507h1m12.430625527s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.574166628 (507h1m12.445249673s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.630407832 (507h1m12.501490877s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.761971351 (507h1m12.633054396s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.785411075 (507h1m12.65649411s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.840918811 (507h1m12.712001856s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.467818829 (507h1m13.338901864s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.468849031 (507h1m13.339932076s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.496876309 (507h1m13.367959354s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015633615 (507h1m13.88671668s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015662509 (507h1m13.886745544s) [seg_pos={0 0}, upload=444902]: Segment reached optimal threshold of 13 pieces. Dec 26 18:30:14.015929808 (507h1m13.887012863s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015941300 (507h1m13.887024345s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.015954555 (507h1m13.887037611s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015961949 (507h1m13.887045045s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016280536 (507h1m13.887363591s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.016289863 (507h1m13.887372908s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016325411 (507h1m13.887408456s) [seg_pos={0 0}, upload=444902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:30:14.016440721 (507h1m13.887523766s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:30:14.016449808 (507h1m13.887532853s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:30:14.318047924 (507h1m14.189130969s) [upload=444902]: Done uploading segments... Dec 26 18:30:14.318072681 (507h1m14.189155716s) [upload=444902]: Upload finished. err: <nil> Duration: 6.8864291s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.XXX.........X..X.....X...XX.........X.....X..X..X.X.X....] duration: 22.35005192s trace: https://jaeger.dev.storj.io/trace/b06b9ed50506f0d FAIL (2025-12-26 18:33:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA) context deadline exceeded ======LOGS====== Dec 26 18:30:10.267967389 (507h1m10.139050425s) [upload=444902]: Starting upload Dec 26 18:30:10.268102978 (507h1m10.139186023s) [upload=444902]: Done starting upload Dec 26 18:30:10.268175255 (507h1m10.1392583s) [upload=444902]: Uploading segments... Dec 26 18:30:10.268312386 (507h1m10.139395432s) [upload=444902]: Got next segment. Inline: false Dec 26 18:30:10.268383913 (507h1m10.139466958s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:30:10.268390245 (507h1m10.139473291s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:30:10.519136000 (507h1m10.390219055s) [seg_pos={0 0}, upload=444902]: Begin upload segment... Dec 26 18:30:10.519192507 (507h1m10.390275552s) [seg_pos={0 0}, upload=444902]: Done begin upload segment. Dec 26 18:30:10.519226561 (507h1m10.390309607s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.519235028 (507h1m10.390318073s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520217739 (507h1m10.391300774s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520400928 (507h1m10.391483973s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520811620 (507h1m10.391894665s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521203566 (507h1m10.392286611s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521573962 (507h1m10.392657007s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521992619 (507h1m10.393075664s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522392450 (507h1m10.393475495s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522960172 (507h1m10.394043207s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.523354963 (507h1m10.394438008s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525289748 (507h1m10.396372783s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525792434 (507h1m10.396875479s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526279041 (507h1m10.397362086s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526812617 (507h1m10.397895652s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.527245061 (507h1m10.398328096s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:12.144404335 (507h1m12.01548738s) [upload=444902]: Next returned nil segment Dec 26 18:30:12.144486131 (507h1m12.015569196s) [upload=444902]: Waiting for error group managing segments... Dec 26 18:30:12.527130614 (507h1m12.398213659s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.530644123 (507h1m12.401727168s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.546942449 (507h1m12.418025494s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.559542481 (507h1m12.430625527s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.574166628 (507h1m12.445249673s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.630407832 (507h1m12.501490877s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.761971351 (507h1m12.633054396s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.785411075 (507h1m12.65649411s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.840918811 (507h1m12.712001856s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.467818829 (507h1m13.338901864s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.468849031 (507h1m13.339932076s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.496876309 (507h1m13.367959354s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015633615 (507h1m13.88671668s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015662509 (507h1m13.886745544s) [seg_pos={0 0}, upload=444902]: Segment reached optimal threshold of 13 pieces. Dec 26 18:30:14.015929808 (507h1m13.887012863s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015941300 (507h1m13.887024345s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.015954555 (507h1m13.887037611s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015961949 (507h1m13.887045045s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016280536 (507h1m13.887363591s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.016289863 (507h1m13.887372908s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016325411 (507h1m13.887408456s) [seg_pos={0 0}, upload=444902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:30:14.016440721 (507h1m13.887523766s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:30:14.016449808 (507h1m13.887532853s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:30:14.318047924 (507h1m14.189130969s) [upload=444902]: Done uploading segments... Dec 26 18:30:14.318072681 (507h1m14.189155716s) [upload=444902]: Upload finished. err: <nil> Duration: 22.350080384s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.X.XXX.........X..X.....X...XX.........X.....X..X..X.X.X...] duration: 6.160071027s trace: https://jaeger.dev.storj.io/trace/b06b9ed50506f0d FAIL (2025-12-26 18:32:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA) context deadline exceeded ======LOGS====== Dec 26 18:30:10.267967389 (507h1m10.139050425s) [upload=444902]: Starting upload Dec 26 18:30:10.268102978 (507h1m10.139186023s) [upload=444902]: Done starting upload Dec 26 18:30:10.268175255 (507h1m10.1392583s) [upload=444902]: Uploading segments... Dec 26 18:30:10.268312386 (507h1m10.139395432s) [upload=444902]: Got next segment. Inline: false Dec 26 18:30:10.268383913 (507h1m10.139466958s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:30:10.268390245 (507h1m10.139473291s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:30:10.519136000 (507h1m10.390219055s) [seg_pos={0 0}, upload=444902]: Begin upload segment... Dec 26 18:30:10.519192507 (507h1m10.390275552s) [seg_pos={0 0}, upload=444902]: Done begin upload segment. Dec 26 18:30:10.519226561 (507h1m10.390309607s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.519235028 (507h1m10.390318073s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520217739 (507h1m10.391300774s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520400928 (507h1m10.391483973s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520811620 (507h1m10.391894665s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521203566 (507h1m10.392286611s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521573962 (507h1m10.392657007s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521992619 (507h1m10.393075664s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522392450 (507h1m10.393475495s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522960172 (507h1m10.394043207s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.523354963 (507h1m10.394438008s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525289748 (507h1m10.396372783s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525792434 (507h1m10.396875479s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526279041 (507h1m10.397362086s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526812617 (507h1m10.397895652s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.527245061 (507h1m10.398328096s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:12.144404335 (507h1m12.01548738s) [upload=444902]: Next returned nil segment Dec 26 18:30:12.144486131 (507h1m12.015569196s) [upload=444902]: Waiting for error group managing segments... Dec 26 18:30:12.527130614 (507h1m12.398213659s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.530644123 (507h1m12.401727168s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.546942449 (507h1m12.418025494s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.559542481 (507h1m12.430625527s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.574166628 (507h1m12.445249673s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.630407832 (507h1m12.501490877s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.761971351 (507h1m12.633054396s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.785411075 (507h1m12.65649411s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.840918811 (507h1m12.712001856s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.467818829 (507h1m13.338901864s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.468849031 (507h1m13.339932076s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.496876309 (507h1m13.367959354s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015633615 (507h1m13.88671668s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015662509 (507h1m13.886745544s) [seg_pos={0 0}, upload=444902]: Segment reached optimal threshold of 13 pieces. Dec 26 18:30:14.015929808 (507h1m13.887012863s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015941300 (507h1m13.887024345s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.015954555 (507h1m13.887037611s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015961949 (507h1m13.887045045s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016280536 (507h1m13.887363591s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.016289863 (507h1m13.887372908s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016325411 (507h1m13.887408456s) [seg_pos={0 0}, upload=444902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:30:14.016440721 (507h1m13.887523766s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:30:14.016449808 (507h1m13.887532853s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:30:14.318047924 (507h1m14.189130969s) [upload=444902]: Done uploading segments... Dec 26 18:30:14.318072681 (507h1m14.189155716s) [upload=444902]: Upload finished. err: <nil> Duration: 6.160089902s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 41.499933054s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:32:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 41.499966446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX.X.XXX.........X..X.....X...XX.........X.....X..X..X.X.X..] duration: 8.443016355s trace: https://jaeger.dev.storj.io/trace/b06b9ed50506f0d FAIL (2025-12-26 18:32:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA) context deadline exceeded ======LOGS====== Dec 26 18:30:10.267967389 (507h1m10.139050425s) [upload=444902]: Starting upload Dec 26 18:30:10.268102978 (507h1m10.139186023s) [upload=444902]: Done starting upload Dec 26 18:30:10.268175255 (507h1m10.1392583s) [upload=444902]: Uploading segments... Dec 26 18:30:10.268312386 (507h1m10.139395432s) [upload=444902]: Got next segment. Inline: false Dec 26 18:30:10.268383913 (507h1m10.139466958s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:30:10.268390245 (507h1m10.139473291s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:30:10.519136000 (507h1m10.390219055s) [seg_pos={0 0}, upload=444902]: Begin upload segment... Dec 26 18:30:10.519192507 (507h1m10.390275552s) [seg_pos={0 0}, upload=444902]: Done begin upload segment. Dec 26 18:30:10.519226561 (507h1m10.390309607s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.519235028 (507h1m10.390318073s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520217739 (507h1m10.391300774s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520400928 (507h1m10.391483973s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520811620 (507h1m10.391894665s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521203566 (507h1m10.392286611s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521573962 (507h1m10.392657007s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521992619 (507h1m10.393075664s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522392450 (507h1m10.393475495s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522960172 (507h1m10.394043207s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.523354963 (507h1m10.394438008s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525289748 (507h1m10.396372783s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525792434 (507h1m10.396875479s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526279041 (507h1m10.397362086s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526812617 (507h1m10.397895652s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.527245061 (507h1m10.398328096s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:12.144404335 (507h1m12.01548738s) [upload=444902]: Next returned nil segment Dec 26 18:30:12.144486131 (507h1m12.015569196s) [upload=444902]: Waiting for error group managing segments... Dec 26 18:30:12.527130614 (507h1m12.398213659s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.530644123 (507h1m12.401727168s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.546942449 (507h1m12.418025494s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.559542481 (507h1m12.430625527s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.574166628 (507h1m12.445249673s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.630407832 (507h1m12.501490877s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.761971351 (507h1m12.633054396s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.785411075 (507h1m12.65649411s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.840918811 (507h1m12.712001856s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.467818829 (507h1m13.338901864s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.468849031 (507h1m13.339932076s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.496876309 (507h1m13.367959354s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015633615 (507h1m13.88671668s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015662509 (507h1m13.886745544s) [seg_pos={0 0}, upload=444902]: Segment reached optimal threshold of 13 pieces. Dec 26 18:30:14.015929808 (507h1m13.887012863s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015941300 (507h1m13.887024345s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.015954555 (507h1m13.887037611s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015961949 (507h1m13.887045045s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016280536 (507h1m13.887363591s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.016289863 (507h1m13.887372908s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016325411 (507h1m13.887408456s) [seg_pos={0 0}, upload=444902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:30:14.016440721 (507h1m13.887523766s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:30:14.016449808 (507h1m13.887532853s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:30:14.318047924 (507h1m14.189130969s) [upload=444902]: Done uploading segments... Dec 26 18:30:14.318072681 (507h1m14.189155716s) [upload=444902]: Upload finished. err: <nil> Duration: 8.443131955s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XX.X.XXX.........X..X.....X...XX.........X.....X..X..X.X.X.] duration: 29.698236632s trace: https://jaeger.dev.storj.io/trace/b06b9ed50506f0d FAIL (2025-12-26 18:31:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA) context deadline exceeded ======LOGS====== Dec 26 18:30:10.267967389 (507h1m10.139050425s) [upload=444902]: Starting upload Dec 26 18:30:10.268102978 (507h1m10.139186023s) [upload=444902]: Done starting upload Dec 26 18:30:10.268175255 (507h1m10.1392583s) [upload=444902]: Uploading segments... Dec 26 18:30:10.268312386 (507h1m10.139395432s) [upload=444902]: Got next segment. Inline: false Dec 26 18:30:10.268383913 (507h1m10.139466958s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:30:10.268390245 (507h1m10.139473291s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:30:10.519136000 (507h1m10.390219055s) [seg_pos={0 0}, upload=444902]: Begin upload segment... Dec 26 18:30:10.519192507 (507h1m10.390275552s) [seg_pos={0 0}, upload=444902]: Done begin upload segment. Dec 26 18:30:10.519226561 (507h1m10.390309607s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.519235028 (507h1m10.390318073s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520217739 (507h1m10.391300774s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520400928 (507h1m10.391483973s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520811620 (507h1m10.391894665s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521203566 (507h1m10.392286611s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521573962 (507h1m10.392657007s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521992619 (507h1m10.393075664s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522392450 (507h1m10.393475495s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522960172 (507h1m10.394043207s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.523354963 (507h1m10.394438008s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525289748 (507h1m10.396372783s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525792434 (507h1m10.396875479s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526279041 (507h1m10.397362086s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526812617 (507h1m10.397895652s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.527245061 (507h1m10.398328096s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:12.144404335 (507h1m12.01548738s) [upload=444902]: Next returned nil segment Dec 26 18:30:12.144486131 (507h1m12.015569196s) [upload=444902]: Waiting for error group managing segments... Dec 26 18:30:12.527130614 (507h1m12.398213659s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.530644123 (507h1m12.401727168s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.546942449 (507h1m12.418025494s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.559542481 (507h1m12.430625527s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.574166628 (507h1m12.445249673s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.630407832 (507h1m12.501490877s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.761971351 (507h1m12.633054396s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.785411075 (507h1m12.65649411s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.840918811 (507h1m12.712001856s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.467818829 (507h1m13.338901864s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.468849031 (507h1m13.339932076s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.496876309 (507h1m13.367959354s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015633615 (507h1m13.88671668s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015662509 (507h1m13.886745544s) [seg_pos={0 0}, upload=444902]: Segment reached optimal threshold of 13 pieces. Dec 26 18:30:14.015929808 (507h1m13.887012863s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015941300 (507h1m13.887024345s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.015954555 (507h1m13.887037611s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015961949 (507h1m13.887045045s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016280536 (507h1m13.887363591s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.016289863 (507h1m13.887372908s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016325411 (507h1m13.887408456s) [seg_pos={0 0}, upload=444902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:30:14.016440721 (507h1m13.887523766s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:30:14.016449808 (507h1m13.887532853s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:30:14.318047924 (507h1m14.189130969s) [upload=444902]: Done uploading segments... Dec 26 18:30:14.318072681 (507h1m14.189155716s) [upload=444902]: Upload finished. err: <nil> Duration: 29.698268424s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 35.095905512s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:31:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 35.095934728s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..XX.X.XXX.........X..X.....X...XX.........X.....X..X..X.X.X] duration: 30.003502052s trace: https://jaeger.dev.storj.io/trace/b06b9ed50506f0d FAIL (2025-12-26 18:30:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA) context deadline exceeded ======LOGS====== Dec 26 18:30:10.267967389 (507h1m10.139050425s) [upload=444902]: Starting upload Dec 26 18:30:10.268102978 (507h1m10.139186023s) [upload=444902]: Done starting upload Dec 26 18:30:10.268175255 (507h1m10.1392583s) [upload=444902]: Uploading segments... Dec 26 18:30:10.268312386 (507h1m10.139395432s) [upload=444902]: Got next segment. Inline: false Dec 26 18:30:10.268383913 (507h1m10.139466958s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:30:10.268390245 (507h1m10.139473291s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:30:10.519136000 (507h1m10.390219055s) [seg_pos={0 0}, upload=444902]: Begin upload segment... Dec 26 18:30:10.519192507 (507h1m10.390275552s) [seg_pos={0 0}, upload=444902]: Done begin upload segment. Dec 26 18:30:10.519226561 (507h1m10.390309607s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.519235028 (507h1m10.390318073s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520217739 (507h1m10.391300774s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520400928 (507h1m10.391483973s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520811620 (507h1m10.391894665s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521203566 (507h1m10.392286611s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521573962 (507h1m10.392657007s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521992619 (507h1m10.393075664s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522392450 (507h1m10.393475495s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522960172 (507h1m10.394043207s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.523354963 (507h1m10.394438008s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525289748 (507h1m10.396372783s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525792434 (507h1m10.396875479s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526279041 (507h1m10.397362086s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526812617 (507h1m10.397895652s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.527245061 (507h1m10.398328096s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:12.144404335 (507h1m12.01548738s) [upload=444902]: Next returned nil segment Dec 26 18:30:12.144486131 (507h1m12.015569196s) [upload=444902]: Waiting for error group managing segments... Dec 26 18:30:12.527130614 (507h1m12.398213659s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.530644123 (507h1m12.401727168s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.546942449 (507h1m12.418025494s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.559542481 (507h1m12.430625527s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.574166628 (507h1m12.445249673s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.630407832 (507h1m12.501490877s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.761971351 (507h1m12.633054396s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.785411075 (507h1m12.65649411s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.840918811 (507h1m12.712001856s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.467818829 (507h1m13.338901864s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.468849031 (507h1m13.339932076s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.496876309 (507h1m13.367959354s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015633615 (507h1m13.88671668s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015662509 (507h1m13.886745544s) [seg_pos={0 0}, upload=444902]: Segment reached optimal threshold of 13 pieces. Dec 26 18:30:14.015929808 (507h1m13.887012863s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015941300 (507h1m13.887024345s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.015954555 (507h1m13.887037611s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015961949 (507h1m13.887045045s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016280536 (507h1m13.887363591s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.016289863 (507h1m13.887372908s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016325411 (507h1m13.887408456s) [seg_pos={0 0}, upload=444902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:30:14.016440721 (507h1m13.887523766s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:30:14.016449808 (507h1m13.887532853s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:30:14.318047924 (507h1m14.189130969s) [upload=444902]: Done uploading segments... Dec 26 18:30:14.318072681 (507h1m14.189155716s) [upload=444902]: Upload finished. err: <nil> Duration: 30.003519515s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 33.32335266s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:30:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 33.323382418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...XX.X.XXX.........X..X.....X...XX.........X.....X..X..X.X.] duration: 23.440573065s trace: https://jaeger.dev.storj.io/trace/524d31df8f35c7ff FAIL (2025-12-26 18:29:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YAWBY3SI2SIZXGFL5RAQW5DY4HLQENKC366QNXRJKGNPLFPTLBRA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: VQJFFUVDIPOSJIYDS5VVV6IJKRUPHHFROUXPG7RWPD7XIYOHC2YA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: RVRJW5JPVMHSR5Y7RN6XQETQLA6LV726S4A5GYAWX2DC5DNUVDBQ) context deadline exceeded ======LOGS====== Dec 26 18:28:23.822347122 (506h59m23.693430157s) [upload=444874]: Starting upload Dec 26 18:28:23.822410964 (506h59m23.693493999s) [upload=444874]: Done starting upload Dec 26 18:28:23.822500815 (506h59m23.69358385s) [upload=444874]: Uploading segments... Dec 26 18:28:23.822596026 (506h59m23.693679071s) [upload=444874]: Got next segment. Inline: false Dec 26 18:28:23.822642454 (506h59m23.693725499s) [upload=444874]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:28:23.822649198 (506h59m23.693732253s) [upload=444874]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:28:24.074649680 (506h59m23.945732725s) [seg_pos={0 0}, upload=444874]: Begin upload segment... Dec 26 18:28:24.074791200 (506h59m23.945874235s) [seg_pos={0 0}, upload=444874]: Done begin upload segment. Dec 26 18:28:24.074824192 (506h59m23.945907237s) [noise=true , address=145.79.227.10:4401 , piece_id=SI3LHI5BPBS3PAJB74FGPESAKVO2IFGE6V4UVZZ7PLMFK3VRXNKQ, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.074855793 (506h59m23.945938828s) [noise=true , address=145.79.227.33:4871 , piece_id=4DQKLIJIMMHFO2W5TYL7I2DY5I3Z323EMBXYWVI4YWEHDW23ZC6A, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.075309056 (506h59m23.946392091s) [noise=true , address=145.79.227.45:4231 , piece_id=T6BOS3MCNPAMPZH3N6BSQNCSQ2R3WA4HKT563MCRVPJGBTYZJLUQ, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.075666957 (506h59m23.946750002s) [noise=true , address=145.79.227.13:4804 , piece_id=ZZKEBY64EAYOQUNXITN34OTKOVKVU265X5UPZQEKREBWJNNRPBPQ, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.076082839 (506h59m23.947165884s) [noise=true , address=45.91.116.5:4300 , piece_id=VQJFFUVDIPOSJIYDS5VVV6IJKRUPHHFROUXPG7RWPD7XIYOHC2YA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.076467922 (506h59m23.947550967s) [noise=true , address=45.91.116.8:4047 , piece_id=RVRJW5JPVMHSR5Y7RN6XQETQLA6LV726S4A5GYAWX2DC5DNUVDBQ, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.076757614 (506h59m23.947840649s) [noise=true , address=45.91.116.4:4940 , piece_id=MZDXJ7XZXICAC5NHIQ5UOGCQBJDOCPM2WFGGTCKIU43ITHOQP35A, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077078766 (506h59m23.948161801s) [noise=true , address=145.79.227.14:4839 , piece_id=3F5RR2FP2BGTP56WJBVEN2X4Q52ZSXX7AZFEDUW66JBFPHLPA6NA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077090618 (506h59m23.948173653s) [noise=true , address=145.79.227.16:4911 , piece_id=HC7BIPT556S6M3UNOKEPW2RYYCK5UI2DAC4ZMFKGXAURPLGASRQA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077126346 (506h59m23.948209381s) [noise=true , address=145.79.227.52:4445 , piece_id=BDIDHUSDZAQIFKEEN2S5MQLF3I6CD66LQ6HFILT53ODN4KXGYD7Q, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077472445 (506h59m23.94855548s) [noise=true , address=145.79.227.3:4149 , piece_id=DHDHV36CYF56CDUNSKLNZL37JD3MKY57LMCYIRZCAW46AFYMU3VA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077594388 (506h59m23.948677433s) [noise=true , address=45.91.116.14:4189 , piece_id=YAWBY3SI2SIZXGFL5RAQW5DY4HLQENKC366QNXRJKGNPLFPTLBRA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077874671 (506h59m23.948957706s) [noise=true , address=145.79.227.38:4979 , piece_id=DLV3K2DP7ASFVJ4XM2B24P5PXVE74OQ5CDT6CUQFBIZIZWNDKC2Q, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.078020459 (506h59m23.949103494s) [noise=true , address=45.91.116.13:4154 , piece_id=RJ2L576LCPQTSWW4UEBIN5VOUXJRTPQJ7PISMGAGM4MOMMTW37KA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.079200857 (506h59m23.950283902s) [noise=true , address=145.79.227.40:4050 , piece_id=VFX5GL57OCE7CFOFTME7JTQQEDXM4EK3PJFUWLFKVYLZTNENQGKA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.079589126 (506h59m23.950672171s) [noise=true , address=145.79.227.46:4603 , piece_id=ZHFEEQ6NOHP4TD5IA5GKTUPW76POCWCOQCSM33WT5JDARRSTKRKQ, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:25.779365970 (506h59m25.650449015s) [upload=444874]: Next returned nil segment Dec 26 18:28:25.779391659 (506h59m25.650474694s) [upload=444874]: Waiting for error group managing segments... Dec 26 18:28:26.066406545 (506h59m25.93748959s) [noise=true , address=145.79.227.10:4401 , piece_id=SI3LHI5BPBS3PAJB74FGPESAKVO2IFGE6V4UVZZ7PLMFK3VRXNKQ, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.083768736 (506h59m25.954851771s) [noise=true , address=145.79.227.46:4603 , piece_id=ZHFEEQ6NOHP4TD5IA5GKTUPW76POCWCOQCSM33WT5JDARRSTKRKQ, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.090536202 (506h59m25.961619237s) [noise=true , address=145.79.227.45:4231 , piece_id=T6BOS3MCNPAMPZH3N6BSQNCSQ2R3WA4HKT563MCRVPJGBTYZJLUQ, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.102954479 (506h59m25.974037514s) [noise=true , address=145.79.227.3:4149 , piece_id=DHDHV36CYF56CDUNSKLNZL37JD3MKY57LMCYIRZCAW46AFYMU3VA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.108136156 (506h59m25.979219191s) [noise=true , address=145.79.227.38:4979 , piece_id=DLV3K2DP7ASFVJ4XM2B24P5PXVE74OQ5CDT6CUQFBIZIZWNDKC2Q, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.109335761 (506h59m25.980418796s) [noise=true , address=145.79.227.13:4804 , piece_id=ZZKEBY64EAYOQUNXITN34OTKOVKVU265X5UPZQEKREBWJNNRPBPQ, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.114107036 (506h59m25.985190071s) [noise=true , address=145.79.227.52:4445 , piece_id=BDIDHUSDZAQIFKEEN2S5MQLF3I6CD66LQ6HFILT53ODN4KXGYD7Q, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.149774493 (506h59m26.020857528s) [noise=true , address=145.79.227.14:4839 , piece_id=3F5RR2FP2BGTP56WJBVEN2X4Q52ZSXX7AZFEDUW66JBFPHLPA6NA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.375250659 (506h59m26.246333704s) [noise=true , address=45.91.116.8:4047 , piece_id=RVRJW5JPVMHSR5Y7RN6XQETQLA6LV726S4A5GYAWX2DC5DNUVDBQ, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.427833536 (506h59m26.298916571s) [noise=true , address=45.91.116.14:4189 , piece_id=YAWBY3SI2SIZXGFL5RAQW5DY4HLQENKC366QNXRJKGNPLFPTLBRA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.443659091 (506h59m26.314742136s) [noise=true , address=45.91.116.13:4154 , piece_id=RJ2L576LCPQTSWW4UEBIN5VOUXJRTPQJ7PISMGAGM4MOMMTW37KA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.443997485 (506h59m26.31508053s) [noise=true , address=145.79.227.16:4911 , piece_id=HC7BIPT556S6M3UNOKEPW2RYYCK5UI2DAC4ZMFKGXAURPLGASRQA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.653512108 (506h59m26.524595143s) [noise=true , address=45.91.116.5:4300 , piece_id=VQJFFUVDIPOSJIYDS5VVV6IJKRUPHHFROUXPG7RWPD7XIYOHC2YA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.653537106 (506h59m26.524620151s) [seg_pos={0 0}, upload=444874]: Segment reached optimal threshold of 13 pieces. Dec 26 18:28:26.653883485 (506h59m26.524966531s) [noise=true , address=145.79.227.33:4871 , piece_id=4DQKLIJIMMHFO2W5TYL7I2DY5I3Z323EMBXYWVI4YWEHDW23ZC6A, seg_pos={0 0}, upload=444874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 18:28:26.653893424 (506h59m26.524976459s) [noise=true , address=145.79.227.33:4871 , piece_id=4DQKLIJIMMHFO2W5TYL7I2DY5I3Z323EMBXYWVI4YWEHDW23ZC6A, seg_pos={0 0}, upload=444874]: Long tail cancellation due to optimal threshold Dec 26 18:28:26.653977345 (506h59m26.52506038s) [noise=true , address=145.79.227.40:4050 , piece_id=VFX5GL57OCE7CFOFTME7JTQQEDXM4EK3PJFUWLFKVYLZTNENQGKA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 18:28:26.653994046 (506h59m26.525077081s) [noise=true , address=145.79.227.40:4050 , piece_id=VFX5GL57OCE7CFOFTME7JTQQEDXM4EK3PJFUWLFKVYLZTNENQGKA, seg_pos={0 0}, upload=444874]: Long tail cancellation due to optimal threshold Dec 26 18:28:26.654058188 (506h59m26.525141233s) [noise=true , address=45.91.116.4:4940 , piece_id=MZDXJ7XZXICAC5NHIQ5UOGCQBJDOCPM2WFGGTCKIU43ITHOQP35A, seg_pos={0 0}, upload=444874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 18:28:26.654067496 (506h59m26.525150531s) [noise=true , address=45.91.116.4:4940 , piece_id=MZDXJ7XZXICAC5NHIQ5UOGCQBJDOCPM2WFGGTCKIU43ITHOQP35A, seg_pos={0 0}, upload=444874]: Long tail cancellation due to optimal threshold Dec 26 18:28:26.654096902 (506h59m26.525179937s) [seg_pos={0 0}, upload=444874]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:28:26.654179278 (506h59m26.525262313s) [upload=444874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:28:26.654187445 (506h59m26.52527048s) [upload=444874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:28:26.972105415 (506h59m26.843188461s) [upload=444874]: Done uploading segments... Dec 26 18:28:26.972128840 (506h59m26.843211885s) [upload=444874]: Upload finished. err: <nil> Duration: 23.44060222s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 15.205669173s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:29:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 15.205702025s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 43.599200089s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:28:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 43.599226629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....XX.X.XXX.........X..X.....X...XX.........X.....X..X..X.X] duration: 30.002943049s trace: https://jaeger.dev.storj.io/trace/524d31df8f35c7ff FAIL (2025-12-26 18:28:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YAWBY3SI2SIZXGFL5RAQW5DY4HLQENKC366QNXRJKGNPLFPTLBRA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: VQJFFUVDIPOSJIYDS5VVV6IJKRUPHHFROUXPG7RWPD7XIYOHC2YA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: RVRJW5JPVMHSR5Y7RN6XQETQLA6LV726S4A5GYAWX2DC5DNUVDBQ) context deadline exceeded ======LOGS====== Dec 26 18:28:23.822347122 (506h59m23.693430157s) [upload=444874]: Starting upload Dec 26 18:28:23.822410964 (506h59m23.693493999s) [upload=444874]: Done starting upload Dec 26 18:28:23.822500815 (506h59m23.69358385s) [upload=444874]: Uploading segments... Dec 26 18:28:23.822596026 (506h59m23.693679071s) [upload=444874]: Got next segment. Inline: false Dec 26 18:28:23.822642454 (506h59m23.693725499s) [upload=444874]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:28:23.822649198 (506h59m23.693732253s) [upload=444874]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:28:24.074649680 (506h59m23.945732725s) [seg_pos={0 0}, upload=444874]: Begin upload segment... Dec 26 18:28:24.074791200 (506h59m23.945874235s) [seg_pos={0 0}, upload=444874]: Done begin upload segment. Dec 26 18:28:24.074824192 (506h59m23.945907237s) [noise=true , address=145.79.227.10:4401 , piece_id=SI3LHI5BPBS3PAJB74FGPESAKVO2IFGE6V4UVZZ7PLMFK3VRXNKQ, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.074855793 (506h59m23.945938828s) [noise=true , address=145.79.227.33:4871 , piece_id=4DQKLIJIMMHFO2W5TYL7I2DY5I3Z323EMBXYWVI4YWEHDW23ZC6A, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.075309056 (506h59m23.946392091s) [noise=true , address=145.79.227.45:4231 , piece_id=T6BOS3MCNPAMPZH3N6BSQNCSQ2R3WA4HKT563MCRVPJGBTYZJLUQ, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.075666957 (506h59m23.946750002s) [noise=true , address=145.79.227.13:4804 , piece_id=ZZKEBY64EAYOQUNXITN34OTKOVKVU265X5UPZQEKREBWJNNRPBPQ, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.076082839 (506h59m23.947165884s) [noise=true , address=45.91.116.5:4300 , piece_id=VQJFFUVDIPOSJIYDS5VVV6IJKRUPHHFROUXPG7RWPD7XIYOHC2YA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.076467922 (506h59m23.947550967s) [noise=true , address=45.91.116.8:4047 , piece_id=RVRJW5JPVMHSR5Y7RN6XQETQLA6LV726S4A5GYAWX2DC5DNUVDBQ, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.076757614 (506h59m23.947840649s) [noise=true , address=45.91.116.4:4940 , piece_id=MZDXJ7XZXICAC5NHIQ5UOGCQBJDOCPM2WFGGTCKIU43ITHOQP35A, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077078766 (506h59m23.948161801s) [noise=true , address=145.79.227.14:4839 , piece_id=3F5RR2FP2BGTP56WJBVEN2X4Q52ZSXX7AZFEDUW66JBFPHLPA6NA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077090618 (506h59m23.948173653s) [noise=true , address=145.79.227.16:4911 , piece_id=HC7BIPT556S6M3UNOKEPW2RYYCK5UI2DAC4ZMFKGXAURPLGASRQA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077126346 (506h59m23.948209381s) [noise=true , address=145.79.227.52:4445 , piece_id=BDIDHUSDZAQIFKEEN2S5MQLF3I6CD66LQ6HFILT53ODN4KXGYD7Q, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077472445 (506h59m23.94855548s) [noise=true , address=145.79.227.3:4149 , piece_id=DHDHV36CYF56CDUNSKLNZL37JD3MKY57LMCYIRZCAW46AFYMU3VA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077594388 (506h59m23.948677433s) [noise=true , address=45.91.116.14:4189 , piece_id=YAWBY3SI2SIZXGFL5RAQW5DY4HLQENKC366QNXRJKGNPLFPTLBRA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077874671 (506h59m23.948957706s) [noise=true , address=145.79.227.38:4979 , piece_id=DLV3K2DP7ASFVJ4XM2B24P5PXVE74OQ5CDT6CUQFBIZIZWNDKC2Q, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.078020459 (506h59m23.949103494s) [noise=true , address=45.91.116.13:4154 , piece_id=RJ2L576LCPQTSWW4UEBIN5VOUXJRTPQJ7PISMGAGM4MOMMTW37KA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.079200857 (506h59m23.950283902s) [noise=true , address=145.79.227.40:4050 , piece_id=VFX5GL57OCE7CFOFTME7JTQQEDXM4EK3PJFUWLFKVYLZTNENQGKA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.079589126 (506h59m23.950672171s) [noise=true , address=145.79.227.46:4603 , piece_id=ZHFEEQ6NOHP4TD5IA5GKTUPW76POCWCOQCSM33WT5JDARRSTKRKQ, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:25.779365970 (506h59m25.650449015s) [upload=444874]: Next returned nil segment Dec 26 18:28:25.779391659 (506h59m25.650474694s) [upload=444874]: Waiting for error group managing segments... Dec 26 18:28:26.066406545 (506h59m25.93748959s) [noise=true , address=145.79.227.10:4401 , piece_id=SI3LHI5BPBS3PAJB74FGPESAKVO2IFGE6V4UVZZ7PLMFK3VRXNKQ, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.083768736 (506h59m25.954851771s) [noise=true , address=145.79.227.46:4603 , piece_id=ZHFEEQ6NOHP4TD5IA5GKTUPW76POCWCOQCSM33WT5JDARRSTKRKQ, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.090536202 (506h59m25.961619237s) [noise=true , address=145.79.227.45:4231 , piece_id=T6BOS3MCNPAMPZH3N6BSQNCSQ2R3WA4HKT563MCRVPJGBTYZJLUQ, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.102954479 (506h59m25.974037514s) [noise=true , address=145.79.227.3:4149 , piece_id=DHDHV36CYF56CDUNSKLNZL37JD3MKY57LMCYIRZCAW46AFYMU3VA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.108136156 (506h59m25.979219191s) [noise=true , address=145.79.227.38:4979 , piece_id=DLV3K2DP7ASFVJ4XM2B24P5PXVE74OQ5CDT6CUQFBIZIZWNDKC2Q, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.109335761 (506h59m25.980418796s) [noise=true , address=145.79.227.13:4804 , piece_id=ZZKEBY64EAYOQUNXITN34OTKOVKVU265X5UPZQEKREBWJNNRPBPQ, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.114107036 (506h59m25.985190071s) [noise=true , address=145.79.227.52:4445 , piece_id=BDIDHUSDZAQIFKEEN2S5MQLF3I6CD66LQ6HFILT53ODN4KXGYD7Q, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.149774493 (506h59m26.020857528s) [noise=true , address=145.79.227.14:4839 , piece_id=3F5RR2FP2BGTP56WJBVEN2X4Q52ZSXX7AZFEDUW66JBFPHLPA6NA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.375250659 (506h59m26.246333704s) [noise=true , address=45.91.116.8:4047 , piece_id=RVRJW5JPVMHSR5Y7RN6XQETQLA6LV726S4A5GYAWX2DC5DNUVDBQ, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.427833536 (506h59m26.298916571s) [noise=true , address=45.91.116.14:4189 , piece_id=YAWBY3SI2SIZXGFL5RAQW5DY4HLQENKC366QNXRJKGNPLFPTLBRA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.443659091 (506h59m26.314742136s) [noise=true , address=45.91.116.13:4154 , piece_id=RJ2L576LCPQTSWW4UEBIN5VOUXJRTPQJ7PISMGAGM4MOMMTW37KA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.443997485 (506h59m26.31508053s) [noise=true , address=145.79.227.16:4911 , piece_id=HC7BIPT556S6M3UNOKEPW2RYYCK5UI2DAC4ZMFKGXAURPLGASRQA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.653512108 (506h59m26.524595143s) [noise=true , address=45.91.116.5:4300 , piece_id=VQJFFUVDIPOSJIYDS5VVV6IJKRUPHHFROUXPG7RWPD7XIYOHC2YA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.653537106 (506h59m26.524620151s) [seg_pos={0 0}, upload=444874]: Segment reached optimal threshold of 13 pieces. Dec 26 18:28:26.653883485 (506h59m26.524966531s) [noise=true , address=145.79.227.33:4871 , piece_id=4DQKLIJIMMHFO2W5TYL7I2DY5I3Z323EMBXYWVI4YWEHDW23ZC6A, seg_pos={0 0}, upload=444874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 18:28:26.653893424 (506h59m26.524976459s) [noise=true , address=145.79.227.33:4871 , piece_id=4DQKLIJIMMHFO2W5TYL7I2DY5I3Z323EMBXYWVI4YWEHDW23ZC6A, seg_pos={0 0}, upload=444874]: Long tail cancellation due to optimal threshold Dec 26 18:28:26.653977345 (506h59m26.52506038s) [noise=true , address=145.79.227.40:4050 , piece_id=VFX5GL57OCE7CFOFTME7JTQQEDXM4EK3PJFUWLFKVYLZTNENQGKA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 18:28:26.653994046 (506h59m26.525077081s) [noise=true , address=145.79.227.40:4050 , piece_id=VFX5GL57OCE7CFOFTME7JTQQEDXM4EK3PJFUWLFKVYLZTNENQGKA, seg_pos={0 0}, upload=444874]: Long tail cancellation due to optimal threshold Dec 26 18:28:26.654058188 (506h59m26.525141233s) [noise=true , address=45.91.116.4:4940 , piece_id=MZDXJ7XZXICAC5NHIQ5UOGCQBJDOCPM2WFGGTCKIU43ITHOQP35A, seg_pos={0 0}, upload=444874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 18:28:26.654067496 (506h59m26.525150531s) [noise=true , address=45.91.116.4:4940 , piece_id=MZDXJ7XZXICAC5NHIQ5UOGCQBJDOCPM2WFGGTCKIU43ITHOQP35A, seg_pos={0 0}, upload=444874]: Long tail cancellation due to optimal threshold Dec 26 18:28:26.654096902 (506h59m26.525179937s) [seg_pos={0 0}, upload=444874]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:28:26.654179278 (506h59m26.525262313s) [upload=444874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:28:26.654187445 (506h59m26.52527048s) [upload=444874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:28:26.972105415 (506h59m26.843188461s) [upload=444874]: Done uploading segments... Dec 26 18:28:26.972128840 (506h59m26.843211885s) [upload=444874]: Upload finished. err: <nil> Duration: 30.002968508s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....XX.X.XXX.........X..X.....X...XX.........X.....X..X..X.] duration: 16.154629659s trace: https://jaeger.dev.storj.io/trace/3e598be3dee99e8d FAIL (2025-12-26 18:27:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 18:27:04.814038872 (506h58m4.685121917s) [upload=444856]: Starting upload Dec 26 18:27:04.814093786 (506h58m4.685176822s) [upload=444856]: Done starting upload Dec 26 18:27:04.814313374 (506h58m4.685396409s) [upload=444856]: Uploading segments... Dec 26 18:27:04.814362368 (506h58m4.685445403s) [upload=444856]: Got next segment. Inline: false Dec 26 18:27:04.814388688 (506h58m4.685471723s) [upload=444856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:27:04.814392967 (506h58m4.685476012s) [upload=444856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:27:05.070533924 (506h58m4.941616959s) [seg_pos={0 0}, upload=444856]: Begin upload segment... Dec 26 18:27:05.070583608 (506h58m4.941666653s) [seg_pos={0 0}, upload=444856]: Done begin upload segment. Dec 26 18:27:05.070611822 (506h58m4.941694857s) [noise=true , address=145.79.227.43:4158 , piece_id=5NKCRIQULIWTV2K3YYD5RPLDAWUPJQ34OACLRDAAJOJVH6BZSN2A, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.070664092 (506h58m4.941747137s) [noise=true , address=45.91.116.9:4264 , piece_id=OGUTLNL4VVGZZ3QGW4PBWC2KXLZONWSPM6GDB3NMP4MWA5ALEBRA, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.071177449 (506h58m4.942260494s) [noise=true , address=77.90.31.18:4808 , piece_id=4GDMSXXA4GAHTKRZGMOOTFQ65E2Z2PJSVWC7KKNYQ6E66F2LVTQA, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.071661871 (506h58m4.942744907s) [noise=true , address=145.79.227.11:4729 , piece_id=QQ3G423RM3SS2QHZZBOLKQ6GN3ZR5QJPNEKOZIEGTVLUEPNHOAJQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.072017379 (506h58m4.943100414s) [noise=true , address=45.91.116.13:4156 , piece_id=WVDRVEBPKNGPRZ3S4W6YFZNCEUZXN54MHEZ3Y6BXFMSLKI3J4TTQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.072301139 (506h58m4.943384184s) [noise=true , address=45.91.116.5:4299 , piece_id=WRQSHAG5ZAIVPYF3ZN2QIEIM4B5XGBDPLQGL52JI7XV77P3SM7OA, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.072582205 (506h58m4.94366525s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWZBJE6QZY6C4NP6X75MOX2KHLPXMK2SIHX2Z235PAM3BG45Z7OQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.072878969 (506h58m4.943962014s) [noise=true , address=145.79.227.48:4303 , piece_id=XJHI2LYEK62QFIIYKMCIONNYHBVJKGEBMYPJZZDQN2QLOUL66Q6A, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.073248253 (506h58m4.944331288s) [noise=true , address=77.90.31.21:4880 , piece_id=DMXNEEWI6V5YRU5YGIAM26PXLG3GFP5OMAU4NTOGOZRGSUKWLTLQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.073591316 (506h58m4.944674361s) [noise=true , address=145.79.227.32:4836 , piece_id=XF7VQ7Q4DV42AGMN74RVKPY26QVFX3FVH726Z55CLGQ7MTTVFBUA, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.075023643 (506h58m4.946106688s) [noise=true , address=145.79.227.21:4475 , piece_id=SGRYERO6Q6DEFR7LZSFNWSTEFRB3Y3EI2HMBBABXKH2WHW7ZJ24A, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.075078929 (506h58m4.946161974s) [noise=true , address=145.79.227.10:4401 , piece_id=7FNANWRWX4ASK6LSH3EVYAMCAMIE7UAKBBNRY4TIFNPK63TROQMQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.076675308 (506h58m4.947758373s) [noise=true , address=145.79.227.39:4015 , piece_id=4TKQLJUJR2GRJCNPS7LZTNE5JKFWCARETVJ4RRELIC7GIYG7KKCQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.077105327 (506h58m4.948188362s) [noise=true , address=145.79.227.14:4839 , piece_id=ICNQQGVL4YX7MBUQUUYTIEFYD7BSXZ3JGKEI23IIE3PYABP5EO7A, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.080306162 (506h58m4.951389207s) [noise=true , address=145.79.227.36:4517 , piece_id=SOR4UXLCDUFWIPG7EQ753EO2ESIVUWWHMXL3VDXTB42N7J67VLHQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.083793563 (506h58m4.954876608s) [noise=true , address=77.90.31.16:4736 , piece_id=4YV4G6OUIWQDOAIQE7PYHAH7GFIPWQVFPPEZ5HXTXXNFQAF35S2A, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:06.692846281 (506h58m6.563929326s) [upload=444856]: Next returned nil segment Dec 26 18:27:06.692883091 (506h58m6.563966136s) [upload=444856]: Waiting for error group managing segments... Dec 26 18:27:07.079343870 (506h58m6.950426905s) [noise=true , address=145.79.227.11:4729 , piece_id=QQ3G423RM3SS2QHZZBOLKQ6GN3ZR5QJPNEKOZIEGTVLUEPNHOAJQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.080151929 (506h58m6.951234964s) [noise=true , address=145.79.227.36:4517 , piece_id=SOR4UXLCDUFWIPG7EQ753EO2ESIVUWWHMXL3VDXTB42N7J67VLHQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.083307027 (506h58m6.954390072s) [noise=true , address=145.79.227.39:4015 , piece_id=4TKQLJUJR2GRJCNPS7LZTNE5JKFWCARETVJ4RRELIC7GIYG7KKCQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.087252359 (506h58m6.958335394s) [noise=true , address=145.79.227.21:4475 , piece_id=SGRYERO6Q6DEFR7LZSFNWSTEFRB3Y3EI2HMBBABXKH2WHW7ZJ24A, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.090869848 (506h58m6.961952893s) [noise=true , address=145.79.227.32:4836 , piece_id=XF7VQ7Q4DV42AGMN74RVKPY26QVFX3FVH726Z55CLGQ7MTTVFBUA, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.101038851 (506h58m6.972121896s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWZBJE6QZY6C4NP6X75MOX2KHLPXMK2SIHX2Z235PAM3BG45Z7OQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.112619303 (506h58m6.983702348s) [noise=true , address=145.79.227.48:4303 , piece_id=XJHI2LYEK62QFIIYKMCIONNYHBVJKGEBMYPJZZDQN2QLOUL66Q6A, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.114945773 (506h58m6.986028808s) [noise=true , address=145.79.227.10:4401 , piece_id=7FNANWRWX4ASK6LSH3EVYAMCAMIE7UAKBBNRY4TIFNPK63TROQMQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.122221778 (506h58m6.993304823s) [noise=true , address=145.79.227.14:4839 , piece_id=ICNQQGVL4YX7MBUQUUYTIEFYD7BSXZ3JGKEI23IIE3PYABP5EO7A, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.361307158 (506h58m7.232390193s) [noise=true , address=45.91.116.9:4264 , piece_id=OGUTLNL4VVGZZ3QGW4PBWC2KXLZONWSPM6GDB3NMP4MWA5ALEBRA, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.472087023 (506h58m7.343170068s) [noise=true , address=45.91.116.5:4299 , piece_id=WRQSHAG5ZAIVPYF3ZN2QIEIM4B5XGBDPLQGL52JI7XV77P3SM7OA, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.481307370 (506h58m7.352390415s) [noise=true , address=45.91.116.13:4156 , piece_id=WVDRVEBPKNGPRZ3S4W6YFZNCEUZXN54MHEZ3Y6BXFMSLKI3J4TTQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:08.096007195 (506h58m7.96709024s) [noise=true , address=77.90.31.18:4808 , piece_id=4GDMSXXA4GAHTKRZGMOOTFQ65E2Z2PJSVWC7KKNYQ6E66F2LVTQA, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:08.096032524 (506h58m7.967115559s) [seg_pos={0 0}, upload=444856]: Segment reached optimal threshold of 13 pieces. Dec 26 18:27:08.096286228 (506h58m7.967369263s) [noise=true , address=145.79.227.43:4158 , piece_id=5NKCRIQULIWTV2K3YYD5RPLDAWUPJQ34OACLRDAAJOJVH6BZSN2A, seg_pos={0 0}, upload=444856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 26 18:27:08.096294854 (506h58m7.967377889s) [noise=true , address=145.79.227.43:4158 , piece_id=5NKCRIQULIWTV2K3YYD5RPLDAWUPJQ34OACLRDAAJOJVH6BZSN2A, seg_pos={0 0}, upload=444856]: Long tail cancellation due to optimal threshold Dec 26 18:27:08.096557273 (506h58m7.967640308s) [noise=true , address=77.90.31.16:4736 , piece_id=4YV4G6OUIWQDOAIQE7PYHAH7GFIPWQVFPPEZ5HXTXXNFQAF35S2A, seg_pos={0 0}, upload=444856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 18:27:08.096565530 (506h58m7.967648575s) [noise=true , address=77.90.31.16:4736 , piece_id=4YV4G6OUIWQDOAIQE7PYHAH7GFIPWQVFPPEZ5HXTXXNFQAF35S2A, seg_pos={0 0}, upload=444856]: Long tail cancellation due to optimal threshold Dec 26 18:27:08.096738468 (506h58m7.967821523s) [noise=true , address=77.90.31.21:4880 , piece_id=DMXNEEWI6V5YRU5YGIAM26PXLG3GFP5OMAU4NTOGOZRGSUKWLTLQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:45402->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:45402->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45402->77.90.31.21:4880: use of closed network connection Dec 26 18:27:08.096757885 (506h58m7.96784093s) [noise=true , address=77.90.31.21:4880 , piece_id=DMXNEEWI6V5YRU5YGIAM26PXLG3GFP5OMAU4NTOGOZRGSUKWLTLQ, seg_pos={0 0}, upload=444856]: Long tail cancellation due to optimal threshold Dec 26 18:27:08.096785638 (506h58m7.967868673s) [seg_pos={0 0}, upload=444856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:27:08.096877743 (506h58m7.967960788s) [upload=444856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:27:08.096884136 (506h58m7.967967171s) [upload=444856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:27:08.398825365 (506h58m8.26990841s) [upload=444856]: Done uploading segments... Dec 26 18:27:08.398846124 (506h58m8.269929159s) [upload=444856]: Upload finished. err: <nil> Duration: 16.154656349s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 28.774871694s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:27:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 28.77489012s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.....XX.X.XXX.........X..X.....X...XX.........X.....X..X..X] duration: 30.002862384s trace: https://jaeger.dev.storj.io/trace/3e598be3dee99e8d FAIL (2025-12-26 18:27:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 18:27:04.814038872 (506h58m4.685121917s) [upload=444856]: Starting upload Dec 26 18:27:04.814093786 (506h58m4.685176822s) [upload=444856]: Done starting upload Dec 26 18:27:04.814313374 (506h58m4.685396409s) [upload=444856]: Uploading segments... Dec 26 18:27:04.814362368 (506h58m4.685445403s) [upload=444856]: Got next segment. Inline: false Dec 26 18:27:04.814388688 (506h58m4.685471723s) [upload=444856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:27:04.814392967 (506h58m4.685476012s) [upload=444856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:27:05.070533924 (506h58m4.941616959s) [seg_pos={0 0}, upload=444856]: Begin upload segment... Dec 26 18:27:05.070583608 (506h58m4.941666653s) [seg_pos={0 0}, upload=444856]: Done begin upload segment. Dec 26 18:27:05.070611822 (506h58m4.941694857s) [noise=true , address=145.79.227.43:4158 , piece_id=5NKCRIQULIWTV2K3YYD5RPLDAWUPJQ34OACLRDAAJOJVH6BZSN2A, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.070664092 (506h58m4.941747137s) [noise=true , address=45.91.116.9:4264 , piece_id=OGUTLNL4VVGZZ3QGW4PBWC2KXLZONWSPM6GDB3NMP4MWA5ALEBRA, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.071177449 (506h58m4.942260494s) [noise=true , address=77.90.31.18:4808 , piece_id=4GDMSXXA4GAHTKRZGMOOTFQ65E2Z2PJSVWC7KKNYQ6E66F2LVTQA, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.071661871 (506h58m4.942744907s) [noise=true , address=145.79.227.11:4729 , piece_id=QQ3G423RM3SS2QHZZBOLKQ6GN3ZR5QJPNEKOZIEGTVLUEPNHOAJQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.072017379 (506h58m4.943100414s) [noise=true , address=45.91.116.13:4156 , piece_id=WVDRVEBPKNGPRZ3S4W6YFZNCEUZXN54MHEZ3Y6BXFMSLKI3J4TTQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.072301139 (506h58m4.943384184s) [noise=true , address=45.91.116.5:4299 , piece_id=WRQSHAG5ZAIVPYF3ZN2QIEIM4B5XGBDPLQGL52JI7XV77P3SM7OA, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.072582205 (506h58m4.94366525s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWZBJE6QZY6C4NP6X75MOX2KHLPXMK2SIHX2Z235PAM3BG45Z7OQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.072878969 (506h58m4.943962014s) [noise=true , address=145.79.227.48:4303 , piece_id=XJHI2LYEK62QFIIYKMCIONNYHBVJKGEBMYPJZZDQN2QLOUL66Q6A, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.073248253 (506h58m4.944331288s) [noise=true , address=77.90.31.21:4880 , piece_id=DMXNEEWI6V5YRU5YGIAM26PXLG3GFP5OMAU4NTOGOZRGSUKWLTLQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.073591316 (506h58m4.944674361s) [noise=true , address=145.79.227.32:4836 , piece_id=XF7VQ7Q4DV42AGMN74RVKPY26QVFX3FVH726Z55CLGQ7MTTVFBUA, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.075023643 (506h58m4.946106688s) [noise=true , address=145.79.227.21:4475 , piece_id=SGRYERO6Q6DEFR7LZSFNWSTEFRB3Y3EI2HMBBABXKH2WHW7ZJ24A, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.075078929 (506h58m4.946161974s) [noise=true , address=145.79.227.10:4401 , piece_id=7FNANWRWX4ASK6LSH3EVYAMCAMIE7UAKBBNRY4TIFNPK63TROQMQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.076675308 (506h58m4.947758373s) [noise=true , address=145.79.227.39:4015 , piece_id=4TKQLJUJR2GRJCNPS7LZTNE5JKFWCARETVJ4RRELIC7GIYG7KKCQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.077105327 (506h58m4.948188362s) [noise=true , address=145.79.227.14:4839 , piece_id=ICNQQGVL4YX7MBUQUUYTIEFYD7BSXZ3JGKEI23IIE3PYABP5EO7A, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.080306162 (506h58m4.951389207s) [noise=true , address=145.79.227.36:4517 , piece_id=SOR4UXLCDUFWIPG7EQ753EO2ESIVUWWHMXL3VDXTB42N7J67VLHQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.083793563 (506h58m4.954876608s) [noise=true , address=77.90.31.16:4736 , piece_id=4YV4G6OUIWQDOAIQE7PYHAH7GFIPWQVFPPEZ5HXTXXNFQAF35S2A, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:06.692846281 (506h58m6.563929326s) [upload=444856]: Next returned nil segment Dec 26 18:27:06.692883091 (506h58m6.563966136s) [upload=444856]: Waiting for error group managing segments... Dec 26 18:27:07.079343870 (506h58m6.950426905s) [noise=true , address=145.79.227.11:4729 , piece_id=QQ3G423RM3SS2QHZZBOLKQ6GN3ZR5QJPNEKOZIEGTVLUEPNHOAJQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.080151929 (506h58m6.951234964s) [noise=true , address=145.79.227.36:4517 , piece_id=SOR4UXLCDUFWIPG7EQ753EO2ESIVUWWHMXL3VDXTB42N7J67VLHQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.083307027 (506h58m6.954390072s) [noise=true , address=145.79.227.39:4015 , piece_id=4TKQLJUJR2GRJCNPS7LZTNE5JKFWCARETVJ4RRELIC7GIYG7KKCQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.087252359 (506h58m6.958335394s) [noise=true , address=145.79.227.21:4475 , piece_id=SGRYERO6Q6DEFR7LZSFNWSTEFRB3Y3EI2HMBBABXKH2WHW7ZJ24A, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.090869848 (506h58m6.961952893s) [noise=true , address=145.79.227.32:4836 , piece_id=XF7VQ7Q4DV42AGMN74RVKPY26QVFX3FVH726Z55CLGQ7MTTVFBUA, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.101038851 (506h58m6.972121896s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWZBJE6QZY6C4NP6X75MOX2KHLPXMK2SIHX2Z235PAM3BG45Z7OQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.112619303 (506h58m6.983702348s) [noise=true , address=145.79.227.48:4303 , piece_id=XJHI2LYEK62QFIIYKMCIONNYHBVJKGEBMYPJZZDQN2QLOUL66Q6A, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.114945773 (506h58m6.986028808s) [noise=true , address=145.79.227.10:4401 , piece_id=7FNANWRWX4ASK6LSH3EVYAMCAMIE7UAKBBNRY4TIFNPK63TROQMQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.122221778 (506h58m6.993304823s) [noise=true , address=145.79.227.14:4839 , piece_id=ICNQQGVL4YX7MBUQUUYTIEFYD7BSXZ3JGKEI23IIE3PYABP5EO7A, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.361307158 (506h58m7.232390193s) [noise=true , address=45.91.116.9:4264 , piece_id=OGUTLNL4VVGZZ3QGW4PBWC2KXLZONWSPM6GDB3NMP4MWA5ALEBRA, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.472087023 (506h58m7.343170068s) [noise=true , address=45.91.116.5:4299 , piece_id=WRQSHAG5ZAIVPYF3ZN2QIEIM4B5XGBDPLQGL52JI7XV77P3SM7OA, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.481307370 (506h58m7.352390415s) [noise=true , address=45.91.116.13:4156 , piece_id=WVDRVEBPKNGPRZ3S4W6YFZNCEUZXN54MHEZ3Y6BXFMSLKI3J4TTQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:08.096007195 (506h58m7.96709024s) [noise=true , address=77.90.31.18:4808 , piece_id=4GDMSXXA4GAHTKRZGMOOTFQ65E2Z2PJSVWC7KKNYQ6E66F2LVTQA, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:08.096032524 (506h58m7.967115559s) [seg_pos={0 0}, upload=444856]: Segment reached optimal threshold of 13 pieces. Dec 26 18:27:08.096286228 (506h58m7.967369263s) [noise=true , address=145.79.227.43:4158 , piece_id=5NKCRIQULIWTV2K3YYD5RPLDAWUPJQ34OACLRDAAJOJVH6BZSN2A, seg_pos={0 0}, upload=444856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 26 18:27:08.096294854 (506h58m7.967377889s) [noise=true , address=145.79.227.43:4158 , piece_id=5NKCRIQULIWTV2K3YYD5RPLDAWUPJQ34OACLRDAAJOJVH6BZSN2A, seg_pos={0 0}, upload=444856]: Long tail cancellation due to optimal threshold Dec 26 18:27:08.096557273 (506h58m7.967640308s) [noise=true , address=77.90.31.16:4736 , piece_id=4YV4G6OUIWQDOAIQE7PYHAH7GFIPWQVFPPEZ5HXTXXNFQAF35S2A, seg_pos={0 0}, upload=444856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 18:27:08.096565530 (506h58m7.967648575s) [noise=true , address=77.90.31.16:4736 , piece_id=4YV4G6OUIWQDOAIQE7PYHAH7GFIPWQVFPPEZ5HXTXXNFQAF35S2A, seg_pos={0 0}, upload=444856]: Long tail cancellation due to optimal threshold Dec 26 18:27:08.096738468 (506h58m7.967821523s) [noise=true , address=77.90.31.21:4880 , piece_id=DMXNEEWI6V5YRU5YGIAM26PXLG3GFP5OMAU4NTOGOZRGSUKWLTLQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:45402->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:45402->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45402->77.90.31.21:4880: use of closed network connection Dec 26 18:27:08.096757885 (506h58m7.96784093s) [noise=true , address=77.90.31.21:4880 , piece_id=DMXNEEWI6V5YRU5YGIAM26PXLG3GFP5OMAU4NTOGOZRGSUKWLTLQ, seg_pos={0 0}, upload=444856]: Long tail cancellation due to optimal threshold Dec 26 18:27:08.096785638 (506h58m7.967868673s) [seg_pos={0 0}, upload=444856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:27:08.096877743 (506h58m7.967960788s) [upload=444856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:27:08.096884136 (506h58m7.967967171s) [upload=444856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:27:08.398825365 (506h58m8.26990841s) [upload=444856]: Done uploading segments... Dec 26 18:27:08.398846124 (506h58m8.269929159s) [upload=444856]: Upload finished. err: <nil> Duration: 30.00288679s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 34.515609263s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:27:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 34.515639021s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XX.....XX.X.XXX.........X..X.....X...XX.........X.....X..X..] duration: 9.17548808s trace: https://jaeger.dev.storj.io/trace/4d484ce3fc14ae0 FAIL (2025-12-26 18:26:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: VKGGWIBE6RX7AFJUBKJ2MSAEABXCUQQG3X36JWTCR6UG73WXVBWQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: CTQ5I3FGJGLOFFSWZC6ZNPYI5RFZBQJTZ2DTDMY4CXTGNCCEZEMQ) context deadline exceeded ======LOGS====== 6 18:25:19.666927494 (506h56m19.538010539s) [upload=444830]: Starting upload Dec 26 18:25:19.667038895 (506h56m19.538121961s) [upload=444830]: Done starting upload Dec 26 18:25:19.667174374 (506h56m19.538257409s) [upload=444830]: Uploading segments... Dec 26 18:25:19.667258674 (506h56m19.538341719s) [upload=444830]: Got next segment. Inline: false Dec 26 18:25:19.667296165 (506h56m19.5383792s) [upload=444830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:25:19.667301856 (506h56m19.538384901s) [upload=444830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:25:19.918625050 (506h56m19.789708105s) [seg_pos={0 0}, upload=444830]: Begin upload segment... Dec 26 18:25:19.918761218 (506h56m19.789844253s) [seg_pos={0 0}, upload=444830]: Done begin upload segment. Dec 26 18:25:19.918834739 (506h56m19.789917784s) [noise=true , address=145.79.227.22:4511 , piece_id=BMFAFUOM3HZAXCDJV2RB3QPI453KK23HPM6FDJJCF3ELECANLVFA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.918928477 (506h56m19.790011522s) [noise=true , address=45.91.116.4:4937 , piece_id=DOM3EPX5HZBZR3VGVEO5WNPI5H7YDOBTNKTEPX5IA5GW5YBKHF2A, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.919612670 (506h56m19.790695705s) [noise=true , address=145.79.227.47:4267 , piece_id=F6AGT52QXC2SOWGU5XOOG4PJK2OCCCFUU3OMLCV5E2FIJ45SATGQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.919839992 (506h56m19.790923037s) [noise=true , address=145.79.227.51:4409 , piece_id=D2G7AM2VNS77FUDFFAZ5IJDI3HGNJYMKQUK4Z5JUBL3VSFRT4ZLQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.919995649 (506h56m19.791078694s) [noise=true , address=77.90.31.18:4808 , piece_id=XOW4J4T67WQW7HUORTKM2MHDYLJKBZYBXOJ5XNYLQEJJTEFLWLPA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920235296 (506h56m19.791318341s) [noise=true , address=45.91.116.5:4300 , piece_id=CTQ5I3FGJGLOFFSWZC6ZNPYI5RFZBQJTZ2DTDMY4CXTGNCCEZEMQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920356225 (506h56m19.79143927s) [noise=true , address=145.79.227.19:4077 , piece_id=EVVLHQGGWORCGY4Z2ZS5MBKWTHJNK7ASJXCSGDG4SLGIXFBGTVGQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920575754 (506h56m19.791658799s) [noise=true , address=145.79.227.49:4339 , piece_id=5BRHLKVAJ2S7LBH5CH5SE3YJN5LSX4IM2IYWZWIUOTF73LMCWPMA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920713106 (506h56m19.791796151s) [noise=true , address=145.79.227.42:4122 , piece_id=ULH3XHAYVXRIQKOTGUTUYIU5M4T2XGQ7UJLKDW4PGPUIDSLQP4IA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920967009 (506h56m19.792050054s) [noise=true , address=145.79.227.16:4909 , piece_id=TUITUPXYBG43YVFXHDW7LKNTIQWHBDXBM3E3D75PPZ5OVGVTC76Q, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921151460 (506h56m19.792234495s) [noise=true , address=145.79.227.3:4151 , piece_id=5H6T6Q3AVDFBA3IE7NNZ7JD5PGSFWLAURSET2HV2EDPKLQWYFHDA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921344848 (506h56m19.792427884s) [noise=true , address=145.79.227.5:4221 , piece_id=H5GK6DCSW55XEWO4IPOG5RVDXLDDNSBAZ3MEUMJ7E3YNVQIAXOKA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921570267 (506h56m19.792653312s) [noise=true , address=145.79.227.21:4474 , piece_id=46DB3FQKA33EDW7GXVXMSDPI3ZEOUR3ADP3WGLRT4X4GXKBN2H7Q, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921714832 (506h56m19.792797877s) [noise=true , address=145.79.227.37:4565 , piece_id=3VJMU5PIWEXDUBP6KGY3JK5J5HYWZ64XZFSZNXL2PPYAE2NWK4OQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.922067303 (506h56m19.793150348s) [noise=true , address=45.91.116.7:4012 , piece_id=VKGGWIBE6RX7AFJUBKJ2MSAEABXCUQQG3X36JWTCR6UG73WXVBWQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.922212541 (506h56m19.793295576s) [noise=true , address=145.79.227.39:4013 , piece_id=CSM5J3CPJ4AYOKHSVLBYIM6T7VGFRGMKBAH7BYEKJWHYLBJ4FMFA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:21.625781415 (506h56m21.49686449s) [upload=444830]: Next returned nil segment Dec 26 18:25:21.625838002 (506h56m21.496921047s) [upload=444830]: Waiting for error group managing segments... Dec 26 18:25:21.919637218 (506h56m21.790720253s) [noise=true , address=145.79.227.51:4409 , piece_id=D2G7AM2VNS77FUDFFAZ5IJDI3HGNJYMKQUK4Z5JUBL3VSFRT4ZLQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.920253681 (506h56m21.791336727s) [noise=true , address=145.79.227.39:4013 , piece_id=CSM5J3CPJ4AYOKHSVLBYIM6T7VGFRGMKBAH7BYEKJWHYLBJ4FMFA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.929953431 (506h56m21.801036466s) [noise=true , address=145.79.227.22:4511 , piece_id=BMFAFUOM3HZAXCDJV2RB3QPI453KK23HPM6FDJJCF3ELECANLVFA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.934488106 (506h56m21.805571141s) [noise=true , address=145.79.227.21:4474 , piece_id=46DB3FQKA33EDW7GXVXMSDPI3ZEOUR3ADP3WGLRT4X4GXKBN2H7Q, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.943745083 (506h56m21.814828128s) [noise=true , address=145.79.227.37:4565 , piece_id=3VJMU5PIWEXDUBP6KGY3JK5J5HYWZ64XZFSZNXL2PPYAE2NWK4OQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.946938384 (506h56m21.818021429s) [noise=true , address=145.79.227.42:4122 , piece_id=ULH3XHAYVXRIQKOTGUTUYIU5M4T2XGQ7UJLKDW4PGPUIDSLQP4IA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.950590989 (506h56m21.821674024s) [noise=true , address=145.79.227.16:4909 , piece_id=TUITUPXYBG43YVFXHDW7LKNTIQWHBDXBM3E3D75PPZ5OVGVTC76Q, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.952817939 (506h56m21.823900974s) [noise=true , address=145.79.227.49:4339 , piece_id=5BRHLKVAJ2S7LBH5CH5SE3YJN5LSX4IM2IYWZWIUOTF73LMCWPMA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.954184631 (506h56m21.825267677s) [noise=true , address=145.79.227.47:4267 , piece_id=F6AGT52QXC2SOWGU5XOOG4PJK2OCCCFUU3OMLCV5E2FIJ45SATGQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.092833324 (506h56m21.963916359s) [noise=true , address=77.90.31.18:4808 , piece_id=XOW4J4T67WQW7HUORTKM2MHDYLJKBZYBXOJ5XNYLQEJJTEFLWLPA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.291735528 (506h56m22.162818583s) [noise=true , address=45.91.116.4:4937 , piece_id=DOM3EPX5HZBZR3VGVEO5WNPI5H7YDOBTNKTEPX5IA5GW5YBKHF2A, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.364544489 (506h56m22.235627534s) [noise=true , address=45.91.116.7:4012 , piece_id=VKGGWIBE6RX7AFJUBKJ2MSAEABXCUQQG3X36JWTCR6UG73WXVBWQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.842766488 (506h56m22.713849533s) [noise=true , address=45.91.116.5:4300 , piece_id=CTQ5I3FGJGLOFFSWZC6ZNPYI5RFZBQJTZ2DTDMY4CXTGNCCEZEMQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.842794591 (506h56m22.713877636s) [seg_pos={0 0}, upload=444830]: Segment reached optimal threshold of 13 pieces. Dec 26 18:25:22.843025661 (506h56m22.714108706s) [noise=true , address=145.79.227.3:4151 , piece_id=5H6T6Q3AVDFBA3IE7NNZ7JD5PGSFWLAURSET2HV2EDPKLQWYFHDA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Dec 26 18:25:22.843036923 (506h56m22.714119968s) [noise=true , address=145.79.227.3:4151 , piece_id=5H6T6Q3AVDFBA3IE7NNZ7JD5PGSFWLAURSET2HV2EDPKLQWYFHDA, seg_pos={0 0}, upload=444830]: Long tail cancellation due to optimal threshold Dec 26 18:25:22.843137895 (506h56m22.71422094s) [noise=true , address=145.79.227.19:4077 , piece_id=EVVLHQGGWORCGY4Z2ZS5MBKWTHJNK7ASJXCSGDG4SLGIXFBGTVGQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: expected piece hash; context canceled; EOF Dec 26 18:25:22.843160578 (506h56m22.714243623s) [noise=true , address=145.79.227.19:4077 , piece_id=EVVLHQGGWORCGY4Z2ZS5MBKWTHJNK7ASJXCSGDG4SLGIXFBGTVGQ, seg_pos={0 0}, upload=444830]: Long tail cancellation due to optimal threshold Dec 26 18:25:22.843279695 (506h56m22.71436274s) [noise=true , address=145.79.227.5:4221 , piece_id=H5GK6DCSW55XEWO4IPOG5RVDXLDDNSBAZ3MEUMJ7E3YNVQIAXOKA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:39562->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:39562->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39562->145.79.227.5:4221: use of closed network connection Dec 26 18:25:22.843293862 (506h56m22.714376907s) [noise=true , address=145.79.227.5:4221 , piece_id=H5GK6DCSW55XEWO4IPOG5RVDXLDDNSBAZ3MEUMJ7E3YNVQIAXOKA, seg_pos={0 0}, upload=444830]: Long tail cancellation due to optimal threshold Dec 26 18:25:22.843317407 (506h56m22.714400452s) [seg_pos={0 0}, upload=444830]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:25:22.843415354 (506h56m22.714498399s) [upload=444830]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:25:22.843422667 (506h56m22.714505712s) [upload=444830]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:25:23.146348141 (506h56m23.017431176s) [upload=444830]: Done uploading segments... Dec 26 18:25:23.146369531 (506h56m23.017452576s) [upload=444830]: Upload finished. err: <nil> Duration: 9.175557062s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXX.....XX.X.XXX.........X..X.....X...XX.........X.....X..X.] duration: 19.483778334s trace: https://jaeger.dev.storj.io/trace/4d484ce3fc14ae0 FAIL (2025-12-26 18:26:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: VKGGWIBE6RX7AFJUBKJ2MSAEABXCUQQG3X36JWTCR6UG73WXVBWQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: CTQ5I3FGJGLOFFSWZC6ZNPYI5RFZBQJTZ2DTDMY4CXTGNCCEZEMQ) context deadline exceeded ======LOGS====== 6 18:25:19.666927494 (506h56m19.538010539s) [upload=444830]: Starting upload Dec 26 18:25:19.667038895 (506h56m19.538121961s) [upload=444830]: Done starting upload Dec 26 18:25:19.667174374 (506h56m19.538257409s) [upload=444830]: Uploading segments... Dec 26 18:25:19.667258674 (506h56m19.538341719s) [upload=444830]: Got next segment. Inline: false Dec 26 18:25:19.667296165 (506h56m19.5383792s) [upload=444830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:25:19.667301856 (506h56m19.538384901s) [upload=444830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:25:19.918625050 (506h56m19.789708105s) [seg_pos={0 0}, upload=444830]: Begin upload segment... Dec 26 18:25:19.918761218 (506h56m19.789844253s) [seg_pos={0 0}, upload=444830]: Done begin upload segment. Dec 26 18:25:19.918834739 (506h56m19.789917784s) [noise=true , address=145.79.227.22:4511 , piece_id=BMFAFUOM3HZAXCDJV2RB3QPI453KK23HPM6FDJJCF3ELECANLVFA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.918928477 (506h56m19.790011522s) [noise=true , address=45.91.116.4:4937 , piece_id=DOM3EPX5HZBZR3VGVEO5WNPI5H7YDOBTNKTEPX5IA5GW5YBKHF2A, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.919612670 (506h56m19.790695705s) [noise=true , address=145.79.227.47:4267 , piece_id=F6AGT52QXC2SOWGU5XOOG4PJK2OCCCFUU3OMLCV5E2FIJ45SATGQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.919839992 (506h56m19.790923037s) [noise=true , address=145.79.227.51:4409 , piece_id=D2G7AM2VNS77FUDFFAZ5IJDI3HGNJYMKQUK4Z5JUBL3VSFRT4ZLQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.919995649 (506h56m19.791078694s) [noise=true , address=77.90.31.18:4808 , piece_id=XOW4J4T67WQW7HUORTKM2MHDYLJKBZYBXOJ5XNYLQEJJTEFLWLPA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920235296 (506h56m19.791318341s) [noise=true , address=45.91.116.5:4300 , piece_id=CTQ5I3FGJGLOFFSWZC6ZNPYI5RFZBQJTZ2DTDMY4CXTGNCCEZEMQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920356225 (506h56m19.79143927s) [noise=true , address=145.79.227.19:4077 , piece_id=EVVLHQGGWORCGY4Z2ZS5MBKWTHJNK7ASJXCSGDG4SLGIXFBGTVGQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920575754 (506h56m19.791658799s) [noise=true , address=145.79.227.49:4339 , piece_id=5BRHLKVAJ2S7LBH5CH5SE3YJN5LSX4IM2IYWZWIUOTF73LMCWPMA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920713106 (506h56m19.791796151s) [noise=true , address=145.79.227.42:4122 , piece_id=ULH3XHAYVXRIQKOTGUTUYIU5M4T2XGQ7UJLKDW4PGPUIDSLQP4IA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920967009 (506h56m19.792050054s) [noise=true , address=145.79.227.16:4909 , piece_id=TUITUPXYBG43YVFXHDW7LKNTIQWHBDXBM3E3D75PPZ5OVGVTC76Q, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921151460 (506h56m19.792234495s) [noise=true , address=145.79.227.3:4151 , piece_id=5H6T6Q3AVDFBA3IE7NNZ7JD5PGSFWLAURSET2HV2EDPKLQWYFHDA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921344848 (506h56m19.792427884s) [noise=true , address=145.79.227.5:4221 , piece_id=H5GK6DCSW55XEWO4IPOG5RVDXLDDNSBAZ3MEUMJ7E3YNVQIAXOKA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921570267 (506h56m19.792653312s) [noise=true , address=145.79.227.21:4474 , piece_id=46DB3FQKA33EDW7GXVXMSDPI3ZEOUR3ADP3WGLRT4X4GXKBN2H7Q, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921714832 (506h56m19.792797877s) [noise=true , address=145.79.227.37:4565 , piece_id=3VJMU5PIWEXDUBP6KGY3JK5J5HYWZ64XZFSZNXL2PPYAE2NWK4OQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.922067303 (506h56m19.793150348s) [noise=true , address=45.91.116.7:4012 , piece_id=VKGGWIBE6RX7AFJUBKJ2MSAEABXCUQQG3X36JWTCR6UG73WXVBWQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.922212541 (506h56m19.793295576s) [noise=true , address=145.79.227.39:4013 , piece_id=CSM5J3CPJ4AYOKHSVLBYIM6T7VGFRGMKBAH7BYEKJWHYLBJ4FMFA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:21.625781415 (506h56m21.49686449s) [upload=444830]: Next returned nil segment Dec 26 18:25:21.625838002 (506h56m21.496921047s) [upload=444830]: Waiting for error group managing segments... Dec 26 18:25:21.919637218 (506h56m21.790720253s) [noise=true , address=145.79.227.51:4409 , piece_id=D2G7AM2VNS77FUDFFAZ5IJDI3HGNJYMKQUK4Z5JUBL3VSFRT4ZLQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.920253681 (506h56m21.791336727s) [noise=true , address=145.79.227.39:4013 , piece_id=CSM5J3CPJ4AYOKHSVLBYIM6T7VGFRGMKBAH7BYEKJWHYLBJ4FMFA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.929953431 (506h56m21.801036466s) [noise=true , address=145.79.227.22:4511 , piece_id=BMFAFUOM3HZAXCDJV2RB3QPI453KK23HPM6FDJJCF3ELECANLVFA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.934488106 (506h56m21.805571141s) [noise=true , address=145.79.227.21:4474 , piece_id=46DB3FQKA33EDW7GXVXMSDPI3ZEOUR3ADP3WGLRT4X4GXKBN2H7Q, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.943745083 (506h56m21.814828128s) [noise=true , address=145.79.227.37:4565 , piece_id=3VJMU5PIWEXDUBP6KGY3JK5J5HYWZ64XZFSZNXL2PPYAE2NWK4OQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.946938384 (506h56m21.818021429s) [noise=true , address=145.79.227.42:4122 , piece_id=ULH3XHAYVXRIQKOTGUTUYIU5M4T2XGQ7UJLKDW4PGPUIDSLQP4IA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.950590989 (506h56m21.821674024s) [noise=true , address=145.79.227.16:4909 , piece_id=TUITUPXYBG43YVFXHDW7LKNTIQWHBDXBM3E3D75PPZ5OVGVTC76Q, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.952817939 (506h56m21.823900974s) [noise=true , address=145.79.227.49:4339 , piece_id=5BRHLKVAJ2S7LBH5CH5SE3YJN5LSX4IM2IYWZWIUOTF73LMCWPMA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.954184631 (506h56m21.825267677s) [noise=true , address=145.79.227.47:4267 , piece_id=F6AGT52QXC2SOWGU5XOOG4PJK2OCCCFUU3OMLCV5E2FIJ45SATGQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.092833324 (506h56m21.963916359s) [noise=true , address=77.90.31.18:4808 , piece_id=XOW4J4T67WQW7HUORTKM2MHDYLJKBZYBXOJ5XNYLQEJJTEFLWLPA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.291735528 (506h56m22.162818583s) [noise=true , address=45.91.116.4:4937 , piece_id=DOM3EPX5HZBZR3VGVEO5WNPI5H7YDOBTNKTEPX5IA5GW5YBKHF2A, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.364544489 (506h56m22.235627534s) [noise=true , address=45.91.116.7:4012 , piece_id=VKGGWIBE6RX7AFJUBKJ2MSAEABXCUQQG3X36JWTCR6UG73WXVBWQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.842766488 (506h56m22.713849533s) [noise=true , address=45.91.116.5:4300 , piece_id=CTQ5I3FGJGLOFFSWZC6ZNPYI5RFZBQJTZ2DTDMY4CXTGNCCEZEMQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.842794591 (506h56m22.713877636s) [seg_pos={0 0}, upload=444830]: Segment reached optimal threshold of 13 pieces. Dec 26 18:25:22.843025661 (506h56m22.714108706s) [noise=true , address=145.79.227.3:4151 , piece_id=5H6T6Q3AVDFBA3IE7NNZ7JD5PGSFWLAURSET2HV2EDPKLQWYFHDA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Dec 26 18:25:22.843036923 (506h56m22.714119968s) [noise=true , address=145.79.227.3:4151 , piece_id=5H6T6Q3AVDFBA3IE7NNZ7JD5PGSFWLAURSET2HV2EDPKLQWYFHDA, seg_pos={0 0}, upload=444830]: Long tail cancellation due to optimal threshold Dec 26 18:25:22.843137895 (506h56m22.71422094s) [noise=true , address=145.79.227.19:4077 , piece_id=EVVLHQGGWORCGY4Z2ZS5MBKWTHJNK7ASJXCSGDG4SLGIXFBGTVGQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: expected piece hash; context canceled; EOF Dec 26 18:25:22.843160578 (506h56m22.714243623s) [noise=true , address=145.79.227.19:4077 , piece_id=EVVLHQGGWORCGY4Z2ZS5MBKWTHJNK7ASJXCSGDG4SLGIXFBGTVGQ, seg_pos={0 0}, upload=444830]: Long tail cancellation due to optimal threshold Dec 26 18:25:22.843279695 (506h56m22.71436274s) [noise=true , address=145.79.227.5:4221 , piece_id=H5GK6DCSW55XEWO4IPOG5RVDXLDDNSBAZ3MEUMJ7E3YNVQIAXOKA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:39562->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:39562->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39562->145.79.227.5:4221: use of closed network connection Dec 26 18:25:22.843293862 (506h56m22.714376907s) [noise=true , address=145.79.227.5:4221 , piece_id=H5GK6DCSW55XEWO4IPOG5RVDXLDDNSBAZ3MEUMJ7E3YNVQIAXOKA, seg_pos={0 0}, upload=444830]: Long tail cancellation due to optimal threshold Dec 26 18:25:22.843317407 (506h56m22.714400452s) [seg_pos={0 0}, upload=444830]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:25:22.843415354 (506h56m22.714498399s) [upload=444830]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:25:22.843422667 (506h56m22.714505712s) [upload=444830]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:25:23.146348141 (506h56m23.017431176s) [upload=444830]: Done uploading segments... Dec 26 18:25:23.146369531 (506h56m23.017452576s) [upload=444830]: Upload finished. err: <nil> Duration: 19.483872503s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XXX.....XX.X.XXX.........X..X.....X...XX.........X.....X..X] duration: 30.004232502s trace: https://jaeger.dev.storj.io/trace/4d484ce3fc14ae0 FAIL (2025-12-26 18:25:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: VKGGWIBE6RX7AFJUBKJ2MSAEABXCUQQG3X36JWTCR6UG73WXVBWQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: CTQ5I3FGJGLOFFSWZC6ZNPYI5RFZBQJTZ2DTDMY4CXTGNCCEZEMQ) context deadline exceeded ======LOGS====== 6 18:25:19.666927494 (506h56m19.538010539s) [upload=444830]: Starting upload Dec 26 18:25:19.667038895 (506h56m19.538121961s) [upload=444830]: Done starting upload Dec 26 18:25:19.667174374 (506h56m19.538257409s) [upload=444830]: Uploading segments... Dec 26 18:25:19.667258674 (506h56m19.538341719s) [upload=444830]: Got next segment. Inline: false Dec 26 18:25:19.667296165 (506h56m19.5383792s) [upload=444830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:25:19.667301856 (506h56m19.538384901s) [upload=444830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:25:19.918625050 (506h56m19.789708105s) [seg_pos={0 0}, upload=444830]: Begin upload segment... Dec 26 18:25:19.918761218 (506h56m19.789844253s) [seg_pos={0 0}, upload=444830]: Done begin upload segment. Dec 26 18:25:19.918834739 (506h56m19.789917784s) [noise=true , address=145.79.227.22:4511 , piece_id=BMFAFUOM3HZAXCDJV2RB3QPI453KK23HPM6FDJJCF3ELECANLVFA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.918928477 (506h56m19.790011522s) [noise=true , address=45.91.116.4:4937 , piece_id=DOM3EPX5HZBZR3VGVEO5WNPI5H7YDOBTNKTEPX5IA5GW5YBKHF2A, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.919612670 (506h56m19.790695705s) [noise=true , address=145.79.227.47:4267 , piece_id=F6AGT52QXC2SOWGU5XOOG4PJK2OCCCFUU3OMLCV5E2FIJ45SATGQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.919839992 (506h56m19.790923037s) [noise=true , address=145.79.227.51:4409 , piece_id=D2G7AM2VNS77FUDFFAZ5IJDI3HGNJYMKQUK4Z5JUBL3VSFRT4ZLQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.919995649 (506h56m19.791078694s) [noise=true , address=77.90.31.18:4808 , piece_id=XOW4J4T67WQW7HUORTKM2MHDYLJKBZYBXOJ5XNYLQEJJTEFLWLPA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920235296 (506h56m19.791318341s) [noise=true , address=45.91.116.5:4300 , piece_id=CTQ5I3FGJGLOFFSWZC6ZNPYI5RFZBQJTZ2DTDMY4CXTGNCCEZEMQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920356225 (506h56m19.79143927s) [noise=true , address=145.79.227.19:4077 , piece_id=EVVLHQGGWORCGY4Z2ZS5MBKWTHJNK7ASJXCSGDG4SLGIXFBGTVGQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920575754 (506h56m19.791658799s) [noise=true , address=145.79.227.49:4339 , piece_id=5BRHLKVAJ2S7LBH5CH5SE3YJN5LSX4IM2IYWZWIUOTF73LMCWPMA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920713106 (506h56m19.791796151s) [noise=true , address=145.79.227.42:4122 , piece_id=ULH3XHAYVXRIQKOTGUTUYIU5M4T2XGQ7UJLKDW4PGPUIDSLQP4IA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920967009 (506h56m19.792050054s) [noise=true , address=145.79.227.16:4909 , piece_id=TUITUPXYBG43YVFXHDW7LKNTIQWHBDXBM3E3D75PPZ5OVGVTC76Q, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921151460 (506h56m19.792234495s) [noise=true , address=145.79.227.3:4151 , piece_id=5H6T6Q3AVDFBA3IE7NNZ7JD5PGSFWLAURSET2HV2EDPKLQWYFHDA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921344848 (506h56m19.792427884s) [noise=true , address=145.79.227.5:4221 , piece_id=H5GK6DCSW55XEWO4IPOG5RVDXLDDNSBAZ3MEUMJ7E3YNVQIAXOKA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921570267 (506h56m19.792653312s) [noise=true , address=145.79.227.21:4474 , piece_id=46DB3FQKA33EDW7GXVXMSDPI3ZEOUR3ADP3WGLRT4X4GXKBN2H7Q, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921714832 (506h56m19.792797877s) [noise=true , address=145.79.227.37:4565 , piece_id=3VJMU5PIWEXDUBP6KGY3JK5J5HYWZ64XZFSZNXL2PPYAE2NWK4OQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.922067303 (506h56m19.793150348s) [noise=true , address=45.91.116.7:4012 , piece_id=VKGGWIBE6RX7AFJUBKJ2MSAEABXCUQQG3X36JWTCR6UG73WXVBWQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.922212541 (506h56m19.793295576s) [noise=true , address=145.79.227.39:4013 , piece_id=CSM5J3CPJ4AYOKHSVLBYIM6T7VGFRGMKBAH7BYEKJWHYLBJ4FMFA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:21.625781415 (506h56m21.49686449s) [upload=444830]: Next returned nil segment Dec 26 18:25:21.625838002 (506h56m21.496921047s) [upload=444830]: Waiting for error group managing segments... Dec 26 18:25:21.919637218 (506h56m21.790720253s) [noise=true , address=145.79.227.51:4409 , piece_id=D2G7AM2VNS77FUDFFAZ5IJDI3HGNJYMKQUK4Z5JUBL3VSFRT4ZLQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.920253681 (506h56m21.791336727s) [noise=true , address=145.79.227.39:4013 , piece_id=CSM5J3CPJ4AYOKHSVLBYIM6T7VGFRGMKBAH7BYEKJWHYLBJ4FMFA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.929953431 (506h56m21.801036466s) [noise=true , address=145.79.227.22:4511 , piece_id=BMFAFUOM3HZAXCDJV2RB3QPI453KK23HPM6FDJJCF3ELECANLVFA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.934488106 (506h56m21.805571141s) [noise=true , address=145.79.227.21:4474 , piece_id=46DB3FQKA33EDW7GXVXMSDPI3ZEOUR3ADP3WGLRT4X4GXKBN2H7Q, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.943745083 (506h56m21.814828128s) [noise=true , address=145.79.227.37:4565 , piece_id=3VJMU5PIWEXDUBP6KGY3JK5J5HYWZ64XZFSZNXL2PPYAE2NWK4OQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.946938384 (506h56m21.818021429s) [noise=true , address=145.79.227.42:4122 , piece_id=ULH3XHAYVXRIQKOTGUTUYIU5M4T2XGQ7UJLKDW4PGPUIDSLQP4IA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.950590989 (506h56m21.821674024s) [noise=true , address=145.79.227.16:4909 , piece_id=TUITUPXYBG43YVFXHDW7LKNTIQWHBDXBM3E3D75PPZ5OVGVTC76Q, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.952817939 (506h56m21.823900974s) [noise=true , address=145.79.227.49:4339 , piece_id=5BRHLKVAJ2S7LBH5CH5SE3YJN5LSX4IM2IYWZWIUOTF73LMCWPMA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.954184631 (506h56m21.825267677s) [noise=true , address=145.79.227.47:4267 , piece_id=F6AGT52QXC2SOWGU5XOOG4PJK2OCCCFUU3OMLCV5E2FIJ45SATGQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.092833324 (506h56m21.963916359s) [noise=true , address=77.90.31.18:4808 , piece_id=XOW4J4T67WQW7HUORTKM2MHDYLJKBZYBXOJ5XNYLQEJJTEFLWLPA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.291735528 (506h56m22.162818583s) [noise=true , address=45.91.116.4:4937 , piece_id=DOM3EPX5HZBZR3VGVEO5WNPI5H7YDOBTNKTEPX5IA5GW5YBKHF2A, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.364544489 (506h56m22.235627534s) [noise=true , address=45.91.116.7:4012 , piece_id=VKGGWIBE6RX7AFJUBKJ2MSAEABXCUQQG3X36JWTCR6UG73WXVBWQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.842766488 (506h56m22.713849533s) [noise=true , address=45.91.116.5:4300 , piece_id=CTQ5I3FGJGLOFFSWZC6ZNPYI5RFZBQJTZ2DTDMY4CXTGNCCEZEMQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.842794591 (506h56m22.713877636s) [seg_pos={0 0}, upload=444830]: Segment reached optimal threshold of 13 pieces. Dec 26 18:25:22.843025661 (506h56m22.714108706s) [noise=true , address=145.79.227.3:4151 , piece_id=5H6T6Q3AVDFBA3IE7NNZ7JD5PGSFWLAURSET2HV2EDPKLQWYFHDA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Dec 26 18:25:22.843036923 (506h56m22.714119968s) [noise=true , address=145.79.227.3:4151 , piece_id=5H6T6Q3AVDFBA3IE7NNZ7JD5PGSFWLAURSET2HV2EDPKLQWYFHDA, seg_pos={0 0}, upload=444830]: Long tail cancellation due to optimal threshold Dec 26 18:25:22.843137895 (506h56m22.71422094s) [noise=true , address=145.79.227.19:4077 , piece_id=EVVLHQGGWORCGY4Z2ZS5MBKWTHJNK7ASJXCSGDG4SLGIXFBGTVGQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: expected piece hash; context canceled; EOF Dec 26 18:25:22.843160578 (506h56m22.714243623s) [noise=true , address=145.79.227.19:4077 , piece_id=EVVLHQGGWORCGY4Z2ZS5MBKWTHJNK7ASJXCSGDG4SLGIXFBGTVGQ, seg_pos={0 0}, upload=444830]: Long tail cancellation due to optimal threshold Dec 26 18:25:22.843279695 (506h56m22.71436274s) [noise=true , address=145.79.227.5:4221 , piece_id=H5GK6DCSW55XEWO4IPOG5RVDXLDDNSBAZ3MEUMJ7E3YNVQIAXOKA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:39562->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:39562->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39562->145.79.227.5:4221: use of closed network connection Dec 26 18:25:22.843293862 (506h56m22.714376907s) [noise=true , address=145.79.227.5:4221 , piece_id=H5GK6DCSW55XEWO4IPOG5RVDXLDDNSBAZ3MEUMJ7E3YNVQIAXOKA, seg_pos={0 0}, upload=444830]: Long tail cancellation due to optimal threshold Dec 26 18:25:22.843317407 (506h56m22.714400452s) [seg_pos={0 0}, upload=444830]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:25:22.843415354 (506h56m22.714498399s) [upload=444830]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:25:22.843422667 (506h56m22.714505712s) [upload=444830]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:25:23.146348141 (506h56m23.017431176s) [upload=444830]: Done uploading segments... Dec 26 18:25:23.146369531 (506h56m23.017452576s) [upload=444830]: Upload finished. err: <nil> Duration: 30.004256046s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.807805469s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:25:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 5.807834505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 34.948449578s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:25:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 34.94848826s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.XXX.....XX.X.XXX.........X..X.....X...XX.........X.....X..] duration: 6.524947205s trace: https://jaeger.dev.storj.io/trace/46a03f883651bcf3 FAIL (2025-12-26 18:24:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: NJC5EMFER5YYISOPZU6UARD6PUPZAHMUAF3TNB4LJPHXA4QJJO5A) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: HWLERTDU6GCEN6IQLRMIXR2TWJHJUFUFAENXZSFNDP4GYMOSQOGQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: FHQYVM44UEO4RUTAHOFIQ7LO3WYXAJGW23DSSWA6JRHJZ3RU6HHQ) context deadline exceeded ======LOGS====== 18:22:20.052924283 (506h53m19.924007328s) [upload=444786]: Starting upload Dec 26 18:22:20.053003855 (506h53m19.92408692s) [upload=444786]: Done starting upload Dec 26 18:22:20.053185570 (506h53m19.924268615s) [upload=444786]: Uploading segments... Dec 26 18:22:20.053283848 (506h53m19.924366983s) [upload=444786]: Got next segment. Inline: false Dec 26 18:22:20.053328452 (506h53m19.924411537s) [upload=444786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:22:20.053338251 (506h53m19.924421296s) [upload=444786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:22:20.300773130 (506h53m20.171856186s) [seg_pos={0 0}, upload=444786]: Begin upload segment... Dec 26 18:22:20.300829749 (506h53m20.171912794s) [seg_pos={0 0}, upload=444786]: Done begin upload segment. Dec 26 18:22:20.300860326 (506h53m20.171943361s) [noise=true , address=45.91.116.13:4155 , piece_id=7Z5CNXM7NK4H4KBR5WXMEVYKVCBBFYJEEVAUDC7NPRBLKMM4PT6Q, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.301243986 (506h53m20.172327052s) [noise=true , address=145.79.227.42:4123 , piece_id=DYVDISG4ZZSGO7TPYQCFM4ZSLCFUR7F35EFWJBLOF47EIKAPVTMQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.302210267 (506h53m20.173293302s) [noise=true , address=145.79.227.51:4410 , piece_id=54EBW2MUNATUOVLSISA7FNI7DKUOEDSN2MNOEUTQHDDQRHBE54XQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.302538793 (506h53m20.173621838s) [noise=true , address=145.79.227.45:4230 , piece_id=3F5NRWI4RRVB7GJBHW3HH34G7MAJBGQ47OWNPHKNDXZYBNYE6MWA, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.302911883 (506h53m20.173994918s) [noise=true , address=45.91.116.4:4937 , piece_id=OILDTM7QBOICW4GLPOMPFTOQGXN7KVPJER3RGT7FUSFHDMQMGUBQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.303243464 (506h53m20.174326499s) [noise=true , address=77.90.31.13:4629 , piece_id=HWLERTDU6GCEN6IQLRMIXR2TWJHJUFUFAENXZSFNDP4GYMOSQOGQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.303608820 (506h53m20.174691855s) [noise=true , address=77.90.31.16:4736 , piece_id=FHQYVM44UEO4RUTAHOFIQ7LO3WYXAJGW23DSSWA6JRHJZ3RU6HHQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.303906076 (506h53m20.174989121s) [noise=true , address=145.79.227.49:4339 , piece_id=EGL3XFJKGTELBWBI5DSKC7JWESOLRM2XMO6LRS4WCB6IJZWZ6CUQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.304052885 (506h53m20.17513592s) [noise=true , address=45.91.116.11:4118 , piece_id=G2O63TBJ7PM2AP6QPIVIIWSMR2EBFOOWJBQTMIB2GECGQUDCGVIQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.304543570 (506h53m20.175626605s) [noise=true , address=45.91.116.7:4012 , piece_id=NJC5EMFER5YYISOPZU6UARD6PUPZAHMUAF3TNB4LJPHXA4QJJO5A, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.305253973 (506h53m20.176337018s) [noise=true , address=145.79.227.6:4257 , piece_id=Q4EUKU62POICD3YVHHTR722DWDNOONOBG3LHMWGTISESZYWWF47Q, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.305734778 (506h53m20.176817833s) [noise=true , address=145.79.227.48:4303 , piece_id=EKVOCHB53K4HIN2JFZGRZPCJJUVIAC6BYKXPMEKJCTTNL7YSJDVA, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.306250310 (506h53m20.177333355s) [noise=true , address=145.79.227.46:4602 , piece_id=HH6MVNSJQCFI4TGAGLNCJS37LS7RB37KP5NDL4JAJJNKBIR7JTTQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.306729412 (506h53m20.177812447s) [noise=true , address=145.79.227.40:4049 , piece_id=A54YZBC5INMC4AIGFW45TDVGXO554WBTIXR7POY7H2SHZXV2LKPQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.307243411 (506h53m20.178326456s) [noise=true , address=145.79.227.8:4330 , piece_id=VUBUBFR73QD4K47IDNI24B6T3KKOUV7VIHO4QI3JFYFOYTHZ76BA, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.308077850 (506h53m20.179160895s) [noise=true , address=77.90.31.10:4520 , piece_id=SDDHFO7ZCRH4B5PLWEXHM7ZCBLX2V5GJLFLNLO3UZPTBKRW4FU7Q, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:22.021189354 (506h53m21.892272399s) [upload=444786]: Next returned nil segment Dec 26 18:22:22.021203520 (506h53m21.892286566s) [upload=444786]: Waiting for error group managing segments... Dec 26 18:22:22.306790919 (506h53m22.177873964s) [noise=true , address=145.79.227.42:4123 , piece_id=DYVDISG4ZZSGO7TPYQCFM4ZSLCFUR7F35EFWJBLOF47EIKAPVTMQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.317440578 (506h53m22.188523623s) [noise=true , address=145.79.227.48:4303 , piece_id=EKVOCHB53K4HIN2JFZGRZPCJJUVIAC6BYKXPMEKJCTTNL7YSJDVA, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.324498949 (506h53m22.195581994s) [noise=true , address=145.79.227.46:4602 , piece_id=HH6MVNSJQCFI4TGAGLNCJS37LS7RB37KP5NDL4JAJJNKBIR7JTTQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.332376839 (506h53m22.203459874s) [noise=true , address=145.79.227.45:4230 , piece_id=3F5NRWI4RRVB7GJBHW3HH34G7MAJBGQ47OWNPHKNDXZYBNYE6MWA, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.335008621 (506h53m22.206091666s) [noise=true , address=145.79.227.8:4330 , piece_id=VUBUBFR73QD4K47IDNI24B6T3KKOUV7VIHO4QI3JFYFOYTHZ76BA, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.337891891 (506h53m22.208974926s) [noise=true , address=145.79.227.51:4410 , piece_id=54EBW2MUNATUOVLSISA7FNI7DKUOEDSN2MNOEUTQHDDQRHBE54XQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.338036998 (506h53m22.209120033s) [noise=true , address=145.79.227.49:4339 , piece_id=EGL3XFJKGTELBWBI5DSKC7JWESOLRM2XMO6LRS4WCB6IJZWZ6CUQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.357245244 (506h53m22.228328289s) [noise=true , address=145.79.227.6:4257 , piece_id=Q4EUKU62POICD3YVHHTR722DWDNOONOBG3LHMWGTISESZYWWF47Q, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.491011368 (506h53m22.362094403s) [noise=true , address=77.90.31.16:4736 , piece_id=FHQYVM44UEO4RUTAHOFIQ7LO3WYXAJGW23DSSWA6JRHJZ3RU6HHQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.492463102 (506h53m22.363546137s) [noise=true , address=77.90.31.10:4520 , piece_id=SDDHFO7ZCRH4B5PLWEXHM7ZCBLX2V5GJLFLNLO3UZPTBKRW4FU7Q, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.543901169 (506h53m22.414984204s) [noise=true , address=45.91.116.4:4937 , piece_id=OILDTM7QBOICW4GLPOMPFTOQGXN7KVPJER3RGT7FUSFHDMQMGUBQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.555202418 (506h53m22.426285454s) [noise=true , address=45.91.116.7:4012 , piece_id=NJC5EMFER5YYISOPZU6UARD6PUPZAHMUAF3TNB4LJPHXA4QJJO5A, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.584924595 (506h53m22.45600763s) [noise=true , address=77.90.31.13:4629 , piece_id=HWLERTDU6GCEN6IQLRMIXR2TWJHJUFUFAENXZSFNDP4GYMOSQOGQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.584944242 (506h53m22.456027277s) [seg_pos={0 0}, upload=444786]: Segment reached optimal threshold of 13 pieces. Dec 26 18:22:22.585279571 (506h53m22.456362616s) [noise=true , address=145.79.227.40:4049 , piece_id=A54YZBC5INMC4AIGFW45TDVGXO554WBTIXR7POY7H2SHZXV2LKPQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 18:22:22.585291072 (506h53m22.456374107s) [noise=true , address=145.79.227.40:4049 , piece_id=A54YZBC5INMC4AIGFW45TDVGXO554WBTIXR7POY7H2SHZXV2LKPQ, seg_pos={0 0}, upload=444786]: Long tail cancellation due to optimal threshold Dec 26 18:22:22.585390301 (506h53m22.456473366s) [noise=true , address=45.91.116.11:4118 , piece_id=G2O63TBJ7PM2AP6QPIVIIWSMR2EBFOOWJBQTMIB2GECGQUDCGVIQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:54922->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:54922->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54922->45.91.116.11:4118: use of closed network connection Dec 26 18:22:22.585420619 (506h53m22.456503674s) [noise=true , address=45.91.116.11:4118 , piece_id=G2O63TBJ7PM2AP6QPIVIIWSMR2EBFOOWJBQTMIB2GECGQUDCGVIQ, seg_pos={0 0}, upload=444786]: Long tail cancellation due to optimal threshold Dec 26 18:22:22.585629497 (506h53m22.456712552s) [noise=true , address=45.91.116.13:4155 , piece_id=7Z5CNXM7NK4H4KBR5WXMEVYKVCBBFYJEEVAUDC7NPRBLKMM4PT6Q, seg_pos={0 0}, upload=444786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 26 18:22:22.585657961 (506h53m22.456741006s) [noise=true , address=45.91.116.13:4155 , piece_id=7Z5CNXM7NK4H4KBR5WXMEVYKVCBBFYJEEVAUDC7NPRBLKMM4PT6Q, seg_pos={0 0}, upload=444786]: Long tail cancellation due to optimal threshold Dec 26 18:22:22.585679984 (506h53m22.456763029s) [seg_pos={0 0}, upload=444786]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:22:22.585785955 (506h53m22.45686899s) [upload=444786]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:22:22.585792888 (506h53m22.456875923s) [upload=444786]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:22:22.894574533 (506h53m22.765657578s) [upload=444786]: Done uploading segments... Dec 26 18:22:22.894598457 (506h53m22.765681492s) [upload=444786]: Upload finished. err: <nil> Duration: 6.524976601s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 20.190971592s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:24:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 20.191000588s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 37.415631616s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:23:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 37.415661073s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X.XXX.....XX.X.XXX.........X..X.....X...XX.........X.....X.] duration: 6.213801429s trace: https://jaeger.dev.storj.io/trace/46a03f883651bcf3 FAIL (2025-12-26 18:23:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: NJC5EMFER5YYISOPZU6UARD6PUPZAHMUAF3TNB4LJPHXA4QJJO5A) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: HWLERTDU6GCEN6IQLRMIXR2TWJHJUFUFAENXZSFNDP4GYMOSQOGQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: FHQYVM44UEO4RUTAHOFIQ7LO3WYXAJGW23DSSWA6JRHJZ3RU6HHQ) context deadline exceeded ======LOGS====== 18:22:20.052924283 (506h53m19.924007328s) [upload=444786]: Starting upload Dec 26 18:22:20.053003855 (506h53m19.92408692s) [upload=444786]: Done starting upload Dec 26 18:22:20.053185570 (506h53m19.924268615s) [upload=444786]: Uploading segments... Dec 26 18:22:20.053283848 (506h53m19.924366983s) [upload=444786]: Got next segment. Inline: false Dec 26 18:22:20.053328452 (506h53m19.924411537s) [upload=444786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:22:20.053338251 (506h53m19.924421296s) [upload=444786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:22:20.300773130 (506h53m20.171856186s) [seg_pos={0 0}, upload=444786]: Begin upload segment... Dec 26 18:22:20.300829749 (506h53m20.171912794s) [seg_pos={0 0}, upload=444786]: Done begin upload segment. Dec 26 18:22:20.300860326 (506h53m20.171943361s) [noise=true , address=45.91.116.13:4155 , piece_id=7Z5CNXM7NK4H4KBR5WXMEVYKVCBBFYJEEVAUDC7NPRBLKMM4PT6Q, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.301243986 (506h53m20.172327052s) [noise=true , address=145.79.227.42:4123 , piece_id=DYVDISG4ZZSGO7TPYQCFM4ZSLCFUR7F35EFWJBLOF47EIKAPVTMQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.302210267 (506h53m20.173293302s) [noise=true , address=145.79.227.51:4410 , piece_id=54EBW2MUNATUOVLSISA7FNI7DKUOEDSN2MNOEUTQHDDQRHBE54XQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.302538793 (506h53m20.173621838s) [noise=true , address=145.79.227.45:4230 , piece_id=3F5NRWI4RRVB7GJBHW3HH34G7MAJBGQ47OWNPHKNDXZYBNYE6MWA, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.302911883 (506h53m20.173994918s) [noise=true , address=45.91.116.4:4937 , piece_id=OILDTM7QBOICW4GLPOMPFTOQGXN7KVPJER3RGT7FUSFHDMQMGUBQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.303243464 (506h53m20.174326499s) [noise=true , address=77.90.31.13:4629 , piece_id=HWLERTDU6GCEN6IQLRMIXR2TWJHJUFUFAENXZSFNDP4GYMOSQOGQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.303608820 (506h53m20.174691855s) [noise=true , address=77.90.31.16:4736 , piece_id=FHQYVM44UEO4RUTAHOFIQ7LO3WYXAJGW23DSSWA6JRHJZ3RU6HHQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.303906076 (506h53m20.174989121s) [noise=true , address=145.79.227.49:4339 , piece_id=EGL3XFJKGTELBWBI5DSKC7JWESOLRM2XMO6LRS4WCB6IJZWZ6CUQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.304052885 (506h53m20.17513592s) [noise=true , address=45.91.116.11:4118 , piece_id=G2O63TBJ7PM2AP6QPIVIIWSMR2EBFOOWJBQTMIB2GECGQUDCGVIQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.304543570 (506h53m20.175626605s) [noise=true , address=45.91.116.7:4012 , piece_id=NJC5EMFER5YYISOPZU6UARD6PUPZAHMUAF3TNB4LJPHXA4QJJO5A, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.305253973 (506h53m20.176337018s) [noise=true , address=145.79.227.6:4257 , piece_id=Q4EUKU62POICD3YVHHTR722DWDNOONOBG3LHMWGTISESZYWWF47Q, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.305734778 (506h53m20.176817833s) [noise=true , address=145.79.227.48:4303 , piece_id=EKVOCHB53K4HIN2JFZGRZPCJJUVIAC6BYKXPMEKJCTTNL7YSJDVA, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.306250310 (506h53m20.177333355s) [noise=true , address=145.79.227.46:4602 , piece_id=HH6MVNSJQCFI4TGAGLNCJS37LS7RB37KP5NDL4JAJJNKBIR7JTTQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.306729412 (506h53m20.177812447s) [noise=true , address=145.79.227.40:4049 , piece_id=A54YZBC5INMC4AIGFW45TDVGXO554WBTIXR7POY7H2SHZXV2LKPQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.307243411 (506h53m20.178326456s) [noise=true , address=145.79.227.8:4330 , piece_id=VUBUBFR73QD4K47IDNI24B6T3KKOUV7VIHO4QI3JFYFOYTHZ76BA, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.308077850 (506h53m20.179160895s) [noise=true , address=77.90.31.10:4520 , piece_id=SDDHFO7ZCRH4B5PLWEXHM7ZCBLX2V5GJLFLNLO3UZPTBKRW4FU7Q, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:22.021189354 (506h53m21.892272399s) [upload=444786]: Next returned nil segment Dec 26 18:22:22.021203520 (506h53m21.892286566s) [upload=444786]: Waiting for error group managing segments... Dec 26 18:22:22.306790919 (506h53m22.177873964s) [noise=true , address=145.79.227.42:4123 , piece_id=DYVDISG4ZZSGO7TPYQCFM4ZSLCFUR7F35EFWJBLOF47EIKAPVTMQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.317440578 (506h53m22.188523623s) [noise=true , address=145.79.227.48:4303 , piece_id=EKVOCHB53K4HIN2JFZGRZPCJJUVIAC6BYKXPMEKJCTTNL7YSJDVA, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.324498949 (506h53m22.195581994s) [noise=true , address=145.79.227.46:4602 , piece_id=HH6MVNSJQCFI4TGAGLNCJS37LS7RB37KP5NDL4JAJJNKBIR7JTTQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.332376839 (506h53m22.203459874s) [noise=true , address=145.79.227.45:4230 , piece_id=3F5NRWI4RRVB7GJBHW3HH34G7MAJBGQ47OWNPHKNDXZYBNYE6MWA, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.335008621 (506h53m22.206091666s) [noise=true , address=145.79.227.8:4330 , piece_id=VUBUBFR73QD4K47IDNI24B6T3KKOUV7VIHO4QI3JFYFOYTHZ76BA, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.337891891 (506h53m22.208974926s) [noise=true , address=145.79.227.51:4410 , piece_id=54EBW2MUNATUOVLSISA7FNI7DKUOEDSN2MNOEUTQHDDQRHBE54XQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.338036998 (506h53m22.209120033s) [noise=true , address=145.79.227.49:4339 , piece_id=EGL3XFJKGTELBWBI5DSKC7JWESOLRM2XMO6LRS4WCB6IJZWZ6CUQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.357245244 (506h53m22.228328289s) [noise=true , address=145.79.227.6:4257 , piece_id=Q4EUKU62POICD3YVHHTR722DWDNOONOBG3LHMWGTISESZYWWF47Q, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.491011368 (506h53m22.362094403s) [noise=true , address=77.90.31.16:4736 , piece_id=FHQYVM44UEO4RUTAHOFIQ7LO3WYXAJGW23DSSWA6JRHJZ3RU6HHQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.492463102 (506h53m22.363546137s) [noise=true , address=77.90.31.10:4520 , piece_id=SDDHFO7ZCRH4B5PLWEXHM7ZCBLX2V5GJLFLNLO3UZPTBKRW4FU7Q, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.543901169 (506h53m22.414984204s) [noise=true , address=45.91.116.4:4937 , piece_id=OILDTM7QBOICW4GLPOMPFTOQGXN7KVPJER3RGT7FUSFHDMQMGUBQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.555202418 (506h53m22.426285454s) [noise=true , address=45.91.116.7:4012 , piece_id=NJC5EMFER5YYISOPZU6UARD6PUPZAHMUAF3TNB4LJPHXA4QJJO5A, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.584924595 (506h53m22.45600763s) [noise=true , address=77.90.31.13:4629 , piece_id=HWLERTDU6GCEN6IQLRMIXR2TWJHJUFUFAENXZSFNDP4GYMOSQOGQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.584944242 (506h53m22.456027277s) [seg_pos={0 0}, upload=444786]: Segment reached optimal threshold of 13 pieces. Dec 26 18:22:22.585279571 (506h53m22.456362616s) [noise=true , address=145.79.227.40:4049 , piece_id=A54YZBC5INMC4AIGFW45TDVGXO554WBTIXR7POY7H2SHZXV2LKPQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 18:22:22.585291072 (506h53m22.456374107s) [noise=true , address=145.79.227.40:4049 , piece_id=A54YZBC5INMC4AIGFW45TDVGXO554WBTIXR7POY7H2SHZXV2LKPQ, seg_pos={0 0}, upload=444786]: Long tail cancellation due to optimal threshold Dec 26 18:22:22.585390301 (506h53m22.456473366s) [noise=true , address=45.91.116.11:4118 , piece_id=G2O63TBJ7PM2AP6QPIVIIWSMR2EBFOOWJBQTMIB2GECGQUDCGVIQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:54922->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:54922->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54922->45.91.116.11:4118: use of closed network connection Dec 26 18:22:22.585420619 (506h53m22.456503674s) [noise=true , address=45.91.116.11:4118 , piece_id=G2O63TBJ7PM2AP6QPIVIIWSMR2EBFOOWJBQTMIB2GECGQUDCGVIQ, seg_pos={0 0}, upload=444786]: Long tail cancellation due to optimal threshold Dec 26 18:22:22.585629497 (506h53m22.456712552s) [noise=true , address=45.91.116.13:4155 , piece_id=7Z5CNXM7NK4H4KBR5WXMEVYKVCBBFYJEEVAUDC7NPRBLKMM4PT6Q, seg_pos={0 0}, upload=444786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 26 18:22:22.585657961 (506h53m22.456741006s) [noise=true , address=45.91.116.13:4155 , piece_id=7Z5CNXM7NK4H4KBR5WXMEVYKVCBBFYJEEVAUDC7NPRBLKMM4PT6Q, seg_pos={0 0}, upload=444786]: Long tail cancellation due to optimal threshold Dec 26 18:22:22.585679984 (506h53m22.456763029s) [seg_pos={0 0}, upload=444786]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:22:22.585785955 (506h53m22.45686899s) [upload=444786]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:22:22.585792888 (506h53m22.456875923s) [upload=444786]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:22:22.894574533 (506h53m22.765657578s) [upload=444786]: Done uploading segments... Dec 26 18:22:22.894598457 (506h53m22.765681492s) [upload=444786]: Upload finished. err: <nil> Duration: 6.21382876s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 48.093075117s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:23:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 48.09310374s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X.XXX.....XX.X.XXX.........X..X.....X...XX.........X.....X] duration: 30.003734751s trace: https://jaeger.dev.storj.io/trace/46a03f883651bcf3 FAIL (2025-12-26 18:22:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: NJC5EMFER5YYISOPZU6UARD6PUPZAHMUAF3TNB4LJPHXA4QJJO5A) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: HWLERTDU6GCEN6IQLRMIXR2TWJHJUFUFAENXZSFNDP4GYMOSQOGQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: FHQYVM44UEO4RUTAHOFIQ7LO3WYXAJGW23DSSWA6JRHJZ3RU6HHQ) context deadline exceeded ======LOGS====== 18:22:20.052924283 (506h53m19.924007328s) [upload=444786]: Starting upload Dec 26 18:22:20.053003855 (506h53m19.92408692s) [upload=444786]: Done starting upload Dec 26 18:22:20.053185570 (506h53m19.924268615s) [upload=444786]: Uploading segments... Dec 26 18:22:20.053283848 (506h53m19.924366983s) [upload=444786]: Got next segment. Inline: false Dec 26 18:22:20.053328452 (506h53m19.924411537s) [upload=444786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:22:20.053338251 (506h53m19.924421296s) [upload=444786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:22:20.300773130 (506h53m20.171856186s) [seg_pos={0 0}, upload=444786]: Begin upload segment... Dec 26 18:22:20.300829749 (506h53m20.171912794s) [seg_pos={0 0}, upload=444786]: Done begin upload segment. Dec 26 18:22:20.300860326 (506h53m20.171943361s) [noise=true , address=45.91.116.13:4155 , piece_id=7Z5CNXM7NK4H4KBR5WXMEVYKVCBBFYJEEVAUDC7NPRBLKMM4PT6Q, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.301243986 (506h53m20.172327052s) [noise=true , address=145.79.227.42:4123 , piece_id=DYVDISG4ZZSGO7TPYQCFM4ZSLCFUR7F35EFWJBLOF47EIKAPVTMQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.302210267 (506h53m20.173293302s) [noise=true , address=145.79.227.51:4410 , piece_id=54EBW2MUNATUOVLSISA7FNI7DKUOEDSN2MNOEUTQHDDQRHBE54XQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.302538793 (506h53m20.173621838s) [noise=true , address=145.79.227.45:4230 , piece_id=3F5NRWI4RRVB7GJBHW3HH34G7MAJBGQ47OWNPHKNDXZYBNYE6MWA, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.302911883 (506h53m20.173994918s) [noise=true , address=45.91.116.4:4937 , piece_id=OILDTM7QBOICW4GLPOMPFTOQGXN7KVPJER3RGT7FUSFHDMQMGUBQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.303243464 (506h53m20.174326499s) [noise=true , address=77.90.31.13:4629 , piece_id=HWLERTDU6GCEN6IQLRMIXR2TWJHJUFUFAENXZSFNDP4GYMOSQOGQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.303608820 (506h53m20.174691855s) [noise=true , address=77.90.31.16:4736 , piece_id=FHQYVM44UEO4RUTAHOFIQ7LO3WYXAJGW23DSSWA6JRHJZ3RU6HHQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.303906076 (506h53m20.174989121s) [noise=true , address=145.79.227.49:4339 , piece_id=EGL3XFJKGTELBWBI5DSKC7JWESOLRM2XMO6LRS4WCB6IJZWZ6CUQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.304052885 (506h53m20.17513592s) [noise=true , address=45.91.116.11:4118 , piece_id=G2O63TBJ7PM2AP6QPIVIIWSMR2EBFOOWJBQTMIB2GECGQUDCGVIQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.304543570 (506h53m20.175626605s) [noise=true , address=45.91.116.7:4012 , piece_id=NJC5EMFER5YYISOPZU6UARD6PUPZAHMUAF3TNB4LJPHXA4QJJO5A, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.305253973 (506h53m20.176337018s) [noise=true , address=145.79.227.6:4257 , piece_id=Q4EUKU62POICD3YVHHTR722DWDNOONOBG3LHMWGTISESZYWWF47Q, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.305734778 (506h53m20.176817833s) [noise=true , address=145.79.227.48:4303 , piece_id=EKVOCHB53K4HIN2JFZGRZPCJJUVIAC6BYKXPMEKJCTTNL7YSJDVA, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.306250310 (506h53m20.177333355s) [noise=true , address=145.79.227.46:4602 , piece_id=HH6MVNSJQCFI4TGAGLNCJS37LS7RB37KP5NDL4JAJJNKBIR7JTTQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.306729412 (506h53m20.177812447s) [noise=true , address=145.79.227.40:4049 , piece_id=A54YZBC5INMC4AIGFW45TDVGXO554WBTIXR7POY7H2SHZXV2LKPQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.307243411 (506h53m20.178326456s) [noise=true , address=145.79.227.8:4330 , piece_id=VUBUBFR73QD4K47IDNI24B6T3KKOUV7VIHO4QI3JFYFOYTHZ76BA, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.308077850 (506h53m20.179160895s) [noise=true , address=77.90.31.10:4520 , piece_id=SDDHFO7ZCRH4B5PLWEXHM7ZCBLX2V5GJLFLNLO3UZPTBKRW4FU7Q, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:22.021189354 (506h53m21.892272399s) [upload=444786]: Next returned nil segment Dec 26 18:22:22.021203520 (506h53m21.892286566s) [upload=444786]: Waiting for error group managing segments... Dec 26 18:22:22.306790919 (506h53m22.177873964s) [noise=true , address=145.79.227.42:4123 , piece_id=DYVDISG4ZZSGO7TPYQCFM4ZSLCFUR7F35EFWJBLOF47EIKAPVTMQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.317440578 (506h53m22.188523623s) [noise=true , address=145.79.227.48:4303 , piece_id=EKVOCHB53K4HIN2JFZGRZPCJJUVIAC6BYKXPMEKJCTTNL7YSJDVA, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.324498949 (506h53m22.195581994s) [noise=true , address=145.79.227.46:4602 , piece_id=HH6MVNSJQCFI4TGAGLNCJS37LS7RB37KP5NDL4JAJJNKBIR7JTTQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.332376839 (506h53m22.203459874s) [noise=true , address=145.79.227.45:4230 , piece_id=3F5NRWI4RRVB7GJBHW3HH34G7MAJBGQ47OWNPHKNDXZYBNYE6MWA, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.335008621 (506h53m22.206091666s) [noise=true , address=145.79.227.8:4330 , piece_id=VUBUBFR73QD4K47IDNI24B6T3KKOUV7VIHO4QI3JFYFOYTHZ76BA, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.337891891 (506h53m22.208974926s) [noise=true , address=145.79.227.51:4410 , piece_id=54EBW2MUNATUOVLSISA7FNI7DKUOEDSN2MNOEUTQHDDQRHBE54XQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.338036998 (506h53m22.209120033s) [noise=true , address=145.79.227.49:4339 , piece_id=EGL3XFJKGTELBWBI5DSKC7JWESOLRM2XMO6LRS4WCB6IJZWZ6CUQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.357245244 (506h53m22.228328289s) [noise=true , address=145.79.227.6:4257 , piece_id=Q4EUKU62POICD3YVHHTR722DWDNOONOBG3LHMWGTISESZYWWF47Q, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.491011368 (506h53m22.362094403s) [noise=true , address=77.90.31.16:4736 , piece_id=FHQYVM44UEO4RUTAHOFIQ7LO3WYXAJGW23DSSWA6JRHJZ3RU6HHQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.492463102 (506h53m22.363546137s) [noise=true , address=77.90.31.10:4520 , piece_id=SDDHFO7ZCRH4B5PLWEXHM7ZCBLX2V5GJLFLNLO3UZPTBKRW4FU7Q, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.543901169 (506h53m22.414984204s) [noise=true , address=45.91.116.4:4937 , piece_id=OILDTM7QBOICW4GLPOMPFTOQGXN7KVPJER3RGT7FUSFHDMQMGUBQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.555202418 (506h53m22.426285454s) [noise=true , address=45.91.116.7:4012 , piece_id=NJC5EMFER5YYISOPZU6UARD6PUPZAHMUAF3TNB4LJPHXA4QJJO5A, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.584924595 (506h53m22.45600763s) [noise=true , address=77.90.31.13:4629 , piece_id=HWLERTDU6GCEN6IQLRMIXR2TWJHJUFUFAENXZSFNDP4GYMOSQOGQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.584944242 (506h53m22.456027277s) [seg_pos={0 0}, upload=444786]: Segment reached optimal threshold of 13 pieces. Dec 26 18:22:22.585279571 (506h53m22.456362616s) [noise=true , address=145.79.227.40:4049 , piece_id=A54YZBC5INMC4AIGFW45TDVGXO554WBTIXR7POY7H2SHZXV2LKPQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 18:22:22.585291072 (506h53m22.456374107s) [noise=true , address=145.79.227.40:4049 , piece_id=A54YZBC5INMC4AIGFW45TDVGXO554WBTIXR7POY7H2SHZXV2LKPQ, seg_pos={0 0}, upload=444786]: Long tail cancellation due to optimal threshold Dec 26 18:22:22.585390301 (506h53m22.456473366s) [noise=true , address=45.91.116.11:4118 , piece_id=G2O63TBJ7PM2AP6QPIVIIWSMR2EBFOOWJBQTMIB2GECGQUDCGVIQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:54922->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:54922->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54922->45.91.116.11:4118: use of closed network connection Dec 26 18:22:22.585420619 (506h53m22.456503674s) [noise=true , address=45.91.116.11:4118 , piece_id=G2O63TBJ7PM2AP6QPIVIIWSMR2EBFOOWJBQTMIB2GECGQUDCGVIQ, seg_pos={0 0}, upload=444786]: Long tail cancellation due to optimal threshold Dec 26 18:22:22.585629497 (506h53m22.456712552s) [noise=true , address=45.91.116.13:4155 , piece_id=7Z5CNXM7NK4H4KBR5WXMEVYKVCBBFYJEEVAUDC7NPRBLKMM4PT6Q, seg_pos={0 0}, upload=444786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 26 18:22:22.585657961 (506h53m22.456741006s) [noise=true , address=45.91.116.13:4155 , piece_id=7Z5CNXM7NK4H4KBR5WXMEVYKVCBBFYJEEVAUDC7NPRBLKMM4PT6Q, seg_pos={0 0}, upload=444786]: Long tail cancellation due to optimal threshold Dec 26 18:22:22.585679984 (506h53m22.456763029s) [seg_pos={0 0}, upload=444786]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:22:22.585785955 (506h53m22.45686899s) [upload=444786]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:22:22.585792888 (506h53m22.456875923s) [upload=444786]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:22:22.894574533 (506h53m22.765657578s) [upload=444786]: Done uploading segments... Dec 26 18:22:22.894598457 (506h53m22.765681492s) [upload=444786]: Upload finished. err: <nil> Duration: 30.003759798s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X.XXX.....XX.X.XXX.........X..X.....X...XX.........X.....] duration: 6.133632138s trace: https://jaeger.dev.storj.io/trace/3b66cb5bbee176a1 FAIL (2025-12-26 18:22:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A) context deadline exceeded ======LOGS====== gments... Dec 26 18:17:44.233899075 (506h48m44.104982321s) [upload=444720]: Got next segment. Inline: false Dec 26 18:17:44.234055674 (506h48m44.105138719s) [upload=444720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:17:44.234062016 (506h48m44.105145051s) [upload=444720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:17:44.482104382 (506h48m44.353187437s) [seg_pos={0 0}, upload=444720]: Begin upload segment... Dec 26 18:17:44.482168003 (506h48m44.353251038s) [seg_pos={0 0}, upload=444720]: Done begin upload segment. Dec 26 18:17:44.482244459 (506h48m44.353327494s) [noise=true , address=45.91.116.14:4190 , piece_id=HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.482284294 (506h48m44.35336734s) [noise=true , address=45.91.116.6:4973 , piece_id=6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483125356 (506h48m44.354208401s) [noise=true , address=145.79.227.7:4294 , piece_id=HMLTVW3MUFCAR6BG74SVDTNSICW72VYZX4J2YNIH4OAZA6M2MYNA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483466666 (506h48m44.354549701s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483697005 (506h48m44.35478005s) [noise=true , address=145.79.227.11:4731 , piece_id=MFVIRTTK4HMMEVYLEQLNJI4GHPHDSKNCKES5265OOKHP35V2BZ4Q, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483845718 (506h48m44.354928753s) [noise=true , address=45.91.116.5:4298 , piece_id=P7URE5INZE6MEYA7OB2EWBMNV264UFRFWHECXGTDCNQWNDEZLUIQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484134077 (506h48m44.355217112s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484298069 (506h48m44.355381104s) [noise=true , address=145.79.227.52:4446 , piece_id=JQV34TEV346A4PWGH3CDKD2GHAP6KNVUHJ3DZJD6C34G2I23EBFA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484400655 (506h48m44.3554837s) [noise=true , address=145.79.227.49:4337 , piece_id=SXX4XTTPZFHVLPQ6MT2S6KDMQBTSUDR35Q4WZMVJSTGHR5S47VFQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484691678 (506h48m44.355774713s) [noise=true , address=145.79.227.37:4566 , piece_id=2UGNFWUMBFESEFINRGNYGP5BSZ2AGBP4V5KGMYJXIWISYGFSY7XQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484819723 (506h48m44.355902768s) [noise=true , address=145.79.227.29:4942 , piece_id=UVESXQQA4WWA6BLWFQYKKXZSB72OS5S7VOWLOTKSGC64CPMVRVCA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484986671 (506h48m44.356069706s) [noise=true , address=145.79.227.36:4517 , piece_id=GNQIP445TCJH3DRP34QUKON66OCCGA54XZ67ILL6G5WXFHBV54UQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.485248549 (506h48m44.356331614s) [noise=true , address=145.79.227.22:4511 , piece_id=JEVHL5N6JFIIZJGIMMKPGGW5O4FRYU3TLL6SGA2DXYUU7IEWFSXA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.485634844 (506h48m44.356717879s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.486143163 (506h48m44.357226208s) [noise=true , address=145.79.227.31:4799 , piece_id=7VDEU4P6RWTT2SL6GFZGAYJYVGMGNEUZBX5ELRGQQME2AVI4EGYA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.486483892 (506h48m44.357566937s) [noise=true , address=145.79.227.47:4266 , piece_id=LMYKFSRNB4UBLK2Z7FRWKCRUIP3PTXNMSZA4VIUUHOIIMVCZEHHA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:46.098958535 (506h48m45.970041571s) [upload=444720]: Next returned nil segment Dec 26 18:17:46.098973043 (506h48m45.970056078s) [upload=444720]: Waiting for error group managing segments... Dec 26 18:17:46.482233768 (506h48m46.353316813s) [noise=true , address=145.79.227.36:4517 , piece_id=GNQIP445TCJH3DRP34QUKON66OCCGA54XZ67ILL6G5WXFHBV54UQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.484253554 (506h48m46.35533662s) [noise=true , address=145.79.227.22:4511 , piece_id=JEVHL5N6JFIIZJGIMMKPGGW5O4FRYU3TLL6SGA2DXYUU7IEWFSXA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.484520213 (506h48m46.355603258s) [noise=true , address=145.79.227.49:4337 , piece_id=SXX4XTTPZFHVLPQ6MT2S6KDMQBTSUDR35Q4WZMVJSTGHR5S47VFQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.485024783 (506h48m46.356107838s) [noise=true , address=145.79.227.29:4942 , piece_id=UVESXQQA4WWA6BLWFQYKKXZSB72OS5S7VOWLOTKSGC64CPMVRVCA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.490941920 (506h48m46.362024955s) [noise=true , address=145.79.227.37:4566 , piece_id=2UGNFWUMBFESEFINRGNYGP5BSZ2AGBP4V5KGMYJXIWISYGFSY7XQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.491343145 (506h48m46.3624262s) [noise=true , address=145.79.227.52:4446 , piece_id=JQV34TEV346A4PWGH3CDKD2GHAP6KNVUHJ3DZJD6C34G2I23EBFA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.491517597 (506h48m46.362600642s) [noise=true , address=145.79.227.11:4731 , piece_id=MFVIRTTK4HMMEVYLEQLNJI4GHPHDSKNCKES5265OOKHP35V2BZ4Q, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.505557661 (506h48m46.376640706s) [noise=true , address=145.79.227.47:4266 , piece_id=LMYKFSRNB4UBLK2Z7FRWKCRUIP3PTXNMSZA4VIUUHOIIMVCZEHHA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.511497923 (506h48m46.382580968s) [noise=true , address=145.79.227.31:4799 , piece_id=7VDEU4P6RWTT2SL6GFZGAYJYVGMGNEUZBX5ELRGQQME2AVI4EGYA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.748964861 (506h48m46.620047906s) [noise=true , address=145.79.227.7:4294 , piece_id=HMLTVW3MUFCAR6BG74SVDTNSICW72VYZX4J2YNIH4OAZA6M2MYNA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.018364295 (506h48m46.88944734s) [noise=true , address=45.91.116.14:4190 , piece_id=HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.520456737 (506h48m47.391539782s) [noise=true , address=45.91.116.6:4973 , piece_id=6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.646618581 (506h48m47.517701626s) [noise=true , address=45.91.116.5:4298 , piece_id=P7URE5INZE6MEYA7OB2EWBMNV264UFRFWHECXGTDCNQWNDEZLUIQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.646719584 (506h48m47.517802619s) [seg_pos={0 0}, upload=444720]: Segment reached optimal threshold of 13 pieces. Dec 26 18:17:47.646919054 (506h48m47.518002089s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection Dec 26 18:17:47.646927289 (506h48m47.518010334s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.646940303 (506h48m47.518023339s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection Dec 26 18:17:47.646950263 (506h48m47.518033298s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.647272486 (506h48m47.518355522s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 18:17:47.647280201 (506h48m47.518363236s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.647299127 (506h48m47.518382162s) [seg_pos={0 0}, upload=444720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:17:47.647384159 (506h48m47.518467194s) [upload=444720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:17:47.647390200 (506h48m47.518473235s) [upload=444720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:17:47.945659407 (506h48m47.816742442s) [upload=444720]: Done uploading segments... Dec 26 18:17:47.945688893 (506h48m47.816771928s) [upload=444720]: Upload finished. err: <nil> Duration: 6.133659181s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 15.379472877s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:22:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 15.379491543s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 33.876030851s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:21:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 33.876051089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X.XXX.....XX.X.XXX.........X..X.....X...XX.........X....] duration: 5.834379679s trace: https://jaeger.dev.storj.io/trace/3b66cb5bbee176a1 FAIL (2025-12-26 18:21:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A) context deadline exceeded ======LOGS====== gments... Dec 26 18:17:44.233899075 (506h48m44.104982321s) [upload=444720]: Got next segment. Inline: false Dec 26 18:17:44.234055674 (506h48m44.105138719s) [upload=444720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:17:44.234062016 (506h48m44.105145051s) [upload=444720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:17:44.482104382 (506h48m44.353187437s) [seg_pos={0 0}, upload=444720]: Begin upload segment... Dec 26 18:17:44.482168003 (506h48m44.353251038s) [seg_pos={0 0}, upload=444720]: Done begin upload segment. Dec 26 18:17:44.482244459 (506h48m44.353327494s) [noise=true , address=45.91.116.14:4190 , piece_id=HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.482284294 (506h48m44.35336734s) [noise=true , address=45.91.116.6:4973 , piece_id=6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483125356 (506h48m44.354208401s) [noise=true , address=145.79.227.7:4294 , piece_id=HMLTVW3MUFCAR6BG74SVDTNSICW72VYZX4J2YNIH4OAZA6M2MYNA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483466666 (506h48m44.354549701s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483697005 (506h48m44.35478005s) [noise=true , address=145.79.227.11:4731 , piece_id=MFVIRTTK4HMMEVYLEQLNJI4GHPHDSKNCKES5265OOKHP35V2BZ4Q, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483845718 (506h48m44.354928753s) [noise=true , address=45.91.116.5:4298 , piece_id=P7URE5INZE6MEYA7OB2EWBMNV264UFRFWHECXGTDCNQWNDEZLUIQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484134077 (506h48m44.355217112s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484298069 (506h48m44.355381104s) [noise=true , address=145.79.227.52:4446 , piece_id=JQV34TEV346A4PWGH3CDKD2GHAP6KNVUHJ3DZJD6C34G2I23EBFA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484400655 (506h48m44.3554837s) [noise=true , address=145.79.227.49:4337 , piece_id=SXX4XTTPZFHVLPQ6MT2S6KDMQBTSUDR35Q4WZMVJSTGHR5S47VFQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484691678 (506h48m44.355774713s) [noise=true , address=145.79.227.37:4566 , piece_id=2UGNFWUMBFESEFINRGNYGP5BSZ2AGBP4V5KGMYJXIWISYGFSY7XQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484819723 (506h48m44.355902768s) [noise=true , address=145.79.227.29:4942 , piece_id=UVESXQQA4WWA6BLWFQYKKXZSB72OS5S7VOWLOTKSGC64CPMVRVCA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484986671 (506h48m44.356069706s) [noise=true , address=145.79.227.36:4517 , piece_id=GNQIP445TCJH3DRP34QUKON66OCCGA54XZ67ILL6G5WXFHBV54UQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.485248549 (506h48m44.356331614s) [noise=true , address=145.79.227.22:4511 , piece_id=JEVHL5N6JFIIZJGIMMKPGGW5O4FRYU3TLL6SGA2DXYUU7IEWFSXA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.485634844 (506h48m44.356717879s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.486143163 (506h48m44.357226208s) [noise=true , address=145.79.227.31:4799 , piece_id=7VDEU4P6RWTT2SL6GFZGAYJYVGMGNEUZBX5ELRGQQME2AVI4EGYA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.486483892 (506h48m44.357566937s) [noise=true , address=145.79.227.47:4266 , piece_id=LMYKFSRNB4UBLK2Z7FRWKCRUIP3PTXNMSZA4VIUUHOIIMVCZEHHA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:46.098958535 (506h48m45.970041571s) [upload=444720]: Next returned nil segment Dec 26 18:17:46.098973043 (506h48m45.970056078s) [upload=444720]: Waiting for error group managing segments... Dec 26 18:17:46.482233768 (506h48m46.353316813s) [noise=true , address=145.79.227.36:4517 , piece_id=GNQIP445TCJH3DRP34QUKON66OCCGA54XZ67ILL6G5WXFHBV54UQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.484253554 (506h48m46.35533662s) [noise=true , address=145.79.227.22:4511 , piece_id=JEVHL5N6JFIIZJGIMMKPGGW5O4FRYU3TLL6SGA2DXYUU7IEWFSXA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.484520213 (506h48m46.355603258s) [noise=true , address=145.79.227.49:4337 , piece_id=SXX4XTTPZFHVLPQ6MT2S6KDMQBTSUDR35Q4WZMVJSTGHR5S47VFQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.485024783 (506h48m46.356107838s) [noise=true , address=145.79.227.29:4942 , piece_id=UVESXQQA4WWA6BLWFQYKKXZSB72OS5S7VOWLOTKSGC64CPMVRVCA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.490941920 (506h48m46.362024955s) [noise=true , address=145.79.227.37:4566 , piece_id=2UGNFWUMBFESEFINRGNYGP5BSZ2AGBP4V5KGMYJXIWISYGFSY7XQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.491343145 (506h48m46.3624262s) [noise=true , address=145.79.227.52:4446 , piece_id=JQV34TEV346A4PWGH3CDKD2GHAP6KNVUHJ3DZJD6C34G2I23EBFA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.491517597 (506h48m46.362600642s) [noise=true , address=145.79.227.11:4731 , piece_id=MFVIRTTK4HMMEVYLEQLNJI4GHPHDSKNCKES5265OOKHP35V2BZ4Q, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.505557661 (506h48m46.376640706s) [noise=true , address=145.79.227.47:4266 , piece_id=LMYKFSRNB4UBLK2Z7FRWKCRUIP3PTXNMSZA4VIUUHOIIMVCZEHHA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.511497923 (506h48m46.382580968s) [noise=true , address=145.79.227.31:4799 , piece_id=7VDEU4P6RWTT2SL6GFZGAYJYVGMGNEUZBX5ELRGQQME2AVI4EGYA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.748964861 (506h48m46.620047906s) [noise=true , address=145.79.227.7:4294 , piece_id=HMLTVW3MUFCAR6BG74SVDTNSICW72VYZX4J2YNIH4OAZA6M2MYNA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.018364295 (506h48m46.88944734s) [noise=true , address=45.91.116.14:4190 , piece_id=HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.520456737 (506h48m47.391539782s) [noise=true , address=45.91.116.6:4973 , piece_id=6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.646618581 (506h48m47.517701626s) [noise=true , address=45.91.116.5:4298 , piece_id=P7URE5INZE6MEYA7OB2EWBMNV264UFRFWHECXGTDCNQWNDEZLUIQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.646719584 (506h48m47.517802619s) [seg_pos={0 0}, upload=444720]: Segment reached optimal threshold of 13 pieces. Dec 26 18:17:47.646919054 (506h48m47.518002089s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection Dec 26 18:17:47.646927289 (506h48m47.518010334s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.646940303 (506h48m47.518023339s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection Dec 26 18:17:47.646950263 (506h48m47.518033298s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.647272486 (506h48m47.518355522s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 18:17:47.647280201 (506h48m47.518363236s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.647299127 (506h48m47.518382162s) [seg_pos={0 0}, upload=444720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:17:47.647384159 (506h48m47.518467194s) [upload=444720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:17:47.647390200 (506h48m47.518473235s) [upload=444720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:17:47.945659407 (506h48m47.816742442s) [upload=444720]: Done uploading segments... Dec 26 18:17:47.945688893 (506h48m47.816771928s) [upload=444720]: Upload finished. err: <nil> Duration: 5.834412691s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 43.42257527s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:21:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 43.422589438s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 41.797177059s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:20:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 41.797207607s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)