Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X..XX..X.......X.......................XX....X.....XX...XXX] duration: 30.003760288s trace: https://jaeger.dev.storj.io/trace/659477683d696568 FAIL (2025-12-28 11:47:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 27NDJKIOWEFXMWIY3IBHZ3WOS3EIXWOLSJGHEM677U2JCDAM7IGA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: BP6WJWGZZ66YFGRB7F4EIO4YPDP4INEW7YMQOSHLROAYCMJI367Q) context deadline exceeded ======LOGS====== Dec 28 11:46:50.596722197 (548h17m50.467805232s) [upload=480773]: Starting upload Dec 28 11:46:50.596785147 (548h17m50.467868183s) [upload=480773]: Done starting upload Dec 28 11:46:50.596945943 (548h17m50.468028988s) [upload=480773]: Uploading segments... Dec 28 11:46:50.597044300 (548h17m50.468127336s) [upload=480773]: Got next segment. Inline: false Dec 28 11:46:50.597090498 (548h17m50.468173533s) [upload=480773]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:46:50.597097252 (548h17m50.468180297s) [upload=480773]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:46:50.842523763 (548h17m50.713606808s) [seg_pos={0 0}, upload=480773]: Begin upload segment... Dec 28 11:46:50.842577756 (548h17m50.713660811s) [seg_pos={0 0}, upload=480773]: Done begin upload segment. Dec 28 11:46:50.842619626 (548h17m50.713702661s) [noise=true , address=145.79.227.51:4410 , piece_id=J6J6UJX4IQQSI7I2GVEC545QM2N6H4SA3CSK6BO3OMB4QMMXEYAQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.842717963 (548h17m50.713801008s) [noise=true , address=145.79.227.14:4839 , piece_id=HXRQKFVVHDALI32SNNF4QZTLWKXOPHJDUKGYBXICIBCKVI6KUPKA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.843224687 (548h17m50.714307733s) [noise=true , address=145.79.227.11:4731 , piece_id=CQD42RUXSNLQ6JYVAJB4HWLEACAG3B3WLIFXI5LS3CY6WIEREMMQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.843679794 (548h17m50.714762849s) [noise=true , address=145.79.227.46:4603 , piece_id=FFJSD3HSVICHVI7V5WJM556XB467JLJIRFBIEIWXRAQBJ5HZH6OQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.844119522 (548h17m50.715202567s) [noise=true , address=77.90.31.13:4626 , piece_id=27NDJKIOWEFXMWIY3IBHZ3WOS3EIXWOLSJGHEM677U2JCDAM7IGA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.844512700 (548h17m50.715595745s) [noise=true , address=145.79.227.47:4266 , piece_id=HPLDUZKXETDTN4Z4NGGKQK4WA4JZYYPNJ6SZHQHVLJRUQTR57QNQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.845063108 (548h17m50.716146143s) [noise=true , address=45.91.116.7:4009 , piece_id=BBONIZKCWXNNSIY4WF7ETQYPTZR2PXVLKI4ISQA2W6YYHXYU3CTA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.845526731 (548h17m50.716609776s) [noise=true , address=145.79.227.50:4373 , piece_id=S6EJBMYLXDLB4GGTGSC2ADZSMYG73QGO4IVQWEW7FCX2A6ZINQNA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.846167531 (548h17m50.717250566s) [noise=true , address=45.91.116.8:4045 , piece_id=EXG3WMGBWISBV5VBQUDQJTEPPIADCABHEEKFTQ2QRACNXH7L2GXA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.846188481 (548h17m50.717271516s) [noise=true , address=45.91.116.11:4118 , piece_id=BP6WJWGZZ66YFGRB7F4EIO4YPDP4INEW7YMQOSHLROAYCMJI367Q, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.846769698 (548h17m50.717852733s) [noise=true , address=145.79.227.39:4015 , piece_id=7T4WEVGEQPHMALPSWOCDYT2AVM73PVGX446EJLOV7ZB2IPJIXJ5Q, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.847176802 (548h17m50.718259838s) [noise=true , address=145.79.227.52:4445 , piece_id=UM5QVDFZZ6YUIEWLM7NSOF5L32ZM2BIGGXFQMXIUDMH3QP7OA2QA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.847670734 (548h17m50.718753779s) [noise=true , address=145.79.227.42:4121 , piece_id=RHVKJ43T4DSMM7ZSGEMZH5IKQJO2BIAYHJ6BD5ZIDFKBCR4IEQJQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.848209459 (548h17m50.719292494s) [noise=true , address=145.79.227.17:4972 , piece_id=UVNAIRYVT7ZRF4WHXUI63NZHZVDDUPW255A7IE4A777TI6EX3ITA, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.848566930 (548h17m50.719649975s) [noise=true , address=77.90.31.19:4842 , piece_id=3MSF5KJS7QTSDUGVLHNA2BMI2JUKNS3E56SJBK27U6KLWSKA5PEQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:50.849049198 (548h17m50.720132233s) [noise=true , address=145.79.227.26:4655 , piece_id=6KZUZFJ3IUKO53NT4TKZBYJPEILSQLSM5OQEQE77TUTVJP32IANQ, seg_pos={0 0}, upload=480773]: Uploading piece... Dec 28 11:46:52.467548681 (548h17m52.338631726s) [upload=480773]: Next returned nil segment Dec 28 11:46:52.467583578 (548h17m52.338666633s) [upload=480773]: Waiting for error group managing segments... Dec 28 11:46:52.854096120 (548h17m52.725179165s) [noise=true , address=145.79.227.39:4015 , piece_id=7T4WEVGEQPHMALPSWOCDYT2AVM73PVGX446EJLOV7ZB2IPJIXJ5Q, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.856871475 (548h17m52.72795451s) [noise=true , address=145.79.227.46:4603 , piece_id=FFJSD3HSVICHVI7V5WJM556XB467JLJIRFBIEIWXRAQBJ5HZH6OQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.857881879 (548h17m52.728964924s) [noise=true , address=145.79.227.17:4972 , piece_id=UVNAIRYVT7ZRF4WHXUI63NZHZVDDUPW255A7IE4A777TI6EX3ITA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.859295701 (548h17m52.730378746s) [noise=true , address=145.79.227.14:4839 , piece_id=HXRQKFVVHDALI32SNNF4QZTLWKXOPHJDUKGYBXICIBCKVI6KUPKA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.861026837 (548h17m52.732109882s) [noise=true , address=145.79.227.26:4655 , piece_id=6KZUZFJ3IUKO53NT4TKZBYJPEILSQLSM5OQEQE77TUTVJP32IANQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.863741557 (548h17m52.734824602s) [noise=true , address=145.79.227.50:4373 , piece_id=S6EJBMYLXDLB4GGTGSC2ADZSMYG73QGO4IVQWEW7FCX2A6ZINQNA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.864182076 (548h17m52.735265121s) [noise=true , address=145.79.227.47:4266 , piece_id=HPLDUZKXETDTN4Z4NGGKQK4WA4JZYYPNJ6SZHQHVLJRUQTR57QNQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.866334654 (548h17m52.737417699s) [noise=true , address=145.79.227.51:4410 , piece_id=J6J6UJX4IQQSI7I2GVEC545QM2N6H4SA3CSK6BO3OMB4QMMXEYAQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.867731293 (548h17m52.738814328s) [noise=true , address=145.79.227.42:4121 , piece_id=RHVKJ43T4DSMM7ZSGEMZH5IKQJO2BIAYHJ6BD5ZIDFKBCR4IEQJQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.871317001 (548h17m52.742400036s) [noise=true , address=145.79.227.52:4445 , piece_id=UM5QVDFZZ6YUIEWLM7NSOF5L32ZM2BIGGXFQMXIUDMH3QP7OA2QA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:52.882538299 (548h17m52.753621344s) [noise=true , address=145.79.227.11:4731 , piece_id=CQD42RUXSNLQ6JYVAJB4HWLEACAG3B3WLIFXI5LS3CY6WIEREMMQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:53.043334249 (548h17m52.914417294s) [noise=true , address=77.90.31.19:4842 , piece_id=3MSF5KJS7QTSDUGVLHNA2BMI2JUKNS3E56SJBK27U6KLWSKA5PEQ, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:53.129100552 (548h17m53.000183597s) [noise=true , address=77.90.31.13:4626 , piece_id=27NDJKIOWEFXMWIY3IBHZ3WOS3EIXWOLSJGHEM677U2JCDAM7IGA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:53.129128024 (548h17m53.00021108s) [seg_pos={0 0}, upload=480773]: Segment reached optimal threshold of 13 pieces. Dec 28 11:46:53.129162600 (548h17m53.000245645s) [noise=true , address=45.91.116.11:4118 , piece_id=BP6WJWGZZ66YFGRB7F4EIO4YPDP4INEW7YMQOSHLROAYCMJI367Q, seg_pos={0 0}, upload=480773]: Done uploading piece. err: <nil> Dec 28 11:46:53.129330580 (548h17m53.000413625s) [noise=true , address=45.91.116.7:4009 , piece_id=BBONIZKCWXNNSIY4WF7ETQYPTZR2PXVLKI4ISQA2W6YYHXYU3CTA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:43986->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:43986->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43986->45.91.116.7:4009: use of closed network connection Dec 28 11:46:53.129340870 (548h17m53.000423905s) [noise=true , address=45.91.116.7:4009 , piece_id=BBONIZKCWXNNSIY4WF7ETQYPTZR2PXVLKI4ISQA2W6YYHXYU3CTA, seg_pos={0 0}, upload=480773]: Long tail cancellation due to optimal threshold Dec 28 11:46:53.129509952 (548h17m53.000592997s) [noise=true , address=45.91.116.8:4045 , piece_id=EXG3WMGBWISBV5VBQUDQJTEPPIADCABHEEKFTQ2QRACNXH7L2GXA, seg_pos={0 0}, upload=480773]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 11:46:53.129519259 (548h17m53.000602294s) [noise=true , address=45.91.116.8:4045 , piece_id=EXG3WMGBWISBV5VBQUDQJTEPPIADCABHEEKFTQ2QRACNXH7L2GXA, seg_pos={0 0}, upload=480773]: Long tail cancellation due to optimal threshold Dec 28 11:46:53.129540931 (548h17m53.000623976s) [seg_pos={0 0}, upload=480773]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 28 11:46:53.129620672 (548h17m53.000703718s) [upload=480773]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:46:53.129628888 (548h17m53.000711933s) [upload=480773]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:46:53.421448279 (548h17m53.292531324s) [upload=480773]: Done uploading segments... Dec 28 11:46:53.421479930 (548h17m53.292562985s) [upload=480773]: Upload finished. err: <nil> Duration: 30.003773945s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X........X...........X..X...................] duration: 37.98738284s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:46:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 37.987400344s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.X..XX..X.......X.......................XX....X.....XX...XX] duration: 30.003340613s trace: https://jaeger.dev.storj.io/trace/78e587b0063a5fa8 FAIL (2025-12-28 11:46:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: BQH2BFJE7CETYFEN46RIFKOWN5HZPU42GBOQJB3H5ZQYCUOODPLQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: JD2BZZSWEGA72K52HTFHYGSEF23CNFTS76UFCZIQNV7ARFPVOHOQ) context deadline exceeded ======LOGS====== segments... Dec 28 11:46:20.377663682 (548h17m20.248746717s) [upload=480766]: Got next segment. Inline: false Dec 28 11:46:20.377716964 (548h17m20.248799999s) [upload=480766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:46:20.377722935 (548h17m20.24880598s) [upload=480766]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:46:20.623231001 (548h17m20.494314037s) [seg_pos={0 0}, upload=480766]: Begin upload segment... Dec 28 11:46:20.623285896 (548h17m20.494368942s) [seg_pos={0 0}, upload=480766]: Done begin upload segment. Dec 28 11:46:20.623333346 (548h17m20.494416381s) [noise=true , address=145.79.227.14:4840 , piece_id=WARXF6SQ5XXSSMJ3WN6RYCTTD52LDBCU7DEPFLHQZ2HR2O2LXCDA, seg_pos={0 0}, upload=480766]: Uploading piece... Dec 28 11:46:20.623373894 (548h17m20.49445694s) [noise=true , address=45.91.116.6:4975 , piece_id=557JHRPZ5T22KX3CY7YYYKKDL5KSGBSJEKYASHNIDWWT3ZCNYTHQ, seg_pos={0 0}, upload=480766]: Uploading piece... Dec 28 11:46:20.623867574 (548h17m20.494950619s) [noise=true , address=145.79.227.19:4077 , piece_id=24TFN77FDBXV4SXNSQ356UZWR6KRS2B4HKGSXO7XPYWMWITQ7NSQ, seg_pos={0 0}, upload=480766]: Uploading piece... Dec 28 11:46:20.624173016 (548h17m20.495256051s) [noise=true , address=77.90.31.15:4700 , piece_id=BQH2BFJE7CETYFEN46RIFKOWN5HZPU42GBOQJB3H5ZQYCUOODPLQ, seg_pos={0 0}, upload=480766]: Uploading piece... Dec 28 11:46:20.624538151 (548h17m20.495621186s) [noise=true , address=145.79.227.10:4401 , piece_id=JIDZQWGKX3QYJLG7PYOD3YRYUZEQKKAHWC5VPOWDMAOLXCBXGELA, seg_pos={0 0}, upload=480766]: Uploading piece... Dec 28 11:46:20.624834666 (548h17m20.495917701s) [noise=true , address=145.79.227.50:4375 , piece_id=2JDZUO3MD2Q4HAZ3BHM4UPLYTYVSNWGWCXLQGJJLUBDL4REZXWXQ, seg_pos={0 0}, upload=480766]: Uploading piece... Dec 28 11:46:20.625190323 (548h17m20.496273358s) [noise=true , address=145.79.227.27:4691 , piece_id=ZTECKDJU2VXQPTG3DR5L5YPPUGE5CYOGD3JDA2MKNBHQ652MJ55Q, seg_pos={0 0}, upload=480766]: Uploading piece... Dec 28 11:46:20.625470847 (548h17m20.496553882s) [noise=true , address=77.90.31.21:4878 , piece_id=2OY46MDP5DRA7SWQSMQNF5VW6EDD43NDHEHLLE6QDKPEH7WPJTQQ, seg_pos={0 0}, upload=480766]: Uploading piece... Dec 28 11:46:20.625848697 (548h17m20.496931732s) [noise=true , address=145.79.227.41:4085 , piece_id=5WA22ON3XDFDUTA7P3OTRFIIC346LIEW6XMW6Z2PYWYF5W3UP6KQ, seg_pos={0 0}, upload=480766]: Uploading piece... Dec 28 11:46:20.627006161 (548h17m20.498089196s) [noise=true , address=145.79.227.3:4149 , piece_id=TLWRF3OE2TBP5WIAKYL3KUYZ3LTVXXMWI2EY7OYT2IU5A6JP2POA, seg_pos={0 0}, upload=480766]: Uploading piece... Dec 28 11:46:20.628694745 (548h17m20.499777781s) [noise=true , address=45.91.116.14:4190 , piece_id=IVFZHG7EGMOL6YV2REX7UN5YPP6QJZZZHHBK5ODHW64XLYGQ434Q, seg_pos={0 0}, upload=480766]: Uploading piece... Dec 28 11:46:20.630418809 (548h17m20.501501844s) [noise=true , address=45.91.116.5:4299 , piece_id=JD2BZZSWEGA72K52HTFHYGSEF23CNFTS76UFCZIQNV7ARFPVOHOQ, seg_pos={0 0}, upload=480766]: Uploading piece... Dec 28 11:46:20.631969943 (548h17m20.503052978s) [noise=true , address=145.79.227.12:4767 , piece_id=NXQZVJMESIA277TFQNMKRGKDYYBMWXRDJCJXZZEIQOF5HGVMVYGQ, seg_pos={0 0}, upload=480766]: Uploading piece... Dec 28 11:46:20.633446935 (548h17m20.50452997s) [noise=true , address=45.91.116.9:4262 , piece_id=ZVY5NDSESC6E7VKVTBEFQO2KS2C2YGOW6MHUMCIR5VYGRDSOSYAA, seg_pos={0 0}, upload=480766]: Uploading piece... Dec 28 11:46:20.634097895 (548h17m20.50518093s) [noise=true , address=77.90.31.19:4844 , piece_id=U3EQIA6ZIFDAL5PKCZ6ARGX3H5QVSG4WJUT7JDDTQ26GOOKPK6XA, seg_pos={0 0}, upload=480766]: Uploading piece... Dec 28 11:46:20.635381479 (548h17m20.506464514s) [noise=true , address=145.79.227.36:4517 , piece_id=GXFKN6HBOVDALM32J4GZ6JGF7EE44OACPW6URWDU5V6LA42GD7KQ, seg_pos={0 0}, upload=480766]: Uploading piece... Dec 28 11:46:22.331503056 (548h17m22.202586091s) [upload=480766]: Next returned nil segment Dec 28 11:46:22.331520320 (548h17m22.202603355s) [upload=480766]: Waiting for error group managing segments... Dec 28 11:46:22.623465149 (548h17m22.494548184s) [noise=true , address=145.79.227.41:4085 , piece_id=5WA22ON3XDFDUTA7P3OTRFIIC346LIEW6XMW6Z2PYWYF5W3UP6KQ, seg_pos={0 0}, upload=480766]: Done uploading piece. err: <nil> Dec 28 11:46:22.629665746 (548h17m22.500748791s) [noise=true , address=145.79.227.19:4077 , piece_id=24TFN77FDBXV4SXNSQ356UZWR6KRS2B4HKGSXO7XPYWMWITQ7NSQ, seg_pos={0 0}, upload=480766]: Done uploading piece. err: <nil> Dec 28 11:46:22.631155553 (548h17m22.502238588s) [noise=true , address=145.79.227.27:4691 , piece_id=ZTECKDJU2VXQPTG3DR5L5YPPUGE5CYOGD3JDA2MKNBHQ652MJ55Q, seg_pos={0 0}, upload=480766]: Done uploading piece. err: <nil> Dec 28 11:46:22.631673339 (548h17m22.502756374s) [noise=true , address=145.79.227.10:4401 , piece_id=JIDZQWGKX3QYJLG7PYOD3YRYUZEQKKAHWC5VPOWDMAOLXCBXGELA, seg_pos={0 0}, upload=480766]: Done uploading piece. err: <nil> Dec 28 11:46:22.633019151 (548h17m22.504102186s) [noise=true , address=145.79.227.3:4149 , piece_id=TLWRF3OE2TBP5WIAKYL3KUYZ3LTVXXMWI2EY7OYT2IU5A6JP2POA, seg_pos={0 0}, upload=480766]: Done uploading piece. err: <nil> Dec 28 11:46:22.638715218 (548h17m22.509798263s) [noise=true , address=145.79.227.36:4517 , piece_id=GXFKN6HBOVDALM32J4GZ6JGF7EE44OACPW6URWDU5V6LA42GD7KQ, seg_pos={0 0}, upload=480766]: Done uploading piece. err: <nil> Dec 28 11:46:22.639099540 (548h17m22.510182575s) [noise=true , address=145.79.227.50:4375 , piece_id=2JDZUO3MD2Q4HAZ3BHM4UPLYTYVSNWGWCXLQGJJLUBDL4REZXWXQ, seg_pos={0 0}, upload=480766]: Done uploading piece. err: <nil> Dec 28 11:46:22.642058073 (548h17m22.513141108s) [noise=true , address=145.79.227.14:4840 , piece_id=WARXF6SQ5XXSSMJ3WN6RYCTTD52LDBCU7DEPFLHQZ2HR2O2LXCDA, seg_pos={0 0}, upload=480766]: Done uploading piece. err: <nil> Dec 28 11:46:22.683048376 (548h17m22.554131421s) [noise=true , address=145.79.227.12:4767 , piece_id=NXQZVJMESIA277TFQNMKRGKDYYBMWXRDJCJXZZEIQOF5HGVMVYGQ, seg_pos={0 0}, upload=480766]: Done uploading piece. err: <nil> Dec 28 11:46:22.803709453 (548h17m22.674792488s) [noise=true , address=77.90.31.21:4878 , piece_id=2OY46MDP5DRA7SWQSMQNF5VW6EDD43NDHEHLLE6QDKPEH7WPJTQQ, seg_pos={0 0}, upload=480766]: Done uploading piece. err: <nil> Dec 28 11:46:22.988391858 (548h17m22.859474893s) [noise=true , address=45.91.116.5:4299 , piece_id=JD2BZZSWEGA72K52HTFHYGSEF23CNFTS76UFCZIQNV7ARFPVOHOQ, seg_pos={0 0}, upload=480766]: Done uploading piece. err: <nil> Dec 28 11:46:23.148074480 (548h17m23.019157525s) [noise=true , address=77.90.31.19:4844 , piece_id=U3EQIA6ZIFDAL5PKCZ6ARGX3H5QVSG4WJUT7JDDTQ26GOOKPK6XA, seg_pos={0 0}, upload=480766]: Done uploading piece. err: <nil> Dec 28 11:46:23.468790975 (548h17m23.33987401s) [noise=true , address=77.90.31.15:4700 , piece_id=BQH2BFJE7CETYFEN46RIFKOWN5HZPU42GBOQJB3H5ZQYCUOODPLQ, seg_pos={0 0}, upload=480766]: Done uploading piece. err: <nil> Dec 28 11:46:23.468809029 (548h17m23.339892064s) [seg_pos={0 0}, upload=480766]: Segment reached optimal threshold of 13 pieces. Dec 28 11:46:23.468995866 (548h17m23.340078911s) [noise=true , address=45.91.116.9:4262 , piece_id=ZVY5NDSESC6E7VKVTBEFQO2KS2C2YGOW6MHUMCIR5VYGRDSOSYAA, seg_pos={0 0}, upload=480766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 28 11:46:23.469006375 (548h17m23.34008941s) [noise=true , address=45.91.116.9:4262 , piece_id=ZVY5NDSESC6E7VKVTBEFQO2KS2C2YGOW6MHUMCIR5VYGRDSOSYAA, seg_pos={0 0}, upload=480766]: Long tail cancellation due to optimal threshold Dec 28 11:46:23.469025392 (548h17m23.340108427s) [noise=true , address=45.91.116.6:4975 , piece_id=557JHRPZ5T22KX3CY7YYYKKDL5KSGBSJEKYASHNIDWWT3ZCNYTHQ, seg_pos={0 0}, upload=480766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:38888->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:38888->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38888->45.91.116.6:4975: use of closed network connection Dec 28 11:46:23.469033116 (548h17m23.340116151s) [noise=true , address=45.91.116.6:4975 , piece_id=557JHRPZ5T22KX3CY7YYYKKDL5KSGBSJEKYASHNIDWWT3ZCNYTHQ, seg_pos={0 0}, upload=480766]: Long tail cancellation due to optimal threshold Dec 28 11:46:23.469103651 (548h17m23.340186686s) [noise=true , address=45.91.116.14:4190 , piece_id=IVFZHG7EGMOL6YV2REX7UN5YPP6QJZZZHHBK5ODHW64XLYGQ434Q, seg_pos={0 0}, upload=480766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:44696->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:44696->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44696->45.91.116.14:4190: use of closed network connection Dec 28 11:46:23.469110203 (548h17m23.340193238s) [noise=true , address=45.91.116.14:4190 , piece_id=IVFZHG7EGMOL6YV2REX7UN5YPP6QJZZZHHBK5ODHW64XLYGQ434Q, seg_pos={0 0}, upload=480766]: Long tail cancellation due to optimal threshold Dec 28 11:46:23.469234179 (548h17m23.340317215s) [seg_pos={0 0}, upload=480766]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:46:23.469300615 (548h17m23.34038365s) [upload=480766]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:46:23.469305505 (548h17m23.34038854s) [upload=480766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:46:23.763258900 (548h17m23.634341945s) [upload=480766]: Done uploading segments... Dec 28 11:46:23.763286763 (548h17m23.634369808s) [upload=480766]: Upload finished. err: <nil> Duration: 30.003364689s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [XX.XX..XXX...X.X....X..XXXX..X.X.XX.....X.X.X..X.XXX..X....X] duration: 5.000759637s trace: https://jaeger.dev.storj.io/trace/3b9025dcf28308d8 FAIL (2025-12-28 11:46:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000782842s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X........X...........X..X..................] duration: 37.755365145s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:46:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 37.755392467s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XX.X..XX..X.......X.......................XX....X.....XX...X] duration: 30.003849236s trace: https://jaeger.dev.storj.io/trace/22f25f0412f5f041 FAIL (2025-12-28 11:45:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KH6ZFJUTXDNFLUBZUAH7KQN6DTTZAXFCVC4PJB2NPDIPZZJOYSQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: 6SRKGQ2A3KEVHUPJAZ6IAEBTHUAN7PAKRMXS2HZC5XSBHWZ37GGQ) context deadline exceeded ======LOGS====== 8 11:45:26.428856174 (548h16m26.299939209s) [upload=480751]: Starting upload Dec 28 11:45:26.428915698 (548h16m26.299998733s) [upload=480751]: Done starting upload Dec 28 11:45:26.429048220 (548h16m26.300131265s) [upload=480751]: Uploading segments... Dec 28 11:45:26.429130787 (548h16m26.300213832s) [upload=480751]: Got next segment. Inline: false Dec 28 11:45:26.429165443 (548h16m26.300248488s) [upload=480751]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:45:26.429171845 (548h16m26.30025489s) [upload=480751]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:45:26.674547963 (548h16m26.545631008s) [seg_pos={0 0}, upload=480751]: Begin upload segment... Dec 28 11:45:26.674590904 (548h16m26.545673949s) [seg_pos={0 0}, upload=480751]: Done begin upload segment. Dec 28 11:45:26.674626802 (548h16m26.545709847s) [noise=true , address=145.79.227.14:4839 , piece_id=XVP4SULUEKSGVS44IP7GS7CJGE3YOQEO7GV3C45542DNFBR5HJMA, seg_pos={0 0}, upload=480751]: Uploading piece... Dec 28 11:45:26.674653634 (548h16m26.545736679s) [noise=true , address=145.79.227.36:4518 , piece_id=3QU3ARAM5WXOYF2PKEUICUIONX7Q4XM2374ONF4XQL2YVNVJJU2A, seg_pos={0 0}, upload=480751]: Uploading piece... Dec 28 11:45:26.675157443 (548h16m26.546240488s) [noise=true , address=145.79.227.45:4229 , piece_id=FSTUKYNRGEEUBFDYE463JQ5TF4ECE2JLPVXKJDVXNL4MQEV2TOTA, seg_pos={0 0}, upload=480751]: Uploading piece... Dec 28 11:45:26.675328809 (548h16m26.546411854s) [noise=true , address=145.79.227.11:4730 , piece_id=UCG4MZS72R7LXNSBH53CTPLATYLMLCCVLKXPUN7G2SUP76TXKJKA, seg_pos={0 0}, upload=480751]: Uploading piece... Dec 28 11:45:26.675586941 (548h16m26.546669986s) [noise=true , address=145.79.227.2:4114 , piece_id=FNHOB3EH7L52IKVAB4ERHEZXL2KJHX6E23TG74T2WMXG2CVF5K6A, seg_pos={0 0}, upload=480751]: Uploading piece... Dec 28 11:45:26.675767094 (548h16m26.546850139s) [noise=true , address=45.91.116.7:4012 , piece_id=6SRKGQ2A3KEVHUPJAZ6IAEBTHUAN7PAKRMXS2HZC5XSBHWZ37GGQ, seg_pos={0 0}, upload=480751]: Uploading piece... Dec 28 11:45:26.675902282 (548h16m26.546985327s) [noise=true , address=145.79.227.29:4941 , piece_id=7F23GTYYBCHN6HNPO7S3L2MBDU7TXMYSWDWIZJ5H6VSIOUZFOMRA, seg_pos={0 0}, upload=480751]: Uploading piece... Dec 28 11:45:26.676232199 (548h16m26.547315244s) [noise=true , address=45.91.116.12:4228 , piece_id=GAYMTS2JUNUP7E4OK5FC5FMK4F5SF5VDTQOWQGTE4YCVAJF2HC6Q, seg_pos={0 0}, upload=480751]: Uploading piece... Dec 28 11:45:26.676263309 (548h16m26.547346344s) [noise=true , address=45.91.116.6:4975 , piece_id=D2KMHLNJK4ZY5BJS4WXX4QA3ZBZIJYSQXOGGHSLEVRDPTB2C7QNQ, seg_pos={0 0}, upload=480751]: Uploading piece... Dec 28 11:45:26.676702856 (548h16m26.547785901s) [noise=true , address=45.91.116.4:4940 , piece_id=4ECTXE364E7XKTWW22Q5GOQT72L6IQL2RP77LCMXQA7WPCSNWOWQ, seg_pos={0 0}, upload=480751]: Uploading piece... Dec 28 11:45:26.677232444 (548h16m26.548315479s) [noise=true , address=45.91.116.9:4261 , piece_id=3KH6ZFJUTXDNFLUBZUAH7KQN6DTTZAXFCVC4PJB2NPDIPZZJOYSQ, seg_pos={0 0}, upload=480751]: Uploading piece... Dec 28 11:45:26.677710354 (548h16m26.5487934s) [noise=true , address=145.79.227.38:4978 , piece_id=P6J5U4V7HLN5YMYJ2YW5TT4ZRDKYADQWO2JAZQSDHPBDIWQGLNTQ, seg_pos={0 0}, upload=480751]: Uploading piece... Dec 28 11:45:26.678461474 (548h16m26.549544509s) [noise=true , address=145.79.227.52:4446 , piece_id=FFB5JCW4HMUZ2Q6GUC4YB3S5QZPO77FJFD3HV73OT2VOVTABC3XQ, seg_pos={0 0}, upload=480751]: Uploading piece... Dec 28 11:45:26.680039339 (548h16m26.551122374s) [noise=true , address=145.79.227.27:4691 , piece_id=JZKSPIOID7AVQL4XQYKFQX4PZDUYHE476PSIDPL7HIDGER7KNE3A, seg_pos={0 0}, upload=480751]: Uploading piece... Dec 28 11:45:26.683988759 (548h16m26.555071794s) [noise=true , address=45.91.116.10:4081 , piece_id=RF6WWGCGSBZZNSDGPCOBPIXIZYTZQ24CU3G2D4U53RGIO7ICA5UA, seg_pos={0 0}, upload=480751]: Uploading piece... Dec 28 11:45:26.686844066 (548h16m26.557927111s) [noise=true , address=145.79.227.6:4258 , piece_id=BVFND7AG7G7PMKSSCWNUNJZEDIGORHICBZET34LN4ZN3O7JOIGHA, seg_pos={0 0}, upload=480751]: Uploading piece... Dec 28 11:45:28.300259597 (548h16m28.171342642s) [upload=480751]: Next returned nil segment Dec 28 11:45:28.300286909 (548h16m28.171369944s) [upload=480751]: Waiting for error group managing segments... Dec 28 11:45:28.678530135 (548h16m28.54961317s) [noise=true , address=145.79.227.45:4229 , piece_id=FSTUKYNRGEEUBFDYE463JQ5TF4ECE2JLPVXKJDVXNL4MQEV2TOTA, seg_pos={0 0}, upload=480751]: Done uploading piece. err: <nil> Dec 28 11:45:28.681658552 (548h16m28.552741587s) [noise=true , address=145.79.227.11:4730 , piece_id=UCG4MZS72R7LXNSBH53CTPLATYLMLCCVLKXPUN7G2SUP76TXKJKA, seg_pos={0 0}, upload=480751]: Done uploading piece. err: <nil> Dec 28 11:45:28.689562703 (548h16m28.560645748s) [noise=true , address=145.79.227.52:4446 , piece_id=FFB5JCW4HMUZ2Q6GUC4YB3S5QZPO77FJFD3HV73OT2VOVTABC3XQ, seg_pos={0 0}, upload=480751]: Done uploading piece. err: <nil> Dec 28 11:45:28.690419003 (548h16m28.561502048s) [noise=true , address=145.79.227.36:4518 , piece_id=3QU3ARAM5WXOYF2PKEUICUIONX7Q4XM2374ONF4XQL2YVNVJJU2A, seg_pos={0 0}, upload=480751]: Done uploading piece. err: <nil> Dec 28 11:45:28.690509956 (548h16m28.561593001s) [noise=true , address=145.79.227.6:4258 , piece_id=BVFND7AG7G7PMKSSCWNUNJZEDIGORHICBZET34LN4ZN3O7JOIGHA, seg_pos={0 0}, upload=480751]: Done uploading piece. err: <nil> Dec 28 11:45:28.691024817 (548h16m28.562107852s) [noise=true , address=145.79.227.29:4941 , piece_id=7F23GTYYBCHN6HNPO7S3L2MBDU7TXMYSWDWIZJ5H6VSIOUZFOMRA, seg_pos={0 0}, upload=480751]: Done uploading piece. err: <nil> Dec 28 11:45:28.691082436 (548h16m28.562165481s) [noise=true , address=145.79.227.27:4691 , piece_id=JZKSPIOID7AVQL4XQYKFQX4PZDUYHE476PSIDPL7HIDGER7KNE3A, seg_pos={0 0}, upload=480751]: Done uploading piece. err: <nil> Dec 28 11:45:28.694298591 (548h16m28.565381636s) [noise=true , address=145.79.227.38:4978 , piece_id=P6J5U4V7HLN5YMYJ2YW5TT4ZRDKYADQWO2JAZQSDHPBDIWQGLNTQ, seg_pos={0 0}, upload=480751]: Done uploading piece. err: <nil> Dec 28 11:45:28.703576306 (548h16m28.574659352s) [noise=true , address=145.79.227.2:4114 , piece_id=FNHOB3EH7L52IKVAB4ERHEZXL2KJHX6E23TG74T2WMXG2CVF5K6A, seg_pos={0 0}, upload=480751]: Done uploading piece. err: <nil> Dec 28 11:45:28.708918630 (548h16m28.580001675s) [noise=true , address=145.79.227.14:4839 , piece_id=XVP4SULUEKSGVS44IP7GS7CJGE3YOQEO7GV3C45542DNFBR5HJMA, seg_pos={0 0}, upload=480751]: Done uploading piece. err: <nil> Dec 28 11:45:28.936158190 (548h16m28.807241225s) [noise=true , address=45.91.116.9:4261 , piece_id=3KH6ZFJUTXDNFLUBZUAH7KQN6DTTZAXFCVC4PJB2NPDIPZZJOYSQ, seg_pos={0 0}, upload=480751]: Done uploading piece. err: <nil> Dec 28 11:45:28.945830819 (548h16m28.816913854s) [noise=true , address=45.91.116.7:4012 , piece_id=6SRKGQ2A3KEVHUPJAZ6IAEBTHUAN7PAKRMXS2HZC5XSBHWZ37GGQ, seg_pos={0 0}, upload=480751]: Done uploading piece. err: <nil> Dec 28 11:45:28.962328733 (548h16m28.833411778s) [noise=true , address=45.91.116.10:4081 , piece_id=RF6WWGCGSBZZNSDGPCOBPIXIZYTZQ24CU3G2D4U53RGIO7ICA5UA, seg_pos={0 0}, upload=480751]: Done uploading piece. err: <nil> Dec 28 11:45:28.962354903 (548h16m28.833437938s) [seg_pos={0 0}, upload=480751]: Segment reached optimal threshold of 13 pieces. Dec 28 11:45:28.962771526 (548h16m28.833854571s) [noise=true , address=45.91.116.6:4975 , piece_id=D2KMHLNJK4ZY5BJS4WXX4QA3ZBZIJYSQXOGGHSLEVRDPTB2C7QNQ, seg_pos={0 0}, upload=480751]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 28 11:45:28.962812564 (548h16m28.833895599s) [noise=true , address=45.91.116.6:4975 , piece_id=D2KMHLNJK4ZY5BJS4WXX4QA3ZBZIJYSQXOGGHSLEVRDPTB2C7QNQ, seg_pos={0 0}, upload=480751]: Long tail cancellation due to optimal threshold Dec 28 11:45:28.962906243 (548h16m28.833989318s) [noise=true , address=45.91.116.4:4940 , piece_id=4ECTXE364E7XKTWW22Q5GOQT72L6IQL2RP77LCMXQA7WPCSNWOWQ, seg_pos={0 0}, upload=480751]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:59670->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:59670->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59670->45.91.116.4:4940: use of closed network connection Dec 28 11:45:28.962970896 (548h16m28.834053931s) [noise=true , address=45.91.116.4:4940 , piece_id=4ECTXE364E7XKTWW22Q5GOQT72L6IQL2RP77LCMXQA7WPCSNWOWQ, seg_pos={0 0}, upload=480751]: Long tail cancellation due to optimal threshold Dec 28 11:45:28.963240028 (548h16m28.834323073s) [noise=true , address=45.91.116.12:4228 , piece_id=GAYMTS2JUNUP7E4OK5FC5FMK4F5SF5VDTQOWQGTE4YCVAJF2HC6Q, seg_pos={0 0}, upload=480751]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 11:45:28.963251230 (548h16m28.834334265s) [noise=true , address=45.91.116.12:4228 , piece_id=GAYMTS2JUNUP7E4OK5FC5FMK4F5SF5VDTQOWQGTE4YCVAJF2HC6Q, seg_pos={0 0}, upload=480751]: Long tail cancellation due to optimal threshold Dec 28 11:45:28.963298069 (548h16m28.834381114s) [seg_pos={0 0}, upload=480751]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:45:28.963452903 (548h16m28.834535948s) [upload=480751]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:45:28.963481818 (548h16m28.834564863s) [upload=480751]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:45:29.251161687 (548h16m29.122244722s) [upload=480751]: Done uploading segments... Dec 28 11:45:29.251183319 (548h16m29.122266364s) [upload=480751]: Upload finished. err: <nil> Duration: 30.003875545s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [.XX.XX..XXX...X.X....X..XXXX..X.X.XX.....X.X.X..X.XXX..X....] duration: 3.508726326s trace: https://jaeger.dev.storj.io/trace/2c87d4bacf984303 FAIL (2025-12-28 11:45:52 UTC) orbiter warning: context deadline exceeded Duration: 3.508755361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XX.X..XX..X.......X.......................XX....X.....XX...] duration: 5.890249901s trace: https://jaeger.dev.storj.io/trace/14a25f94a1912b1c FAIL (2025-12-28 11:45:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: CTPWG4XCW43M7S3I5PNKI6UJ3HC52E7XBKX2OM4VMGX6C5XBDRMQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: GHCMJJP3L7RAPPH6ZC3ASYV5LO4HA2BBV5RKINQMA7JBNSI2LDJQ) context deadline exceeded ======LOGS====== Dec 28 11:43:03.153674892 (548h14m3.024757937s) [upload=480719]: Starting upload Dec 28 11:43:03.153770594 (548h14m3.024853659s) [upload=480719]: Done starting upload Dec 28 11:43:03.153828585 (548h14m3.02491163s) [upload=480719]: Uploading segments... Dec 28 11:43:03.154070375 (548h14m3.02515343s) [upload=480719]: Got next segment. Inline: false Dec 28 11:43:03.154105502 (548h14m3.025188547s) [upload=480719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:43:03.154111974 (548h14m3.025195019s) [upload=480719]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:43:03.399935160 (548h14m3.271018195s) [seg_pos={0 0}, upload=480719]: Begin upload segment... Dec 28 11:43:03.399997618 (548h14m3.271080663s) [seg_pos={0 0}, upload=480719]: Done begin upload segment. Dec 28 11:43:03.400037966 (548h14m3.271121011s) [noise=true , address=145.79.227.42:4122 , piece_id=VHK6G5SAXYYJT4KRHAVFJCR3OKX7E3H5WVBM5F77WL7EY3OCKLYQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.400141453 (548h14m3.271224498s) [noise=true , address=45.91.116.8:4045 , piece_id=MB76HIY2R4WSBS5S7Q5AVUT7XHXJS7VQ5IZF3VB6O4OUKFDYHXIA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.400848770 (548h14m3.271931815s) [noise=true , address=145.79.227.52:4447 , piece_id=UCQBBKHI5GRZQJH56XJHSN3LOTMGUGHYDRL5XUMY3FAVQTD7D3AA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.401382976 (548h14m3.272466021s) [noise=true , address=45.91.116.6:4974 , piece_id=OI5PSHBDQG66DQFOX3RCJIVMPGTXBFMYYCMW2ET2TO2I7QS3NJLA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.401873961 (548h14m3.272957016s) [noise=true , address=145.79.227.51:4410 , piece_id=NTRMOWDVB62NZKJ5P5COGJH3GUT7NL3ZJD76NKB25IRZ53LJQLDA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.402033185 (548h14m3.27311623s) [noise=true , address=145.79.227.19:4079 , piece_id=I4GSOARSSTTUIU3JZVHMZMQ6B52SS5GHASLBLFCWEHTIOFLD7EVQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.402464967 (548h14m3.273548022s) [noise=true , address=145.79.227.45:4229 , piece_id=EBHCBS24EV65ODROABQUZFA3C347X7LBQRHSNE4UVELCGCW2EYLA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.402764578 (548h14m3.273847623s) [noise=true , address=145.79.227.41:4087 , piece_id=6IB2AW5NCG7ALSOOIJRKENJYI4GAEE72N6RE6J24DOXVFZF3NKOA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.403065000 (548h14m3.274148045s) [noise=true , address=77.90.31.21:4880 , piece_id=6URQMPZU4GZ7Y4JCAMC7TPIXEB7PKSDFRHH72ZY7AZNCT7J234OA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.403300137 (548h14m3.274383182s) [noise=true , address=45.91.116.4:4939 , piece_id=EJFNFQZKNZK723PJBITR75DYCLBBJWRTIMWHXYMMIEK74OVBI6NA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.403526298 (548h14m3.274609343s) [noise=true , address=145.79.227.26:4655 , piece_id=L2M4QAUCAORBJ6BWUHJGFFEUSTIOKPRDTEFH4CQLPEVBNTJJWAJA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.403855565 (548h14m3.27493861s) [noise=true , address=145.79.227.13:4803 , piece_id=XBVD7MQEK7ZQB7YYKH6CVB36ATQPEU43RMKTI75IG5SDYQRAPE2Q, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.404270605 (548h14m3.27535365s) [noise=true , address=77.90.31.12:4589 , piece_id=CTPWG4XCW43M7S3I5PNKI6UJ3HC52E7XBKX2OM4VMGX6C5XBDRMQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.404406113 (548h14m3.275489158s) [noise=true , address=145.79.227.37:4567 , piece_id=C6PJTJS54NJJW2ZU2OCA5TVXCHEMQZE7O7CQ7CES5SLVOQKK5SQQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.404714901 (548h14m3.275797946s) [noise=true , address=45.91.116.14:4190 , piece_id=GHCMJJP3L7RAPPH6ZC3ASYV5LO4HA2BBV5RKINQMA7JBNSI2LDJQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.404976981 (548h14m3.276060026s) [noise=true , address=45.91.116.13:4153 , piece_id=4OMS7INO5VHFQEYFOSPPPP3JQMS7MRAF66YTYUEDYUYNT756EDGQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:05.111835332 (548h14m4.982918367s) [upload=480719]: Next returned nil segment Dec 28 11:43:05.111852295 (548h14m4.98293533s) [upload=480719]: Waiting for error group managing segments... Dec 28 11:43:05.399405133 (548h14m5.270488178s) [noise=true , address=145.79.227.19:4079 , piece_id=I4GSOARSSTTUIU3JZVHMZMQ6B52SS5GHASLBLFCWEHTIOFLD7EVQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.399556911 (548h14m5.270639957s) [noise=true , address=145.79.227.51:4410 , piece_id=NTRMOWDVB62NZKJ5P5COGJH3GUT7NL3ZJD76NKB25IRZ53LJQLDA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.414139549 (548h14m5.285222595s) [noise=true , address=145.79.227.45:4229 , piece_id=EBHCBS24EV65ODROABQUZFA3C347X7LBQRHSNE4UVELCGCW2EYLA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.415887087 (548h14m5.286970132s) [noise=true , address=145.79.227.37:4567 , piece_id=C6PJTJS54NJJW2ZU2OCA5TVXCHEMQZE7O7CQ7CES5SLVOQKK5SQQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.416590606 (548h14m5.287673651s) [noise=true , address=145.79.227.42:4122 , piece_id=VHK6G5SAXYYJT4KRHAVFJCR3OKX7E3H5WVBM5F77WL7EY3OCKLYQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.422948263 (548h14m5.294031298s) [noise=true , address=145.79.227.52:4447 , piece_id=UCQBBKHI5GRZQJH56XJHSN3LOTMGUGHYDRL5XUMY3FAVQTD7D3AA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.429791414 (548h14m5.300874449s) [noise=true , address=145.79.227.41:4087 , piece_id=6IB2AW5NCG7ALSOOIJRKENJYI4GAEE72N6RE6J24DOXVFZF3NKOA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.431689648 (548h14m5.302772683s) [noise=true , address=145.79.227.13:4803 , piece_id=XBVD7MQEK7ZQB7YYKH6CVB36ATQPEU43RMKTI75IG5SDYQRAPE2Q, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.433491559 (548h14m5.304574604s) [noise=true , address=145.79.227.26:4655 , piece_id=L2M4QAUCAORBJ6BWUHJGFFEUSTIOKPRDTEFH4CQLPEVBNTJJWAJA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.644963917 (548h14m5.516046962s) [noise=true , address=45.91.116.6:4974 , piece_id=OI5PSHBDQG66DQFOX3RCJIVMPGTXBFMYYCMW2ET2TO2I7QS3NJLA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.671643319 (548h14m5.542726364s) [noise=true , address=77.90.31.12:4589 , piece_id=CTPWG4XCW43M7S3I5PNKI6UJ3HC52E7XBKX2OM4VMGX6C5XBDRMQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.674846027 (548h14m5.545929062s) [noise=true , address=45.91.116.4:4939 , piece_id=EJFNFQZKNZK723PJBITR75DYCLBBJWRTIMWHXYMMIEK74OVBI6NA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:06.342722614 (548h14m6.213805659s) [noise=true , address=45.91.116.14:4190 , piece_id=GHCMJJP3L7RAPPH6ZC3ASYV5LO4HA2BBV5RKINQMA7JBNSI2LDJQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:06.342746930 (548h14m6.213829965s) [seg_pos={0 0}, upload=480719]: Segment reached optimal threshold of 13 pieces. Dec 28 11:43:06.342939807 (548h14m6.214022842s) [noise=true , address=77.90.31.21:4880 , piece_id=6URQMPZU4GZ7Y4JCAMC7TPIXEB7PKSDFRHH72ZY7AZNCT7J234OA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 28 11:43:06.342946451 (548h14m6.214029496s) [noise=true , address=77.90.31.21:4880 , piece_id=6URQMPZU4GZ7Y4JCAMC7TPIXEB7PKSDFRHH72ZY7AZNCT7J234OA, seg_pos={0 0}, upload=480719]: Long tail cancellation due to optimal threshold Dec 28 11:43:06.343216424 (548h14m6.214299459s) [noise=true , address=45.91.116.8:4045 , piece_id=MB76HIY2R4WSBS5S7Q5AVUT7XHXJS7VQ5IZF3VB6O4OUKFDYHXIA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 11:43:06.343222667 (548h14m6.214305702s) [noise=true , address=45.91.116.8:4045 , piece_id=MB76HIY2R4WSBS5S7Q5AVUT7XHXJS7VQ5IZF3VB6O4OUKFDYHXIA, seg_pos={0 0}, upload=480719]: Long tail cancellation due to optimal threshold Dec 28 11:43:06.343341052 (548h14m6.214424097s) [noise=true , address=45.91.116.13:4153 , piece_id=4OMS7INO5VHFQEYFOSPPPP3JQMS7MRAF66YTYUEDYUYNT756EDGQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 28 11:43:06.343346862 (548h14m6.214429908s) [noise=true , address=45.91.116.13:4153 , piece_id=4OMS7INO5VHFQEYFOSPPPP3JQMS7MRAF66YTYUEDYUYNT756EDGQ, seg_pos={0 0}, upload=480719]: Long tail cancellation due to optimal threshold Dec 28 11:43:06.343365298 (548h14m6.214448333s) [seg_pos={0 0}, upload=480719]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:43:06.343439409 (548h14m6.214522454s) [upload=480719]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:43:06.343445400 (548h14m6.214528445s) [upload=480719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:43:06.644189937 (548h14m6.515272982s) [upload=480719]: Done uploading segments... Dec 28 11:43:06.644217320 (548h14m6.515300365s) [upload=480719]: Upload finished. err: <nil> Duration: 5.890281101s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X........X...........X..X.................] duration: 42.264803635s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:45:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 42.264820757s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [..XX.XX..XXX...X.X....X..XXXX..X.X.XX.....X.X.X..X.XXX..X...] duration: 3.465562114s trace: https://jaeger.dev.storj.io/trace/2c87d4bacf984303 FAIL (2025-12-28 11:45:06 UTC) orbiter warning: context deadline exceeded Duration: 3.46559131s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X........X...........X..X................] duration: 38.7600779s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:44:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 38.760109923s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.XX.X..XX..X.......X.......................XX....X.....XX..] duration: 6.032453577s trace: https://jaeger.dev.storj.io/trace/14a25f94a1912b1c FAIL (2025-12-28 11:44:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: CTPWG4XCW43M7S3I5PNKI6UJ3HC52E7XBKX2OM4VMGX6C5XBDRMQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: GHCMJJP3L7RAPPH6ZC3ASYV5LO4HA2BBV5RKINQMA7JBNSI2LDJQ) context deadline exceeded ======LOGS====== Dec 28 11:43:03.153674892 (548h14m3.024757937s) [upload=480719]: Starting upload Dec 28 11:43:03.153770594 (548h14m3.024853659s) [upload=480719]: Done starting upload Dec 28 11:43:03.153828585 (548h14m3.02491163s) [upload=480719]: Uploading segments... Dec 28 11:43:03.154070375 (548h14m3.02515343s) [upload=480719]: Got next segment. Inline: false Dec 28 11:43:03.154105502 (548h14m3.025188547s) [upload=480719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:43:03.154111974 (548h14m3.025195019s) [upload=480719]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:43:03.399935160 (548h14m3.271018195s) [seg_pos={0 0}, upload=480719]: Begin upload segment... Dec 28 11:43:03.399997618 (548h14m3.271080663s) [seg_pos={0 0}, upload=480719]: Done begin upload segment. Dec 28 11:43:03.400037966 (548h14m3.271121011s) [noise=true , address=145.79.227.42:4122 , piece_id=VHK6G5SAXYYJT4KRHAVFJCR3OKX7E3H5WVBM5F77WL7EY3OCKLYQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.400141453 (548h14m3.271224498s) [noise=true , address=45.91.116.8:4045 , piece_id=MB76HIY2R4WSBS5S7Q5AVUT7XHXJS7VQ5IZF3VB6O4OUKFDYHXIA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.400848770 (548h14m3.271931815s) [noise=true , address=145.79.227.52:4447 , piece_id=UCQBBKHI5GRZQJH56XJHSN3LOTMGUGHYDRL5XUMY3FAVQTD7D3AA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.401382976 (548h14m3.272466021s) [noise=true , address=45.91.116.6:4974 , piece_id=OI5PSHBDQG66DQFOX3RCJIVMPGTXBFMYYCMW2ET2TO2I7QS3NJLA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.401873961 (548h14m3.272957016s) [noise=true , address=145.79.227.51:4410 , piece_id=NTRMOWDVB62NZKJ5P5COGJH3GUT7NL3ZJD76NKB25IRZ53LJQLDA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.402033185 (548h14m3.27311623s) [noise=true , address=145.79.227.19:4079 , piece_id=I4GSOARSSTTUIU3JZVHMZMQ6B52SS5GHASLBLFCWEHTIOFLD7EVQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.402464967 (548h14m3.273548022s) [noise=true , address=145.79.227.45:4229 , piece_id=EBHCBS24EV65ODROABQUZFA3C347X7LBQRHSNE4UVELCGCW2EYLA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.402764578 (548h14m3.273847623s) [noise=true , address=145.79.227.41:4087 , piece_id=6IB2AW5NCG7ALSOOIJRKENJYI4GAEE72N6RE6J24DOXVFZF3NKOA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.403065000 (548h14m3.274148045s) [noise=true , address=77.90.31.21:4880 , piece_id=6URQMPZU4GZ7Y4JCAMC7TPIXEB7PKSDFRHH72ZY7AZNCT7J234OA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.403300137 (548h14m3.274383182s) [noise=true , address=45.91.116.4:4939 , piece_id=EJFNFQZKNZK723PJBITR75DYCLBBJWRTIMWHXYMMIEK74OVBI6NA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.403526298 (548h14m3.274609343s) [noise=true , address=145.79.227.26:4655 , piece_id=L2M4QAUCAORBJ6BWUHJGFFEUSTIOKPRDTEFH4CQLPEVBNTJJWAJA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.403855565 (548h14m3.27493861s) [noise=true , address=145.79.227.13:4803 , piece_id=XBVD7MQEK7ZQB7YYKH6CVB36ATQPEU43RMKTI75IG5SDYQRAPE2Q, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.404270605 (548h14m3.27535365s) [noise=true , address=77.90.31.12:4589 , piece_id=CTPWG4XCW43M7S3I5PNKI6UJ3HC52E7XBKX2OM4VMGX6C5XBDRMQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.404406113 (548h14m3.275489158s) [noise=true , address=145.79.227.37:4567 , piece_id=C6PJTJS54NJJW2ZU2OCA5TVXCHEMQZE7O7CQ7CES5SLVOQKK5SQQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.404714901 (548h14m3.275797946s) [noise=true , address=45.91.116.14:4190 , piece_id=GHCMJJP3L7RAPPH6ZC3ASYV5LO4HA2BBV5RKINQMA7JBNSI2LDJQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.404976981 (548h14m3.276060026s) [noise=true , address=45.91.116.13:4153 , piece_id=4OMS7INO5VHFQEYFOSPPPP3JQMS7MRAF66YTYUEDYUYNT756EDGQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:05.111835332 (548h14m4.982918367s) [upload=480719]: Next returned nil segment Dec 28 11:43:05.111852295 (548h14m4.98293533s) [upload=480719]: Waiting for error group managing segments... Dec 28 11:43:05.399405133 (548h14m5.270488178s) [noise=true , address=145.79.227.19:4079 , piece_id=I4GSOARSSTTUIU3JZVHMZMQ6B52SS5GHASLBLFCWEHTIOFLD7EVQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.399556911 (548h14m5.270639957s) [noise=true , address=145.79.227.51:4410 , piece_id=NTRMOWDVB62NZKJ5P5COGJH3GUT7NL3ZJD76NKB25IRZ53LJQLDA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.414139549 (548h14m5.285222595s) [noise=true , address=145.79.227.45:4229 , piece_id=EBHCBS24EV65ODROABQUZFA3C347X7LBQRHSNE4UVELCGCW2EYLA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.415887087 (548h14m5.286970132s) [noise=true , address=145.79.227.37:4567 , piece_id=C6PJTJS54NJJW2ZU2OCA5TVXCHEMQZE7O7CQ7CES5SLVOQKK5SQQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.416590606 (548h14m5.287673651s) [noise=true , address=145.79.227.42:4122 , piece_id=VHK6G5SAXYYJT4KRHAVFJCR3OKX7E3H5WVBM5F77WL7EY3OCKLYQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.422948263 (548h14m5.294031298s) [noise=true , address=145.79.227.52:4447 , piece_id=UCQBBKHI5GRZQJH56XJHSN3LOTMGUGHYDRL5XUMY3FAVQTD7D3AA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.429791414 (548h14m5.300874449s) [noise=true , address=145.79.227.41:4087 , piece_id=6IB2AW5NCG7ALSOOIJRKENJYI4GAEE72N6RE6J24DOXVFZF3NKOA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.431689648 (548h14m5.302772683s) [noise=true , address=145.79.227.13:4803 , piece_id=XBVD7MQEK7ZQB7YYKH6CVB36ATQPEU43RMKTI75IG5SDYQRAPE2Q, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.433491559 (548h14m5.304574604s) [noise=true , address=145.79.227.26:4655 , piece_id=L2M4QAUCAORBJ6BWUHJGFFEUSTIOKPRDTEFH4CQLPEVBNTJJWAJA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.644963917 (548h14m5.516046962s) [noise=true , address=45.91.116.6:4974 , piece_id=OI5PSHBDQG66DQFOX3RCJIVMPGTXBFMYYCMW2ET2TO2I7QS3NJLA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.671643319 (548h14m5.542726364s) [noise=true , address=77.90.31.12:4589 , piece_id=CTPWG4XCW43M7S3I5PNKI6UJ3HC52E7XBKX2OM4VMGX6C5XBDRMQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.674846027 (548h14m5.545929062s) [noise=true , address=45.91.116.4:4939 , piece_id=EJFNFQZKNZK723PJBITR75DYCLBBJWRTIMWHXYMMIEK74OVBI6NA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:06.342722614 (548h14m6.213805659s) [noise=true , address=45.91.116.14:4190 , piece_id=GHCMJJP3L7RAPPH6ZC3ASYV5LO4HA2BBV5RKINQMA7JBNSI2LDJQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:06.342746930 (548h14m6.213829965s) [seg_pos={0 0}, upload=480719]: Segment reached optimal threshold of 13 pieces. Dec 28 11:43:06.342939807 (548h14m6.214022842s) [noise=true , address=77.90.31.21:4880 , piece_id=6URQMPZU4GZ7Y4JCAMC7TPIXEB7PKSDFRHH72ZY7AZNCT7J234OA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 28 11:43:06.342946451 (548h14m6.214029496s) [noise=true , address=77.90.31.21:4880 , piece_id=6URQMPZU4GZ7Y4JCAMC7TPIXEB7PKSDFRHH72ZY7AZNCT7J234OA, seg_pos={0 0}, upload=480719]: Long tail cancellation due to optimal threshold Dec 28 11:43:06.343216424 (548h14m6.214299459s) [noise=true , address=45.91.116.8:4045 , piece_id=MB76HIY2R4WSBS5S7Q5AVUT7XHXJS7VQ5IZF3VB6O4OUKFDYHXIA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 11:43:06.343222667 (548h14m6.214305702s) [noise=true , address=45.91.116.8:4045 , piece_id=MB76HIY2R4WSBS5S7Q5AVUT7XHXJS7VQ5IZF3VB6O4OUKFDYHXIA, seg_pos={0 0}, upload=480719]: Long tail cancellation due to optimal threshold Dec 28 11:43:06.343341052 (548h14m6.214424097s) [noise=true , address=45.91.116.13:4153 , piece_id=4OMS7INO5VHFQEYFOSPPPP3JQMS7MRAF66YTYUEDYUYNT756EDGQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 28 11:43:06.343346862 (548h14m6.214429908s) [noise=true , address=45.91.116.13:4153 , piece_id=4OMS7INO5VHFQEYFOSPPPP3JQMS7MRAF66YTYUEDYUYNT756EDGQ, seg_pos={0 0}, upload=480719]: Long tail cancellation due to optimal threshold Dec 28 11:43:06.343365298 (548h14m6.214448333s) [seg_pos={0 0}, upload=480719]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:43:06.343439409 (548h14m6.214522454s) [upload=480719]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:43:06.343445400 (548h14m6.214528445s) [upload=480719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:43:06.644189937 (548h14m6.515272982s) [upload=480719]: Done uploading segments... Dec 28 11:43:06.644217320 (548h14m6.515300365s) [upload=480719]: Upload finished. err: <nil> Duration: 6.032478976s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [...XX.XX..XXX...X.X....X..XXXX..X.X.XX.....X.X.X..X.XXX..X..] duration: 4.850649795s trace: https://jaeger.dev.storj.io/trace/2c87d4bacf984303 FAIL (2025-12-28 11:44:14 UTC) orbiter warning: context deadline exceeded Duration: 4.850683119s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [....XX.XX..XXX...X.X....X..XXXX..X.X.XX.....X.X.X..X.XXX..X.] duration: 3.391957085s trace: https://jaeger.dev.storj.io/trace/2c87d4bacf984303 FAIL (2025-12-28 11:43:52 UTC) orbiter warning: context deadline exceeded Duration: 3.391982874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.XX.X..XX..X.......X.......................XX....X.....XX.] duration: 5.885240686s trace: https://jaeger.dev.storj.io/trace/14a25f94a1912b1c FAIL (2025-12-28 11:43:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: CTPWG4XCW43M7S3I5PNKI6UJ3HC52E7XBKX2OM4VMGX6C5XBDRMQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: GHCMJJP3L7RAPPH6ZC3ASYV5LO4HA2BBV5RKINQMA7JBNSI2LDJQ) context deadline exceeded ======LOGS====== Dec 28 11:43:03.153674892 (548h14m3.024757937s) [upload=480719]: Starting upload Dec 28 11:43:03.153770594 (548h14m3.024853659s) [upload=480719]: Done starting upload Dec 28 11:43:03.153828585 (548h14m3.02491163s) [upload=480719]: Uploading segments... Dec 28 11:43:03.154070375 (548h14m3.02515343s) [upload=480719]: Got next segment. Inline: false Dec 28 11:43:03.154105502 (548h14m3.025188547s) [upload=480719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:43:03.154111974 (548h14m3.025195019s) [upload=480719]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:43:03.399935160 (548h14m3.271018195s) [seg_pos={0 0}, upload=480719]: Begin upload segment... Dec 28 11:43:03.399997618 (548h14m3.271080663s) [seg_pos={0 0}, upload=480719]: Done begin upload segment. Dec 28 11:43:03.400037966 (548h14m3.271121011s) [noise=true , address=145.79.227.42:4122 , piece_id=VHK6G5SAXYYJT4KRHAVFJCR3OKX7E3H5WVBM5F77WL7EY3OCKLYQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.400141453 (548h14m3.271224498s) [noise=true , address=45.91.116.8:4045 , piece_id=MB76HIY2R4WSBS5S7Q5AVUT7XHXJS7VQ5IZF3VB6O4OUKFDYHXIA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.400848770 (548h14m3.271931815s) [noise=true , address=145.79.227.52:4447 , piece_id=UCQBBKHI5GRZQJH56XJHSN3LOTMGUGHYDRL5XUMY3FAVQTD7D3AA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.401382976 (548h14m3.272466021s) [noise=true , address=45.91.116.6:4974 , piece_id=OI5PSHBDQG66DQFOX3RCJIVMPGTXBFMYYCMW2ET2TO2I7QS3NJLA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.401873961 (548h14m3.272957016s) [noise=true , address=145.79.227.51:4410 , piece_id=NTRMOWDVB62NZKJ5P5COGJH3GUT7NL3ZJD76NKB25IRZ53LJQLDA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.402033185 (548h14m3.27311623s) [noise=true , address=145.79.227.19:4079 , piece_id=I4GSOARSSTTUIU3JZVHMZMQ6B52SS5GHASLBLFCWEHTIOFLD7EVQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.402464967 (548h14m3.273548022s) [noise=true , address=145.79.227.45:4229 , piece_id=EBHCBS24EV65ODROABQUZFA3C347X7LBQRHSNE4UVELCGCW2EYLA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.402764578 (548h14m3.273847623s) [noise=true , address=145.79.227.41:4087 , piece_id=6IB2AW5NCG7ALSOOIJRKENJYI4GAEE72N6RE6J24DOXVFZF3NKOA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.403065000 (548h14m3.274148045s) [noise=true , address=77.90.31.21:4880 , piece_id=6URQMPZU4GZ7Y4JCAMC7TPIXEB7PKSDFRHH72ZY7AZNCT7J234OA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.403300137 (548h14m3.274383182s) [noise=true , address=45.91.116.4:4939 , piece_id=EJFNFQZKNZK723PJBITR75DYCLBBJWRTIMWHXYMMIEK74OVBI6NA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.403526298 (548h14m3.274609343s) [noise=true , address=145.79.227.26:4655 , piece_id=L2M4QAUCAORBJ6BWUHJGFFEUSTIOKPRDTEFH4CQLPEVBNTJJWAJA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.403855565 (548h14m3.27493861s) [noise=true , address=145.79.227.13:4803 , piece_id=XBVD7MQEK7ZQB7YYKH6CVB36ATQPEU43RMKTI75IG5SDYQRAPE2Q, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.404270605 (548h14m3.27535365s) [noise=true , address=77.90.31.12:4589 , piece_id=CTPWG4XCW43M7S3I5PNKI6UJ3HC52E7XBKX2OM4VMGX6C5XBDRMQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.404406113 (548h14m3.275489158s) [noise=true , address=145.79.227.37:4567 , piece_id=C6PJTJS54NJJW2ZU2OCA5TVXCHEMQZE7O7CQ7CES5SLVOQKK5SQQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.404714901 (548h14m3.275797946s) [noise=true , address=45.91.116.14:4190 , piece_id=GHCMJJP3L7RAPPH6ZC3ASYV5LO4HA2BBV5RKINQMA7JBNSI2LDJQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.404976981 (548h14m3.276060026s) [noise=true , address=45.91.116.13:4153 , piece_id=4OMS7INO5VHFQEYFOSPPPP3JQMS7MRAF66YTYUEDYUYNT756EDGQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:05.111835332 (548h14m4.982918367s) [upload=480719]: Next returned nil segment Dec 28 11:43:05.111852295 (548h14m4.98293533s) [upload=480719]: Waiting for error group managing segments... Dec 28 11:43:05.399405133 (548h14m5.270488178s) [noise=true , address=145.79.227.19:4079 , piece_id=I4GSOARSSTTUIU3JZVHMZMQ6B52SS5GHASLBLFCWEHTIOFLD7EVQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.399556911 (548h14m5.270639957s) [noise=true , address=145.79.227.51:4410 , piece_id=NTRMOWDVB62NZKJ5P5COGJH3GUT7NL3ZJD76NKB25IRZ53LJQLDA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.414139549 (548h14m5.285222595s) [noise=true , address=145.79.227.45:4229 , piece_id=EBHCBS24EV65ODROABQUZFA3C347X7LBQRHSNE4UVELCGCW2EYLA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.415887087 (548h14m5.286970132s) [noise=true , address=145.79.227.37:4567 , piece_id=C6PJTJS54NJJW2ZU2OCA5TVXCHEMQZE7O7CQ7CES5SLVOQKK5SQQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.416590606 (548h14m5.287673651s) [noise=true , address=145.79.227.42:4122 , piece_id=VHK6G5SAXYYJT4KRHAVFJCR3OKX7E3H5WVBM5F77WL7EY3OCKLYQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.422948263 (548h14m5.294031298s) [noise=true , address=145.79.227.52:4447 , piece_id=UCQBBKHI5GRZQJH56XJHSN3LOTMGUGHYDRL5XUMY3FAVQTD7D3AA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.429791414 (548h14m5.300874449s) [noise=true , address=145.79.227.41:4087 , piece_id=6IB2AW5NCG7ALSOOIJRKENJYI4GAEE72N6RE6J24DOXVFZF3NKOA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.431689648 (548h14m5.302772683s) [noise=true , address=145.79.227.13:4803 , piece_id=XBVD7MQEK7ZQB7YYKH6CVB36ATQPEU43RMKTI75IG5SDYQRAPE2Q, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.433491559 (548h14m5.304574604s) [noise=true , address=145.79.227.26:4655 , piece_id=L2M4QAUCAORBJ6BWUHJGFFEUSTIOKPRDTEFH4CQLPEVBNTJJWAJA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.644963917 (548h14m5.516046962s) [noise=true , address=45.91.116.6:4974 , piece_id=OI5PSHBDQG66DQFOX3RCJIVMPGTXBFMYYCMW2ET2TO2I7QS3NJLA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.671643319 (548h14m5.542726364s) [noise=true , address=77.90.31.12:4589 , piece_id=CTPWG4XCW43M7S3I5PNKI6UJ3HC52E7XBKX2OM4VMGX6C5XBDRMQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.674846027 (548h14m5.545929062s) [noise=true , address=45.91.116.4:4939 , piece_id=EJFNFQZKNZK723PJBITR75DYCLBBJWRTIMWHXYMMIEK74OVBI6NA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:06.342722614 (548h14m6.213805659s) [noise=true , address=45.91.116.14:4190 , piece_id=GHCMJJP3L7RAPPH6ZC3ASYV5LO4HA2BBV5RKINQMA7JBNSI2LDJQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:06.342746930 (548h14m6.213829965s) [seg_pos={0 0}, upload=480719]: Segment reached optimal threshold of 13 pieces. Dec 28 11:43:06.342939807 (548h14m6.214022842s) [noise=true , address=77.90.31.21:4880 , piece_id=6URQMPZU4GZ7Y4JCAMC7TPIXEB7PKSDFRHH72ZY7AZNCT7J234OA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 28 11:43:06.342946451 (548h14m6.214029496s) [noise=true , address=77.90.31.21:4880 , piece_id=6URQMPZU4GZ7Y4JCAMC7TPIXEB7PKSDFRHH72ZY7AZNCT7J234OA, seg_pos={0 0}, upload=480719]: Long tail cancellation due to optimal threshold Dec 28 11:43:06.343216424 (548h14m6.214299459s) [noise=true , address=45.91.116.8:4045 , piece_id=MB76HIY2R4WSBS5S7Q5AVUT7XHXJS7VQ5IZF3VB6O4OUKFDYHXIA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 11:43:06.343222667 (548h14m6.214305702s) [noise=true , address=45.91.116.8:4045 , piece_id=MB76HIY2R4WSBS5S7Q5AVUT7XHXJS7VQ5IZF3VB6O4OUKFDYHXIA, seg_pos={0 0}, upload=480719]: Long tail cancellation due to optimal threshold Dec 28 11:43:06.343341052 (548h14m6.214424097s) [noise=true , address=45.91.116.13:4153 , piece_id=4OMS7INO5VHFQEYFOSPPPP3JQMS7MRAF66YTYUEDYUYNT756EDGQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 28 11:43:06.343346862 (548h14m6.214429908s) [noise=true , address=45.91.116.13:4153 , piece_id=4OMS7INO5VHFQEYFOSPPPP3JQMS7MRAF66YTYUEDYUYNT756EDGQ, seg_pos={0 0}, upload=480719]: Long tail cancellation due to optimal threshold Dec 28 11:43:06.343365298 (548h14m6.214448333s) [seg_pos={0 0}, upload=480719]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:43:06.343439409 (548h14m6.214522454s) [upload=480719]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:43:06.343445400 (548h14m6.214528445s) [upload=480719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:43:06.644189937 (548h14m6.515272982s) [upload=480719]: Done uploading segments... Dec 28 11:43:06.644217320 (548h14m6.515300365s) [upload=480719]: Upload finished. err: <nil> Duration: 5.885282133s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.XX.X..XX..X.......X.......................XX....X.....XX] duration: 30.005804094s trace: https://jaeger.dev.storj.io/trace/14a25f94a1912b1c FAIL (2025-12-28 11:43:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: CTPWG4XCW43M7S3I5PNKI6UJ3HC52E7XBKX2OM4VMGX6C5XBDRMQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: GHCMJJP3L7RAPPH6ZC3ASYV5LO4HA2BBV5RKINQMA7JBNSI2LDJQ) context deadline exceeded ======LOGS====== Dec 28 11:43:03.153674892 (548h14m3.024757937s) [upload=480719]: Starting upload Dec 28 11:43:03.153770594 (548h14m3.024853659s) [upload=480719]: Done starting upload Dec 28 11:43:03.153828585 (548h14m3.02491163s) [upload=480719]: Uploading segments... Dec 28 11:43:03.154070375 (548h14m3.02515343s) [upload=480719]: Got next segment. Inline: false Dec 28 11:43:03.154105502 (548h14m3.025188547s) [upload=480719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:43:03.154111974 (548h14m3.025195019s) [upload=480719]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:43:03.399935160 (548h14m3.271018195s) [seg_pos={0 0}, upload=480719]: Begin upload segment... Dec 28 11:43:03.399997618 (548h14m3.271080663s) [seg_pos={0 0}, upload=480719]: Done begin upload segment. Dec 28 11:43:03.400037966 (548h14m3.271121011s) [noise=true , address=145.79.227.42:4122 , piece_id=VHK6G5SAXYYJT4KRHAVFJCR3OKX7E3H5WVBM5F77WL7EY3OCKLYQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.400141453 (548h14m3.271224498s) [noise=true , address=45.91.116.8:4045 , piece_id=MB76HIY2R4WSBS5S7Q5AVUT7XHXJS7VQ5IZF3VB6O4OUKFDYHXIA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.400848770 (548h14m3.271931815s) [noise=true , address=145.79.227.52:4447 , piece_id=UCQBBKHI5GRZQJH56XJHSN3LOTMGUGHYDRL5XUMY3FAVQTD7D3AA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.401382976 (548h14m3.272466021s) [noise=true , address=45.91.116.6:4974 , piece_id=OI5PSHBDQG66DQFOX3RCJIVMPGTXBFMYYCMW2ET2TO2I7QS3NJLA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.401873961 (548h14m3.272957016s) [noise=true , address=145.79.227.51:4410 , piece_id=NTRMOWDVB62NZKJ5P5COGJH3GUT7NL3ZJD76NKB25IRZ53LJQLDA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.402033185 (548h14m3.27311623s) [noise=true , address=145.79.227.19:4079 , piece_id=I4GSOARSSTTUIU3JZVHMZMQ6B52SS5GHASLBLFCWEHTIOFLD7EVQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.402464967 (548h14m3.273548022s) [noise=true , address=145.79.227.45:4229 , piece_id=EBHCBS24EV65ODROABQUZFA3C347X7LBQRHSNE4UVELCGCW2EYLA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.402764578 (548h14m3.273847623s) [noise=true , address=145.79.227.41:4087 , piece_id=6IB2AW5NCG7ALSOOIJRKENJYI4GAEE72N6RE6J24DOXVFZF3NKOA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.403065000 (548h14m3.274148045s) [noise=true , address=77.90.31.21:4880 , piece_id=6URQMPZU4GZ7Y4JCAMC7TPIXEB7PKSDFRHH72ZY7AZNCT7J234OA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.403300137 (548h14m3.274383182s) [noise=true , address=45.91.116.4:4939 , piece_id=EJFNFQZKNZK723PJBITR75DYCLBBJWRTIMWHXYMMIEK74OVBI6NA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.403526298 (548h14m3.274609343s) [noise=true , address=145.79.227.26:4655 , piece_id=L2M4QAUCAORBJ6BWUHJGFFEUSTIOKPRDTEFH4CQLPEVBNTJJWAJA, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.403855565 (548h14m3.27493861s) [noise=true , address=145.79.227.13:4803 , piece_id=XBVD7MQEK7ZQB7YYKH6CVB36ATQPEU43RMKTI75IG5SDYQRAPE2Q, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.404270605 (548h14m3.27535365s) [noise=true , address=77.90.31.12:4589 , piece_id=CTPWG4XCW43M7S3I5PNKI6UJ3HC52E7XBKX2OM4VMGX6C5XBDRMQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.404406113 (548h14m3.275489158s) [noise=true , address=145.79.227.37:4567 , piece_id=C6PJTJS54NJJW2ZU2OCA5TVXCHEMQZE7O7CQ7CES5SLVOQKK5SQQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.404714901 (548h14m3.275797946s) [noise=true , address=45.91.116.14:4190 , piece_id=GHCMJJP3L7RAPPH6ZC3ASYV5LO4HA2BBV5RKINQMA7JBNSI2LDJQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:03.404976981 (548h14m3.276060026s) [noise=true , address=45.91.116.13:4153 , piece_id=4OMS7INO5VHFQEYFOSPPPP3JQMS7MRAF66YTYUEDYUYNT756EDGQ, seg_pos={0 0}, upload=480719]: Uploading piece... Dec 28 11:43:05.111835332 (548h14m4.982918367s) [upload=480719]: Next returned nil segment Dec 28 11:43:05.111852295 (548h14m4.98293533s) [upload=480719]: Waiting for error group managing segments... Dec 28 11:43:05.399405133 (548h14m5.270488178s) [noise=true , address=145.79.227.19:4079 , piece_id=I4GSOARSSTTUIU3JZVHMZMQ6B52SS5GHASLBLFCWEHTIOFLD7EVQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.399556911 (548h14m5.270639957s) [noise=true , address=145.79.227.51:4410 , piece_id=NTRMOWDVB62NZKJ5P5COGJH3GUT7NL3ZJD76NKB25IRZ53LJQLDA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.414139549 (548h14m5.285222595s) [noise=true , address=145.79.227.45:4229 , piece_id=EBHCBS24EV65ODROABQUZFA3C347X7LBQRHSNE4UVELCGCW2EYLA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.415887087 (548h14m5.286970132s) [noise=true , address=145.79.227.37:4567 , piece_id=C6PJTJS54NJJW2ZU2OCA5TVXCHEMQZE7O7CQ7CES5SLVOQKK5SQQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.416590606 (548h14m5.287673651s) [noise=true , address=145.79.227.42:4122 , piece_id=VHK6G5SAXYYJT4KRHAVFJCR3OKX7E3H5WVBM5F77WL7EY3OCKLYQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.422948263 (548h14m5.294031298s) [noise=true , address=145.79.227.52:4447 , piece_id=UCQBBKHI5GRZQJH56XJHSN3LOTMGUGHYDRL5XUMY3FAVQTD7D3AA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.429791414 (548h14m5.300874449s) [noise=true , address=145.79.227.41:4087 , piece_id=6IB2AW5NCG7ALSOOIJRKENJYI4GAEE72N6RE6J24DOXVFZF3NKOA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.431689648 (548h14m5.302772683s) [noise=true , address=145.79.227.13:4803 , piece_id=XBVD7MQEK7ZQB7YYKH6CVB36ATQPEU43RMKTI75IG5SDYQRAPE2Q, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.433491559 (548h14m5.304574604s) [noise=true , address=145.79.227.26:4655 , piece_id=L2M4QAUCAORBJ6BWUHJGFFEUSTIOKPRDTEFH4CQLPEVBNTJJWAJA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.644963917 (548h14m5.516046962s) [noise=true , address=45.91.116.6:4974 , piece_id=OI5PSHBDQG66DQFOX3RCJIVMPGTXBFMYYCMW2ET2TO2I7QS3NJLA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.671643319 (548h14m5.542726364s) [noise=true , address=77.90.31.12:4589 , piece_id=CTPWG4XCW43M7S3I5PNKI6UJ3HC52E7XBKX2OM4VMGX6C5XBDRMQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:05.674846027 (548h14m5.545929062s) [noise=true , address=45.91.116.4:4939 , piece_id=EJFNFQZKNZK723PJBITR75DYCLBBJWRTIMWHXYMMIEK74OVBI6NA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:06.342722614 (548h14m6.213805659s) [noise=true , address=45.91.116.14:4190 , piece_id=GHCMJJP3L7RAPPH6ZC3ASYV5LO4HA2BBV5RKINQMA7JBNSI2LDJQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: <nil> Dec 28 11:43:06.342746930 (548h14m6.213829965s) [seg_pos={0 0}, upload=480719]: Segment reached optimal threshold of 13 pieces. Dec 28 11:43:06.342939807 (548h14m6.214022842s) [noise=true , address=77.90.31.21:4880 , piece_id=6URQMPZU4GZ7Y4JCAMC7TPIXEB7PKSDFRHH72ZY7AZNCT7J234OA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 28 11:43:06.342946451 (548h14m6.214029496s) [noise=true , address=77.90.31.21:4880 , piece_id=6URQMPZU4GZ7Y4JCAMC7TPIXEB7PKSDFRHH72ZY7AZNCT7J234OA, seg_pos={0 0}, upload=480719]: Long tail cancellation due to optimal threshold Dec 28 11:43:06.343216424 (548h14m6.214299459s) [noise=true , address=45.91.116.8:4045 , piece_id=MB76HIY2R4WSBS5S7Q5AVUT7XHXJS7VQ5IZF3VB6O4OUKFDYHXIA, seg_pos={0 0}, upload=480719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 11:43:06.343222667 (548h14m6.214305702s) [noise=true , address=45.91.116.8:4045 , piece_id=MB76HIY2R4WSBS5S7Q5AVUT7XHXJS7VQ5IZF3VB6O4OUKFDYHXIA, seg_pos={0 0}, upload=480719]: Long tail cancellation due to optimal threshold Dec 28 11:43:06.343341052 (548h14m6.214424097s) [noise=true , address=45.91.116.13:4153 , piece_id=4OMS7INO5VHFQEYFOSPPPP3JQMS7MRAF66YTYUEDYUYNT756EDGQ, seg_pos={0 0}, upload=480719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 28 11:43:06.343346862 (548h14m6.214429908s) [noise=true , address=45.91.116.13:4153 , piece_id=4OMS7INO5VHFQEYFOSPPPP3JQMS7MRAF66YTYUEDYUYNT756EDGQ, seg_pos={0 0}, upload=480719]: Long tail cancellation due to optimal threshold Dec 28 11:43:06.343365298 (548h14m6.214448333s) [seg_pos={0 0}, upload=480719]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:43:06.343439409 (548h14m6.214522454s) [upload=480719]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:43:06.343445400 (548h14m6.214528445s) [upload=480719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:43:06.644189937 (548h14m6.515272982s) [upload=480719]: Done uploading segments... Dec 28 11:43:06.644217320 (548h14m6.515300365s) [upload=480719]: Upload finished. err: <nil> Duration: 30.005828993s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X........X...........X..X...............] duration: 49.976251246s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:43:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 49.976274501s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [.....XX.XX..XXX...X.X....X..XXXX..X.X.XX.....X.X.X..X.XXX..X] duration: 5.001545113s trace: https://jaeger.dev.storj.io/trace/2c87d4bacf984303 FAIL (2025-12-28 11:43:11 UTC) orbiter warning: context deadline exceeded Duration: 5.001568748s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X.XX.X..XX..X.......X.......................XX....X.....X] duration: 30.003017777s trace: https://jaeger.dev.storj.io/trace/f9f99de95ed93d5 FAIL (2025-12-28 11:42:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: 62CT2XVVJKKQVE7L6QW6CXUFSWLM5VBFMLBHUW5IJBVQATGQCO4Q) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: VLI7SNITSXWFIWQG6VQ7XRBUFARWCPV6R7MI3GEOA777SZJ73PNA) context deadline exceeded ======LOGS====== Dec 28 11:42:10.544016768 (548h13m10.415099823s) [upload=480705]: Starting upload Dec 28 11:42:10.544115977 (548h13m10.415199022s) [upload=480705]: Done starting upload Dec 28 11:42:10.544178275 (548h13m10.415261311s) [upload=480705]: Uploading segments... Dec 28 11:42:10.544383255 (548h13m10.4154663s) [upload=480705]: Got next segment. Inline: false Dec 28 11:42:10.544463278 (548h13m10.415546323s) [upload=480705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:42:10.544472716 (548h13m10.415555751s) [upload=480705]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:42:10.788635923 (548h13m10.659718958s) [seg_pos={0 0}, upload=480705]: Begin upload segment... Dec 28 11:42:10.788697299 (548h13m10.659780334s) [seg_pos={0 0}, upload=480705]: Done begin upload segment. Dec 28 11:42:10.788734800 (548h13m10.659817835s) [noise=true , address=145.79.227.14:4837 , piece_id=UE7KQILD75LNMDOHSNC2A3R7CUKKXETT2GRR6URAJNB24ZXO322A, seg_pos={0 0}, upload=480705]: Uploading piece... Dec 28 11:42:10.788746964 (548h13m10.659829999s) [noise=true , address=145.79.227.38:4979 , piece_id=OQWZMMST5L4UP4DYG2PKZTFNW3XK3EI4UDDT3KL727HXRZQNRAJQ, seg_pos={0 0}, upload=480705]: Uploading piece... Dec 28 11:42:10.788840271 (548h13m10.659923306s) [noise=true , address=145.79.227.50:4374 , piece_id=A77ZYZ5VSYSSWVGQDOEGQLQF3DMYIYCZ756G55IG34OIQVV7Q73A, seg_pos={0 0}, upload=480705]: Uploading piece... Dec 28 11:42:10.788947014 (548h13m10.660030049s) [noise=true , address=145.79.227.37:4565 , piece_id=AMC5R3CAIVZHRMVRTGRNJPN435S53TTZQJZ4VRBZ4SPF67V5B6HQ, seg_pos={0 0}, upload=480705]: Uploading piece... Dec 28 11:42:10.789498665 (548h13m10.6605817s) [noise=true , address=145.79.227.17:4971 , piece_id=BYAZESX35XHQDQLGWSRBYTAIPSOWILAQMLFHCGHBARWQUBQJM5PQ, seg_pos={0 0}, upload=480705]: Uploading piece... Dec 28 11:42:10.789723503 (548h13m10.660806538s) [noise=true , address=145.79.227.4:4185 , piece_id=YQOXFYIDHRI36YSLU26FA2CSFOTV2LX6D64EPXSPLWX2FGZYSD3A, seg_pos={0 0}, upload=480705]: Uploading piece... Dec 28 11:42:10.789989790 (548h13m10.661072835s) [noise=true , address=45.91.116.11:4117 , piece_id=QI3FRTNG7LTMDY7RRYFZMWBYLLL7MBUFV4J63J6LQDBGL4M2NVHA, seg_pos={0 0}, upload=480705]: Uploading piece... Dec 28 11:42:10.790149324 (548h13m10.661232369s) [noise=true , address=77.90.31.21:4877 , piece_id=6TQ2XTDAOQDJ7YQCBSE6FAYMVLQWO3RTHPPOX4PARJ7UDXKESGAA, seg_pos={0 0}, upload=480705]: Uploading piece... Dec 28 11:42:10.790485104 (548h13m10.661568139s) [noise=true , address=145.79.227.15:4874 , piece_id=YLK7JVPN2XEFTUNSDMZKOBJ7SHXBE67KHAV65GUOJQJOZBQJL5TQ, seg_pos={0 0}, upload=480705]: Uploading piece... Dec 28 11:42:10.790676558 (548h13m10.661759603s) [noise=true , address=45.91.116.5:4297 , piece_id=VLI7SNITSXWFIWQG6VQ7XRBUFARWCPV6R7MI3GEOA777SZJ73PNA, seg_pos={0 0}, upload=480705]: Uploading piece... Dec 28 11:42:10.791048186 (548h13m10.662131221s) [noise=true , address=45.91.116.6:4973 , piece_id=TAGSF4YCF5QA4ZRZKNQRXWD2FND2NIGFSAM7CRJODJ5GRPHW5LHQ, seg_pos={0 0}, upload=480705]: Uploading piece... Dec 28 11:42:10.791464889 (548h13m10.662547924s) [noise=true , address=145.79.227.35:4482 , piece_id=KKVBWWWCKFMDUGOTERZNY3OYNVKBDGXLLY55244VK3SLHDKOB77A, seg_pos={0 0}, upload=480705]: Uploading piece... Dec 28 11:42:10.792043570 (548h13m10.663126605s) [noise=true , address=45.91.116.4:4940 , piece_id=2ZKQ5DZXMOACKMQ6NBRIYSVC5PRV6POLSYJRQNQFCFOADL5FKSMQ, seg_pos={0 0}, upload=480705]: Uploading piece... Dec 28 11:42:10.792660996 (548h13m10.663744031s) [noise=true , address=145.79.227.43:4157 , piece_id=HK6DVR7YTDZ5CSCXE3IZWCVY3OPUEKCZCNBM6GHDOG7SDXXO2LEA, seg_pos={0 0}, upload=480705]: Uploading piece... Dec 28 11:42:10.793045669 (548h13m10.664128704s) [noise=true , address=77.90.31.15:4700 , piece_id=62CT2XVVJKKQVE7L6QW6CXUFSWLM5VBFMLBHUW5IJBVQATGQCO4Q, seg_pos={0 0}, upload=480705]: Uploading piece... Dec 28 11:42:10.795808090 (548h13m10.666891125s) [noise=true , address=145.79.227.42:4122 , piece_id=A7IKONMDQWODMYPPF77WXNPOQPQYVQ7TSW5D2OKP6ICHQ3M4HC2Q, seg_pos={0 0}, upload=480705]: Uploading piece... Dec 28 11:42:12.508691080 (548h13m12.379774125s) [upload=480705]: Next returned nil segment Dec 28 11:42:12.508707822 (548h13m12.379790857s) [upload=480705]: Waiting for error group managing segments... Dec 28 11:42:12.798911548 (548h13m12.669994583s) [noise=true , address=145.79.227.50:4374 , piece_id=A77ZYZ5VSYSSWVGQDOEGQLQF3DMYIYCZ756G55IG34OIQVV7Q73A, seg_pos={0 0}, upload=480705]: Done uploading piece. err: <nil> Dec 28 11:42:12.799346626 (548h13m12.670429671s) [noise=true , address=145.79.227.15:4874 , piece_id=YLK7JVPN2XEFTUNSDMZKOBJ7SHXBE67KHAV65GUOJQJOZBQJL5TQ, seg_pos={0 0}, upload=480705]: Done uploading piece. err: <nil> Dec 28 11:42:12.802098546 (548h13m12.673181581s) [noise=true , address=145.79.227.14:4837 , piece_id=UE7KQILD75LNMDOHSNC2A3R7CUKKXETT2GRR6URAJNB24ZXO322A, seg_pos={0 0}, upload=480705]: Done uploading piece. err: <nil> Dec 28 11:42:12.802505761 (548h13m12.673588807s) [noise=true , address=145.79.227.38:4979 , piece_id=OQWZMMST5L4UP4DYG2PKZTFNW3XK3EI4UDDT3KL727HXRZQNRAJQ, seg_pos={0 0}, upload=480705]: Done uploading piece. err: <nil> Dec 28 11:42:12.802563311 (548h13m12.673646346s) [noise=true , address=145.79.227.35:4482 , piece_id=KKVBWWWCKFMDUGOTERZNY3OYNVKBDGXLLY55244VK3SLHDKOB77A, seg_pos={0 0}, upload=480705]: Done uploading piece. err: <nil> Dec 28 11:42:12.803172811 (548h13m12.674255847s) [noise=true , address=145.79.227.4:4185 , piece_id=YQOXFYIDHRI36YSLU26FA2CSFOTV2LX6D64EPXSPLWX2FGZYSD3A, seg_pos={0 0}, upload=480705]: Done uploading piece. err: <nil> Dec 28 11:42:12.803455019 (548h13m12.674538054s) [noise=true , address=145.79.227.37:4565 , piece_id=AMC5R3CAIVZHRMVRTGRNJPN435S53TTZQJZ4VRBZ4SPF67V5B6HQ, seg_pos={0 0}, upload=480705]: Done uploading piece. err: <nil> Dec 28 11:42:12.803783785 (548h13m12.67486682s) [noise=true , address=145.79.227.17:4971 , piece_id=BYAZESX35XHQDQLGWSRBYTAIPSOWILAQMLFHCGHBARWQUBQJM5PQ, seg_pos={0 0}, upload=480705]: Done uploading piece. err: <nil> Dec 28 11:42:12.803838529 (548h13m12.674921564s) [noise=true , address=145.79.227.43:4157 , piece_id=HK6DVR7YTDZ5CSCXE3IZWCVY3OPUEKCZCNBM6GHDOG7SDXXO2LEA, seg_pos={0 0}, upload=480705]: Done uploading piece. err: <nil> Dec 28 11:42:12.817009940 (548h13m12.688092995s) [noise=true , address=145.79.227.42:4122 , piece_id=A7IKONMDQWODMYPPF77WXNPOQPQYVQ7TSW5D2OKP6ICHQ3M4HC2Q, seg_pos={0 0}, upload=480705]: Done uploading piece. err: <nil> Dec 28 11:42:13.070017388 (548h13m12.941100433s) [noise=true , address=45.91.116.5:4297 , piece_id=VLI7SNITSXWFIWQG6VQ7XRBUFARWCPV6R7MI3GEOA777SZJ73PNA, seg_pos={0 0}, upload=480705]: Done uploading piece. err: <nil> Dec 28 11:42:13.122067580 (548h13m12.993150625s) [noise=true , address=77.90.31.15:4700 , piece_id=62CT2XVVJKKQVE7L6QW6CXUFSWLM5VBFMLBHUW5IJBVQATGQCO4Q, seg_pos={0 0}, upload=480705]: Done uploading piece. err: <nil> Dec 28 11:42:13.176321429 (548h13m13.047404464s) [noise=true , address=45.91.116.6:4973 , piece_id=TAGSF4YCF5QA4ZRZKNQRXWD2FND2NIGFSAM7CRJODJ5GRPHW5LHQ, seg_pos={0 0}, upload=480705]: Done uploading piece. err: <nil> Dec 28 11:42:13.176350544 (548h13m13.047433589s) [seg_pos={0 0}, upload=480705]: Segment reached optimal threshold of 13 pieces. Dec 28 11:42:13.176615779 (548h13m13.047698824s) [noise=true , address=45.91.116.11:4117 , piece_id=QI3FRTNG7LTMDY7RRYFZMWBYLLL7MBUFV4J63J6LQDBGL4M2NVHA, seg_pos={0 0}, upload=480705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 28 11:42:13.176626118 (548h13m13.047709153s) [noise=true , address=45.91.116.11:4117 , piece_id=QI3FRTNG7LTMDY7RRYFZMWBYLLL7MBUFV4J63J6LQDBGL4M2NVHA, seg_pos={0 0}, upload=480705]: Long tail cancellation due to optimal threshold Dec 28 11:42:13.176804127 (548h13m13.047887172s) [noise=true , address=45.91.116.4:4940 , piece_id=2ZKQ5DZXMOACKMQ6NBRIYSVC5PRV6POLSYJRQNQFCFOADL5FKSMQ, seg_pos={0 0}, upload=480705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 28 11:42:13.176815379 (548h13m13.047898424s) [noise=true , address=45.91.116.4:4940 , piece_id=2ZKQ5DZXMOACKMQ6NBRIYSVC5PRV6POLSYJRQNQFCFOADL5FKSMQ, seg_pos={0 0}, upload=480705]: Long tail cancellation due to optimal threshold Dec 28 11:42:13.176836949 (548h13m13.047919994s) [noise=true , address=77.90.31.21:4877 , piece_id=6TQ2XTDAOQDJ7YQCBSE6FAYMVLQWO3RTHPPOX4PARJ7UDXKESGAA, seg_pos={0 0}, upload=480705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:54488->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 11:42:13.176846498 (548h13m13.047929533s) [noise=true , address=77.90.31.21:4877 , piece_id=6TQ2XTDAOQDJ7YQCBSE6FAYMVLQWO3RTHPPOX4PARJ7UDXKESGAA, seg_pos={0 0}, upload=480705]: Long tail cancellation due to optimal threshold Dec 28 11:42:13.176864983 (548h13m13.047948028s) [seg_pos={0 0}, upload=480705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:42:13.176973139 (548h13m13.048056184s) [upload=480705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:42:13.176979030 (548h13m13.048062066s) [upload=480705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:42:13.473328618 (548h13m13.344411663s) [upload=480705]: Done uploading segments... Dec 28 11:42:13.473355199 (548h13m13.344438244s) [upload=480705]: Upload finished. err: <nil> Duration: 30.003042654s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [X.....XX.XX..XXX...X.X....X..XXXX..X.X.XX.....X.X.X..X.XXX..] duration: 3.56005453s trace: https://jaeger.dev.storj.io/trace/496a4e47ee73e6d8 FAIL (2025-12-28 11:42:27 UTC) orbiter warning: context deadline exceeded Duration: 3.560075039s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X........X...........X..X..............] duration: 15.212049478s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:42:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 15.212066229s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X........X...........X..X.............] duration: 54.698655038s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:42:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 54.698683854s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X...X.XX.X..XX..X.......X.......................XX....X.....] duration: 5.729849535s trace: https://jaeger.dev.storj.io/trace/2fefc5bfc20bdb01 FAIL (2025-12-28 11:41:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8h9m47.099694332s) [upload=480657]: Uploading segments... Dec 28 11:38:47.228817881 (548h9m47.099900926s) [upload=480657]: Got next segment. Inline: false Dec 28 11:38:47.228860011 (548h9m47.099943056s) [upload=480657]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:38:47.228865792 (548h9m47.099948827s) [upload=480657]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:38:47.473831737 (548h9m47.344914782s) [seg_pos={0 0}, upload=480657]: Begin upload segment... Dec 28 11:38:47.473887965 (548h9m47.34497101s) [seg_pos={0 0}, upload=480657]: Done begin upload segment. Dec 28 11:38:47.473925045 (548h9m47.34500809s) [noise=true , address=45.91.116.5:4297 , piece_id=AFKJWUGSB566MVE64PST3PSWZGK5VA3YOAI7ZUYKSSFGADLAQHAA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.474004907 (548h9m47.345087952s) [noise=true , address=145.79.227.47:4265 , piece_id=POXVT6RXKKWGQJSZZSB4Q733VQXBJGNZBNXWMXFEVYU6S6FUFSYA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.474561437 (548h9m47.345644472s) [noise=true , address=145.79.227.11:4729 , piece_id=THEOLKVVM2B4BS4H3J5EW3VFRAW36NM4Z65FVLBVA5JINHHBVYNQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.474866628 (548h9m47.345949663s) [noise=true , address=45.91.116.11:4118 , piece_id=ZHWNRZJI73UJBC5Z5NZB7LSAO7KRNDT7R7AH2IFZLTLMR2AHVQPA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.475139508 (548h9m47.346222543s) [noise=true , address=45.91.116.14:4190 , piece_id=JY6LQDWYSFGYTZFCBI7XKWPWLC7FP7HNZMY2OKLUUF6ISN7UCBIQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.475427065 (548h9m47.34651011s) [noise=true , address=45.91.116.9:4262 , piece_id=7BQ2CIDXBSY54FLSL6VO3JZC46BEJ3APNNXHLZMG5TPOE3LSPJ4A, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.475719562 (548h9m47.346802607s) [noise=true , address=145.79.227.15:4875 , piece_id=64HMNDX7SUWKB6CDIE44XEOVCL523GP6BT3TK52GUJYCLWCWQMSQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.476142047 (548h9m47.347225092s) [noise=true , address=145.79.227.39:4014 , piece_id=4RQHZIGPG5MSZWSJOXC25FA2GOWRTKVXSKXKT5MQNIQF52JVZURQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.476600019 (548h9m47.347683064s) [noise=true , address=145.79.227.16:4911 , piece_id=TKJRVZZJHEBVRLEGLKSL6A43PELBQ5T2VSC6EKV4I77ISCP4KCKA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.477869055 (548h9m47.34895209s) [noise=true , address=145.79.227.52:4447 , piece_id=26WZQBGHPBUMCEXYGAZULOGI2A2FE46LDL42ERWXUOO3BLPUSQUQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.479013264 (548h9m47.350096299s) [noise=true , address=145.79.227.26:4655 , piece_id=3EURLMKAKAXAV5E5WGP62QNP6N34FXSZVIICDES2IBH7URBI5XOQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.481790542 (548h9m47.352873587s) [noise=true , address=145.79.227.2:4114 , piece_id=WJRWUD35OW3S4NKXDWUT3F2X34JG6CWUGXDOKIUDNXEWTVAYB6JA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.482069744 (548h9m47.353152779s) [noise=true , address=45.91.116.4:4939 , piece_id=4IST6DJ5O3ORIO6BHOGFEURKSKAPPLXFNZPTDF35RAGOTFTAKWAQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.486092984 (548h9m47.357176019s) [noise=true , address=145.79.227.45:4231 , piece_id=H3DBZR6ENJUOFX7D3PXLW2SIY3LYHJAD2EI4BEEC2R4G5TA3CF7A, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.489275004 (548h9m47.360358049s) [noise=true , address=45.91.116.13:4156 , piece_id=XNBPVP6ITRD2XUUD7SFWW5DCY2AT5JMGG3WUX5SJS4GGHFDWJ3LQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.491105800 (548h9m47.362188845s) [noise=true , address=145.79.227.14:4839 , piece_id=5X62CRU2CW2MV4G5GQGUO4J4B2PY7YI6VQFQ7SXLG2C4R4Z2IPZQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:49.204036963 (548h9m49.075120008s) [upload=480657]: Next returned nil segment Dec 28 11:38:49.204082359 (548h9m49.075165395s) [upload=480657]: Waiting for error group managing segments... Dec 28 11:38:49.481600209 (548h9m49.352683244s) [noise=true , address=145.79.227.15:4875 , piece_id=64HMNDX7SUWKB6CDIE44XEOVCL523GP6BT3TK52GUJYCLWCWQMSQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.483491170 (548h9m49.354574215s) [noise=true , address=145.79.227.11:4729 , piece_id=THEOLKVVM2B4BS4H3J5EW3VFRAW36NM4Z65FVLBVA5JINHHBVYNQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.493301911 (548h9m49.364384946s) [noise=true , address=145.79.227.45:4231 , piece_id=H3DBZR6ENJUOFX7D3PXLW2SIY3LYHJAD2EI4BEEC2R4G5TA3CF7A, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.495779899 (548h9m49.366862944s) [noise=true , address=145.79.227.2:4114 , piece_id=WJRWUD35OW3S4NKXDWUT3F2X34JG6CWUGXDOKIUDNXEWTVAYB6JA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.498460213 (548h9m49.369543258s) [noise=true , address=145.79.227.14:4839 , piece_id=5X62CRU2CW2MV4G5GQGUO4J4B2PY7YI6VQFQ7SXLG2C4R4Z2IPZQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.499105783 (548h9m49.370188818s) [noise=true , address=145.79.227.26:4655 , piece_id=3EURLMKAKAXAV5E5WGP62QNP6N34FXSZVIICDES2IBH7URBI5XOQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.499901318 (548h9m49.370984363s) [noise=true , address=145.79.227.16:4911 , piece_id=TKJRVZZJHEBVRLEGLKSL6A43PELBQ5T2VSC6EKV4I77ISCP4KCKA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.507835054 (548h9m49.378918099s) [noise=true , address=145.79.227.39:4014 , piece_id=4RQHZIGPG5MSZWSJOXC25FA2GOWRTKVXSKXKT5MQNIQF52JVZURQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.516648407 (548h9m49.387731442s) [noise=true , address=145.79.227.47:4265 , piece_id=POXVT6RXKKWGQJSZZSB4Q733VQXBJGNZBNXWMXFEVYU6S6FUFSYA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.518784324 (548h9m49.389867359s) [noise=true , address=145.79.227.52:4447 , piece_id=26WZQBGHPBUMCEXYGAZULOGI2A2FE46LDL42ERWXUOO3BLPUSQUQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.731298456 (548h9m49.602381501s) [noise=true , address=45.91.116.4:4939 , piece_id=4IST6DJ5O3ORIO6BHOGFEURKSKAPPLXFNZPTDF35RAGOTFTAKWAQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.749791860 (548h9m49.620874895s) [noise=true , address=45.91.116.14:4190 , piece_id=JY6LQDWYSFGYTZFCBI7XKWPWLC7FP7HNZMY2OKLUUF6ISN7UCBIQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.762986425 (548h9m49.63406947s) [noise=true , address=45.91.116.13:4156 , piece_id=XNBPVP6ITRD2XUUD7SFWW5DCY2AT5JMGG3WUX5SJS4GGHFDWJ3LQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.763007395 (548h9m49.63409043s) [seg_pos={0 0}, upload=480657]: Segment reached optimal threshold of 13 pieces. Dec 28 11:38:49.763213838 (548h9m49.634296883s) [noise=true , address=45.91.116.9:4262 , piece_id=7BQ2CIDXBSY54FLSL6VO3JZC46BEJ3APNNXHLZMG5TPOE3LSPJ4A, seg_pos={0 0}, upload=480657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:45844->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:45844->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45844->45.91.116.9:4262: use of closed network connection Dec 28 11:38:49.763220290 (548h9m49.634303325s) [noise=true , address=45.91.116.9:4262 , piece_id=7BQ2CIDXBSY54FLSL6VO3JZC46BEJ3APNNXHLZMG5TPOE3LSPJ4A, seg_pos={0 0}, upload=480657]: Long tail cancellation due to optimal threshold Dec 28 11:38:49.763265466 (548h9m49.634348501s) [noise=true , address=45.91.116.11:4118 , piece_id=ZHWNRZJI73UJBC5Z5NZB7LSAO7KRNDT7R7AH2IFZLTLMR2AHVQPA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:53220->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:53220->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53220->45.91.116.11:4118: use of closed network connection Dec 28 11:38:49.763273912 (548h9m49.634356957s) [noise=true , address=45.91.116.11:4118 , piece_id=ZHWNRZJI73UJBC5Z5NZB7LSAO7KRNDT7R7AH2IFZLTLMR2AHVQPA, seg_pos={0 0}, upload=480657]: Long tail cancellation due to optimal threshold Dec 28 11:38:49.763427144 (548h9m49.634510179s) [noise=true , address=45.91.116.5:4297 , piece_id=AFKJWUGSB566MVE64PST3PSWZGK5VA3YOAI7ZUYKSSFGADLAQHAA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 11:38:49.763434107 (548h9m49.634517142s) [noise=true , address=45.91.116.5:4297 , piece_id=AFKJWUGSB566MVE64PST3PSWZGK5VA3YOAI7ZUYKSSFGADLAQHAA, seg_pos={0 0}, upload=480657]: Long tail cancellation due to optimal threshold Dec 28 11:38:49.763449908 (548h9m49.634532953s) [seg_pos={0 0}, upload=480657]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:38:49.763519551 (548h9m49.634602596s) [upload=480657]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:38:49.763530692 (548h9m49.634613737s) [upload=480657]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:38:50.053154022 (548h9m49.924237057s) [upload=480657]: Done uploading segments... Dec 28 11:38:50.053173539 (548h9m49.924256574s) [upload=480657]: Upload finished. err: <nil> Duration: 5.729910291s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [.X.....XX.XX..XXX...X.X....X..XXXX..X.X.XX.....X.X.X..X.XXX.] duration: 3.631135019s trace: https://jaeger.dev.storj.io/trace/496a4e47ee73e6d8 FAIL (2025-12-28 11:41:48 UTC) orbiter warning: context deadline exceeded Duration: 3.631152031s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XX...X.XX.X..XX..X.......X.......................XX....X....] duration: 6.020101794s trace: https://jaeger.dev.storj.io/trace/2fefc5bfc20bdb01 FAIL (2025-12-28 11:41:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8h9m47.099694332s) [upload=480657]: Uploading segments... Dec 28 11:38:47.228817881 (548h9m47.099900926s) [upload=480657]: Got next segment. Inline: false Dec 28 11:38:47.228860011 (548h9m47.099943056s) [upload=480657]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:38:47.228865792 (548h9m47.099948827s) [upload=480657]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:38:47.473831737 (548h9m47.344914782s) [seg_pos={0 0}, upload=480657]: Begin upload segment... Dec 28 11:38:47.473887965 (548h9m47.34497101s) [seg_pos={0 0}, upload=480657]: Done begin upload segment. Dec 28 11:38:47.473925045 (548h9m47.34500809s) [noise=true , address=45.91.116.5:4297 , piece_id=AFKJWUGSB566MVE64PST3PSWZGK5VA3YOAI7ZUYKSSFGADLAQHAA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.474004907 (548h9m47.345087952s) [noise=true , address=145.79.227.47:4265 , piece_id=POXVT6RXKKWGQJSZZSB4Q733VQXBJGNZBNXWMXFEVYU6S6FUFSYA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.474561437 (548h9m47.345644472s) [noise=true , address=145.79.227.11:4729 , piece_id=THEOLKVVM2B4BS4H3J5EW3VFRAW36NM4Z65FVLBVA5JINHHBVYNQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.474866628 (548h9m47.345949663s) [noise=true , address=45.91.116.11:4118 , piece_id=ZHWNRZJI73UJBC5Z5NZB7LSAO7KRNDT7R7AH2IFZLTLMR2AHVQPA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.475139508 (548h9m47.346222543s) [noise=true , address=45.91.116.14:4190 , piece_id=JY6LQDWYSFGYTZFCBI7XKWPWLC7FP7HNZMY2OKLUUF6ISN7UCBIQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.475427065 (548h9m47.34651011s) [noise=true , address=45.91.116.9:4262 , piece_id=7BQ2CIDXBSY54FLSL6VO3JZC46BEJ3APNNXHLZMG5TPOE3LSPJ4A, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.475719562 (548h9m47.346802607s) [noise=true , address=145.79.227.15:4875 , piece_id=64HMNDX7SUWKB6CDIE44XEOVCL523GP6BT3TK52GUJYCLWCWQMSQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.476142047 (548h9m47.347225092s) [noise=true , address=145.79.227.39:4014 , piece_id=4RQHZIGPG5MSZWSJOXC25FA2GOWRTKVXSKXKT5MQNIQF52JVZURQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.476600019 (548h9m47.347683064s) [noise=true , address=145.79.227.16:4911 , piece_id=TKJRVZZJHEBVRLEGLKSL6A43PELBQ5T2VSC6EKV4I77ISCP4KCKA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.477869055 (548h9m47.34895209s) [noise=true , address=145.79.227.52:4447 , piece_id=26WZQBGHPBUMCEXYGAZULOGI2A2FE46LDL42ERWXUOO3BLPUSQUQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.479013264 (548h9m47.350096299s) [noise=true , address=145.79.227.26:4655 , piece_id=3EURLMKAKAXAV5E5WGP62QNP6N34FXSZVIICDES2IBH7URBI5XOQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.481790542 (548h9m47.352873587s) [noise=true , address=145.79.227.2:4114 , piece_id=WJRWUD35OW3S4NKXDWUT3F2X34JG6CWUGXDOKIUDNXEWTVAYB6JA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.482069744 (548h9m47.353152779s) [noise=true , address=45.91.116.4:4939 , piece_id=4IST6DJ5O3ORIO6BHOGFEURKSKAPPLXFNZPTDF35RAGOTFTAKWAQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.486092984 (548h9m47.357176019s) [noise=true , address=145.79.227.45:4231 , piece_id=H3DBZR6ENJUOFX7D3PXLW2SIY3LYHJAD2EI4BEEC2R4G5TA3CF7A, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.489275004 (548h9m47.360358049s) [noise=true , address=45.91.116.13:4156 , piece_id=XNBPVP6ITRD2XUUD7SFWW5DCY2AT5JMGG3WUX5SJS4GGHFDWJ3LQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.491105800 (548h9m47.362188845s) [noise=true , address=145.79.227.14:4839 , piece_id=5X62CRU2CW2MV4G5GQGUO4J4B2PY7YI6VQFQ7SXLG2C4R4Z2IPZQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:49.204036963 (548h9m49.075120008s) [upload=480657]: Next returned nil segment Dec 28 11:38:49.204082359 (548h9m49.075165395s) [upload=480657]: Waiting for error group managing segments... Dec 28 11:38:49.481600209 (548h9m49.352683244s) [noise=true , address=145.79.227.15:4875 , piece_id=64HMNDX7SUWKB6CDIE44XEOVCL523GP6BT3TK52GUJYCLWCWQMSQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.483491170 (548h9m49.354574215s) [noise=true , address=145.79.227.11:4729 , piece_id=THEOLKVVM2B4BS4H3J5EW3VFRAW36NM4Z65FVLBVA5JINHHBVYNQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.493301911 (548h9m49.364384946s) [noise=true , address=145.79.227.45:4231 , piece_id=H3DBZR6ENJUOFX7D3PXLW2SIY3LYHJAD2EI4BEEC2R4G5TA3CF7A, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.495779899 (548h9m49.366862944s) [noise=true , address=145.79.227.2:4114 , piece_id=WJRWUD35OW3S4NKXDWUT3F2X34JG6CWUGXDOKIUDNXEWTVAYB6JA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.498460213 (548h9m49.369543258s) [noise=true , address=145.79.227.14:4839 , piece_id=5X62CRU2CW2MV4G5GQGUO4J4B2PY7YI6VQFQ7SXLG2C4R4Z2IPZQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.499105783 (548h9m49.370188818s) [noise=true , address=145.79.227.26:4655 , piece_id=3EURLMKAKAXAV5E5WGP62QNP6N34FXSZVIICDES2IBH7URBI5XOQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.499901318 (548h9m49.370984363s) [noise=true , address=145.79.227.16:4911 , piece_id=TKJRVZZJHEBVRLEGLKSL6A43PELBQ5T2VSC6EKV4I77ISCP4KCKA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.507835054 (548h9m49.378918099s) [noise=true , address=145.79.227.39:4014 , piece_id=4RQHZIGPG5MSZWSJOXC25FA2GOWRTKVXSKXKT5MQNIQF52JVZURQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.516648407 (548h9m49.387731442s) [noise=true , address=145.79.227.47:4265 , piece_id=POXVT6RXKKWGQJSZZSB4Q733VQXBJGNZBNXWMXFEVYU6S6FUFSYA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.518784324 (548h9m49.389867359s) [noise=true , address=145.79.227.52:4447 , piece_id=26WZQBGHPBUMCEXYGAZULOGI2A2FE46LDL42ERWXUOO3BLPUSQUQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.731298456 (548h9m49.602381501s) [noise=true , address=45.91.116.4:4939 , piece_id=4IST6DJ5O3ORIO6BHOGFEURKSKAPPLXFNZPTDF35RAGOTFTAKWAQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.749791860 (548h9m49.620874895s) [noise=true , address=45.91.116.14:4190 , piece_id=JY6LQDWYSFGYTZFCBI7XKWPWLC7FP7HNZMY2OKLUUF6ISN7UCBIQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.762986425 (548h9m49.63406947s) [noise=true , address=45.91.116.13:4156 , piece_id=XNBPVP6ITRD2XUUD7SFWW5DCY2AT5JMGG3WUX5SJS4GGHFDWJ3LQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.763007395 (548h9m49.63409043s) [seg_pos={0 0}, upload=480657]: Segment reached optimal threshold of 13 pieces. Dec 28 11:38:49.763213838 (548h9m49.634296883s) [noise=true , address=45.91.116.9:4262 , piece_id=7BQ2CIDXBSY54FLSL6VO3JZC46BEJ3APNNXHLZMG5TPOE3LSPJ4A, seg_pos={0 0}, upload=480657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:45844->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:45844->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45844->45.91.116.9:4262: use of closed network connection Dec 28 11:38:49.763220290 (548h9m49.634303325s) [noise=true , address=45.91.116.9:4262 , piece_id=7BQ2CIDXBSY54FLSL6VO3JZC46BEJ3APNNXHLZMG5TPOE3LSPJ4A, seg_pos={0 0}, upload=480657]: Long tail cancellation due to optimal threshold Dec 28 11:38:49.763265466 (548h9m49.634348501s) [noise=true , address=45.91.116.11:4118 , piece_id=ZHWNRZJI73UJBC5Z5NZB7LSAO7KRNDT7R7AH2IFZLTLMR2AHVQPA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:53220->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:53220->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53220->45.91.116.11:4118: use of closed network connection Dec 28 11:38:49.763273912 (548h9m49.634356957s) [noise=true , address=45.91.116.11:4118 , piece_id=ZHWNRZJI73UJBC5Z5NZB7LSAO7KRNDT7R7AH2IFZLTLMR2AHVQPA, seg_pos={0 0}, upload=480657]: Long tail cancellation due to optimal threshold Dec 28 11:38:49.763427144 (548h9m49.634510179s) [noise=true , address=45.91.116.5:4297 , piece_id=AFKJWUGSB566MVE64PST3PSWZGK5VA3YOAI7ZUYKSSFGADLAQHAA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 11:38:49.763434107 (548h9m49.634517142s) [noise=true , address=45.91.116.5:4297 , piece_id=AFKJWUGSB566MVE64PST3PSWZGK5VA3YOAI7ZUYKSSFGADLAQHAA, seg_pos={0 0}, upload=480657]: Long tail cancellation due to optimal threshold Dec 28 11:38:49.763449908 (548h9m49.634532953s) [seg_pos={0 0}, upload=480657]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:38:49.763519551 (548h9m49.634602596s) [upload=480657]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:38:49.763530692 (548h9m49.634613737s) [upload=480657]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:38:50.053154022 (548h9m49.924237057s) [upload=480657]: Done uploading segments... Dec 28 11:38:50.053173539 (548h9m49.924256574s) [upload=480657]: Upload finished. err: <nil> Duration: 6.020226381s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................X........X...........X..X............] duration: 47.62684969s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:41:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 47.626876942s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XXX...X.XX.X..XX..X.......X.......................XX....X...] duration: 5.876496567s trace: https://jaeger.dev.storj.io/trace/2fefc5bfc20bdb01 FAIL (2025-12-28 11:41:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8h9m47.099694332s) [upload=480657]: Uploading segments... Dec 28 11:38:47.228817881 (548h9m47.099900926s) [upload=480657]: Got next segment. Inline: false Dec 28 11:38:47.228860011 (548h9m47.099943056s) [upload=480657]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:38:47.228865792 (548h9m47.099948827s) [upload=480657]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:38:47.473831737 (548h9m47.344914782s) [seg_pos={0 0}, upload=480657]: Begin upload segment... Dec 28 11:38:47.473887965 (548h9m47.34497101s) [seg_pos={0 0}, upload=480657]: Done begin upload segment. Dec 28 11:38:47.473925045 (548h9m47.34500809s) [noise=true , address=45.91.116.5:4297 , piece_id=AFKJWUGSB566MVE64PST3PSWZGK5VA3YOAI7ZUYKSSFGADLAQHAA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.474004907 (548h9m47.345087952s) [noise=true , address=145.79.227.47:4265 , piece_id=POXVT6RXKKWGQJSZZSB4Q733VQXBJGNZBNXWMXFEVYU6S6FUFSYA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.474561437 (548h9m47.345644472s) [noise=true , address=145.79.227.11:4729 , piece_id=THEOLKVVM2B4BS4H3J5EW3VFRAW36NM4Z65FVLBVA5JINHHBVYNQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.474866628 (548h9m47.345949663s) [noise=true , address=45.91.116.11:4118 , piece_id=ZHWNRZJI73UJBC5Z5NZB7LSAO7KRNDT7R7AH2IFZLTLMR2AHVQPA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.475139508 (548h9m47.346222543s) [noise=true , address=45.91.116.14:4190 , piece_id=JY6LQDWYSFGYTZFCBI7XKWPWLC7FP7HNZMY2OKLUUF6ISN7UCBIQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.475427065 (548h9m47.34651011s) [noise=true , address=45.91.116.9:4262 , piece_id=7BQ2CIDXBSY54FLSL6VO3JZC46BEJ3APNNXHLZMG5TPOE3LSPJ4A, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.475719562 (548h9m47.346802607s) [noise=true , address=145.79.227.15:4875 , piece_id=64HMNDX7SUWKB6CDIE44XEOVCL523GP6BT3TK52GUJYCLWCWQMSQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.476142047 (548h9m47.347225092s) [noise=true , address=145.79.227.39:4014 , piece_id=4RQHZIGPG5MSZWSJOXC25FA2GOWRTKVXSKXKT5MQNIQF52JVZURQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.476600019 (548h9m47.347683064s) [noise=true , address=145.79.227.16:4911 , piece_id=TKJRVZZJHEBVRLEGLKSL6A43PELBQ5T2VSC6EKV4I77ISCP4KCKA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.477869055 (548h9m47.34895209s) [noise=true , address=145.79.227.52:4447 , piece_id=26WZQBGHPBUMCEXYGAZULOGI2A2FE46LDL42ERWXUOO3BLPUSQUQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.479013264 (548h9m47.350096299s) [noise=true , address=145.79.227.26:4655 , piece_id=3EURLMKAKAXAV5E5WGP62QNP6N34FXSZVIICDES2IBH7URBI5XOQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.481790542 (548h9m47.352873587s) [noise=true , address=145.79.227.2:4114 , piece_id=WJRWUD35OW3S4NKXDWUT3F2X34JG6CWUGXDOKIUDNXEWTVAYB6JA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.482069744 (548h9m47.353152779s) [noise=true , address=45.91.116.4:4939 , piece_id=4IST6DJ5O3ORIO6BHOGFEURKSKAPPLXFNZPTDF35RAGOTFTAKWAQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.486092984 (548h9m47.357176019s) [noise=true , address=145.79.227.45:4231 , piece_id=H3DBZR6ENJUOFX7D3PXLW2SIY3LYHJAD2EI4BEEC2R4G5TA3CF7A, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.489275004 (548h9m47.360358049s) [noise=true , address=45.91.116.13:4156 , piece_id=XNBPVP6ITRD2XUUD7SFWW5DCY2AT5JMGG3WUX5SJS4GGHFDWJ3LQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.491105800 (548h9m47.362188845s) [noise=true , address=145.79.227.14:4839 , piece_id=5X62CRU2CW2MV4G5GQGUO4J4B2PY7YI6VQFQ7SXLG2C4R4Z2IPZQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:49.204036963 (548h9m49.075120008s) [upload=480657]: Next returned nil segment Dec 28 11:38:49.204082359 (548h9m49.075165395s) [upload=480657]: Waiting for error group managing segments... Dec 28 11:38:49.481600209 (548h9m49.352683244s) [noise=true , address=145.79.227.15:4875 , piece_id=64HMNDX7SUWKB6CDIE44XEOVCL523GP6BT3TK52GUJYCLWCWQMSQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.483491170 (548h9m49.354574215s) [noise=true , address=145.79.227.11:4729 , piece_id=THEOLKVVM2B4BS4H3J5EW3VFRAW36NM4Z65FVLBVA5JINHHBVYNQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.493301911 (548h9m49.364384946s) [noise=true , address=145.79.227.45:4231 , piece_id=H3DBZR6ENJUOFX7D3PXLW2SIY3LYHJAD2EI4BEEC2R4G5TA3CF7A, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.495779899 (548h9m49.366862944s) [noise=true , address=145.79.227.2:4114 , piece_id=WJRWUD35OW3S4NKXDWUT3F2X34JG6CWUGXDOKIUDNXEWTVAYB6JA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.498460213 (548h9m49.369543258s) [noise=true , address=145.79.227.14:4839 , piece_id=5X62CRU2CW2MV4G5GQGUO4J4B2PY7YI6VQFQ7SXLG2C4R4Z2IPZQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.499105783 (548h9m49.370188818s) [noise=true , address=145.79.227.26:4655 , piece_id=3EURLMKAKAXAV5E5WGP62QNP6N34FXSZVIICDES2IBH7URBI5XOQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.499901318 (548h9m49.370984363s) [noise=true , address=145.79.227.16:4911 , piece_id=TKJRVZZJHEBVRLEGLKSL6A43PELBQ5T2VSC6EKV4I77ISCP4KCKA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.507835054 (548h9m49.378918099s) [noise=true , address=145.79.227.39:4014 , piece_id=4RQHZIGPG5MSZWSJOXC25FA2GOWRTKVXSKXKT5MQNIQF52JVZURQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.516648407 (548h9m49.387731442s) [noise=true , address=145.79.227.47:4265 , piece_id=POXVT6RXKKWGQJSZZSB4Q733VQXBJGNZBNXWMXFEVYU6S6FUFSYA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.518784324 (548h9m49.389867359s) [noise=true , address=145.79.227.52:4447 , piece_id=26WZQBGHPBUMCEXYGAZULOGI2A2FE46LDL42ERWXUOO3BLPUSQUQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.731298456 (548h9m49.602381501s) [noise=true , address=45.91.116.4:4939 , piece_id=4IST6DJ5O3ORIO6BHOGFEURKSKAPPLXFNZPTDF35RAGOTFTAKWAQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.749791860 (548h9m49.620874895s) [noise=true , address=45.91.116.14:4190 , piece_id=JY6LQDWYSFGYTZFCBI7XKWPWLC7FP7HNZMY2OKLUUF6ISN7UCBIQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.762986425 (548h9m49.63406947s) [noise=true , address=45.91.116.13:4156 , piece_id=XNBPVP6ITRD2XUUD7SFWW5DCY2AT5JMGG3WUX5SJS4GGHFDWJ3LQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.763007395 (548h9m49.63409043s) [seg_pos={0 0}, upload=480657]: Segment reached optimal threshold of 13 pieces. Dec 28 11:38:49.763213838 (548h9m49.634296883s) [noise=true , address=45.91.116.9:4262 , piece_id=7BQ2CIDXBSY54FLSL6VO3JZC46BEJ3APNNXHLZMG5TPOE3LSPJ4A, seg_pos={0 0}, upload=480657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:45844->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:45844->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45844->45.91.116.9:4262: use of closed network connection Dec 28 11:38:49.763220290 (548h9m49.634303325s) [noise=true , address=45.91.116.9:4262 , piece_id=7BQ2CIDXBSY54FLSL6VO3JZC46BEJ3APNNXHLZMG5TPOE3LSPJ4A, seg_pos={0 0}, upload=480657]: Long tail cancellation due to optimal threshold Dec 28 11:38:49.763265466 (548h9m49.634348501s) [noise=true , address=45.91.116.11:4118 , piece_id=ZHWNRZJI73UJBC5Z5NZB7LSAO7KRNDT7R7AH2IFZLTLMR2AHVQPA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:53220->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:53220->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53220->45.91.116.11:4118: use of closed network connection Dec 28 11:38:49.763273912 (548h9m49.634356957s) [noise=true , address=45.91.116.11:4118 , piece_id=ZHWNRZJI73UJBC5Z5NZB7LSAO7KRNDT7R7AH2IFZLTLMR2AHVQPA, seg_pos={0 0}, upload=480657]: Long tail cancellation due to optimal threshold Dec 28 11:38:49.763427144 (548h9m49.634510179s) [noise=true , address=45.91.116.5:4297 , piece_id=AFKJWUGSB566MVE64PST3PSWZGK5VA3YOAI7ZUYKSSFGADLAQHAA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 11:38:49.763434107 (548h9m49.634517142s) [noise=true , address=45.91.116.5:4297 , piece_id=AFKJWUGSB566MVE64PST3PSWZGK5VA3YOAI7ZUYKSSFGADLAQHAA, seg_pos={0 0}, upload=480657]: Long tail cancellation due to optimal threshold Dec 28 11:38:49.763449908 (548h9m49.634532953s) [seg_pos={0 0}, upload=480657]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:38:49.763519551 (548h9m49.634602596s) [upload=480657]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:38:49.763530692 (548h9m49.634613737s) [upload=480657]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:38:50.053154022 (548h9m49.924237057s) [upload=480657]: Done uploading segments... Dec 28 11:38:50.053173539 (548h9m49.924256574s) [upload=480657]: Upload finished. err: <nil> Duration: 5.876526124s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [..X.....XX.XX..XXX...X.X....X..XXXX..X.X.XX.....X.X.X..X.XXX] duration: 5.000820701s trace: https://jaeger.dev.storj.io/trace/496a4e47ee73e6d8 FAIL (2025-12-28 11:40:58 UTC) orbiter warning: context deadline exceeded Duration: 5.000845228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.XXX...X.XX.X..XX..X.......X.......................XX....X..] duration: 5.952308184s trace: https://jaeger.dev.storj.io/trace/2fefc5bfc20bdb01 FAIL (2025-12-28 11:40:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8h9m47.099694332s) [upload=480657]: Uploading segments... Dec 28 11:38:47.228817881 (548h9m47.099900926s) [upload=480657]: Got next segment. Inline: false Dec 28 11:38:47.228860011 (548h9m47.099943056s) [upload=480657]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:38:47.228865792 (548h9m47.099948827s) [upload=480657]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:38:47.473831737 (548h9m47.344914782s) [seg_pos={0 0}, upload=480657]: Begin upload segment... Dec 28 11:38:47.473887965 (548h9m47.34497101s) [seg_pos={0 0}, upload=480657]: Done begin upload segment. Dec 28 11:38:47.473925045 (548h9m47.34500809s) [noise=true , address=45.91.116.5:4297 , piece_id=AFKJWUGSB566MVE64PST3PSWZGK5VA3YOAI7ZUYKSSFGADLAQHAA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.474004907 (548h9m47.345087952s) [noise=true , address=145.79.227.47:4265 , piece_id=POXVT6RXKKWGQJSZZSB4Q733VQXBJGNZBNXWMXFEVYU6S6FUFSYA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.474561437 (548h9m47.345644472s) [noise=true , address=145.79.227.11:4729 , piece_id=THEOLKVVM2B4BS4H3J5EW3VFRAW36NM4Z65FVLBVA5JINHHBVYNQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.474866628 (548h9m47.345949663s) [noise=true , address=45.91.116.11:4118 , piece_id=ZHWNRZJI73UJBC5Z5NZB7LSAO7KRNDT7R7AH2IFZLTLMR2AHVQPA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.475139508 (548h9m47.346222543s) [noise=true , address=45.91.116.14:4190 , piece_id=JY6LQDWYSFGYTZFCBI7XKWPWLC7FP7HNZMY2OKLUUF6ISN7UCBIQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.475427065 (548h9m47.34651011s) [noise=true , address=45.91.116.9:4262 , piece_id=7BQ2CIDXBSY54FLSL6VO3JZC46BEJ3APNNXHLZMG5TPOE3LSPJ4A, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.475719562 (548h9m47.346802607s) [noise=true , address=145.79.227.15:4875 , piece_id=64HMNDX7SUWKB6CDIE44XEOVCL523GP6BT3TK52GUJYCLWCWQMSQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.476142047 (548h9m47.347225092s) [noise=true , address=145.79.227.39:4014 , piece_id=4RQHZIGPG5MSZWSJOXC25FA2GOWRTKVXSKXKT5MQNIQF52JVZURQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.476600019 (548h9m47.347683064s) [noise=true , address=145.79.227.16:4911 , piece_id=TKJRVZZJHEBVRLEGLKSL6A43PELBQ5T2VSC6EKV4I77ISCP4KCKA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.477869055 (548h9m47.34895209s) [noise=true , address=145.79.227.52:4447 , piece_id=26WZQBGHPBUMCEXYGAZULOGI2A2FE46LDL42ERWXUOO3BLPUSQUQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.479013264 (548h9m47.350096299s) [noise=true , address=145.79.227.26:4655 , piece_id=3EURLMKAKAXAV5E5WGP62QNP6N34FXSZVIICDES2IBH7URBI5XOQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.481790542 (548h9m47.352873587s) [noise=true , address=145.79.227.2:4114 , piece_id=WJRWUD35OW3S4NKXDWUT3F2X34JG6CWUGXDOKIUDNXEWTVAYB6JA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.482069744 (548h9m47.353152779s) [noise=true , address=45.91.116.4:4939 , piece_id=4IST6DJ5O3ORIO6BHOGFEURKSKAPPLXFNZPTDF35RAGOTFTAKWAQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.486092984 (548h9m47.357176019s) [noise=true , address=145.79.227.45:4231 , piece_id=H3DBZR6ENJUOFX7D3PXLW2SIY3LYHJAD2EI4BEEC2R4G5TA3CF7A, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.489275004 (548h9m47.360358049s) [noise=true , address=45.91.116.13:4156 , piece_id=XNBPVP6ITRD2XUUD7SFWW5DCY2AT5JMGG3WUX5SJS4GGHFDWJ3LQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.491105800 (548h9m47.362188845s) [noise=true , address=145.79.227.14:4839 , piece_id=5X62CRU2CW2MV4G5GQGUO4J4B2PY7YI6VQFQ7SXLG2C4R4Z2IPZQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:49.204036963 (548h9m49.075120008s) [upload=480657]: Next returned nil segment Dec 28 11:38:49.204082359 (548h9m49.075165395s) [upload=480657]: Waiting for error group managing segments... Dec 28 11:38:49.481600209 (548h9m49.352683244s) [noise=true , address=145.79.227.15:4875 , piece_id=64HMNDX7SUWKB6CDIE44XEOVCL523GP6BT3TK52GUJYCLWCWQMSQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.483491170 (548h9m49.354574215s) [noise=true , address=145.79.227.11:4729 , piece_id=THEOLKVVM2B4BS4H3J5EW3VFRAW36NM4Z65FVLBVA5JINHHBVYNQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.493301911 (548h9m49.364384946s) [noise=true , address=145.79.227.45:4231 , piece_id=H3DBZR6ENJUOFX7D3PXLW2SIY3LYHJAD2EI4BEEC2R4G5TA3CF7A, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.495779899 (548h9m49.366862944s) [noise=true , address=145.79.227.2:4114 , piece_id=WJRWUD35OW3S4NKXDWUT3F2X34JG6CWUGXDOKIUDNXEWTVAYB6JA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.498460213 (548h9m49.369543258s) [noise=true , address=145.79.227.14:4839 , piece_id=5X62CRU2CW2MV4G5GQGUO4J4B2PY7YI6VQFQ7SXLG2C4R4Z2IPZQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.499105783 (548h9m49.370188818s) [noise=true , address=145.79.227.26:4655 , piece_id=3EURLMKAKAXAV5E5WGP62QNP6N34FXSZVIICDES2IBH7URBI5XOQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.499901318 (548h9m49.370984363s) [noise=true , address=145.79.227.16:4911 , piece_id=TKJRVZZJHEBVRLEGLKSL6A43PELBQ5T2VSC6EKV4I77ISCP4KCKA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.507835054 (548h9m49.378918099s) [noise=true , address=145.79.227.39:4014 , piece_id=4RQHZIGPG5MSZWSJOXC25FA2GOWRTKVXSKXKT5MQNIQF52JVZURQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.516648407 (548h9m49.387731442s) [noise=true , address=145.79.227.47:4265 , piece_id=POXVT6RXKKWGQJSZZSB4Q733VQXBJGNZBNXWMXFEVYU6S6FUFSYA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.518784324 (548h9m49.389867359s) [noise=true , address=145.79.227.52:4447 , piece_id=26WZQBGHPBUMCEXYGAZULOGI2A2FE46LDL42ERWXUOO3BLPUSQUQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.731298456 (548h9m49.602381501s) [noise=true , address=45.91.116.4:4939 , piece_id=4IST6DJ5O3ORIO6BHOGFEURKSKAPPLXFNZPTDF35RAGOTFTAKWAQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.749791860 (548h9m49.620874895s) [noise=true , address=45.91.116.14:4190 , piece_id=JY6LQDWYSFGYTZFCBI7XKWPWLC7FP7HNZMY2OKLUUF6ISN7UCBIQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.762986425 (548h9m49.63406947s) [noise=true , address=45.91.116.13:4156 , piece_id=XNBPVP6ITRD2XUUD7SFWW5DCY2AT5JMGG3WUX5SJS4GGHFDWJ3LQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.763007395 (548h9m49.63409043s) [seg_pos={0 0}, upload=480657]: Segment reached optimal threshold of 13 pieces. Dec 28 11:38:49.763213838 (548h9m49.634296883s) [noise=true , address=45.91.116.9:4262 , piece_id=7BQ2CIDXBSY54FLSL6VO3JZC46BEJ3APNNXHLZMG5TPOE3LSPJ4A, seg_pos={0 0}, upload=480657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:45844->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:45844->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45844->45.91.116.9:4262: use of closed network connection Dec 28 11:38:49.763220290 (548h9m49.634303325s) [noise=true , address=45.91.116.9:4262 , piece_id=7BQ2CIDXBSY54FLSL6VO3JZC46BEJ3APNNXHLZMG5TPOE3LSPJ4A, seg_pos={0 0}, upload=480657]: Long tail cancellation due to optimal threshold Dec 28 11:38:49.763265466 (548h9m49.634348501s) [noise=true , address=45.91.116.11:4118 , piece_id=ZHWNRZJI73UJBC5Z5NZB7LSAO7KRNDT7R7AH2IFZLTLMR2AHVQPA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:53220->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:53220->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53220->45.91.116.11:4118: use of closed network connection Dec 28 11:38:49.763273912 (548h9m49.634356957s) [noise=true , address=45.91.116.11:4118 , piece_id=ZHWNRZJI73UJBC5Z5NZB7LSAO7KRNDT7R7AH2IFZLTLMR2AHVQPA, seg_pos={0 0}, upload=480657]: Long tail cancellation due to optimal threshold Dec 28 11:38:49.763427144 (548h9m49.634510179s) [noise=true , address=45.91.116.5:4297 , piece_id=AFKJWUGSB566MVE64PST3PSWZGK5VA3YOAI7ZUYKSSFGADLAQHAA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 11:38:49.763434107 (548h9m49.634517142s) [noise=true , address=45.91.116.5:4297 , piece_id=AFKJWUGSB566MVE64PST3PSWZGK5VA3YOAI7ZUYKSSFGADLAQHAA, seg_pos={0 0}, upload=480657]: Long tail cancellation due to optimal threshold Dec 28 11:38:49.763449908 (548h9m49.634532953s) [seg_pos={0 0}, upload=480657]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:38:49.763519551 (548h9m49.634602596s) [upload=480657]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:38:49.763530692 (548h9m49.634613737s) [upload=480657]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:38:50.053154022 (548h9m49.924237057s) [upload=480657]: Done uploading segments... Dec 28 11:38:50.053173539 (548h9m49.924256574s) [upload=480657]: Upload finished. err: <nil> Duration: 5.952336098s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [...X.....XX.XX..XXX...X.X....X..XXXX..X.X.XX.....X.X.X..X.XX] duration: 5.000783675s trace: https://jaeger.dev.storj.io/trace/3f519586aaa50c5a FAIL (2025-12-28 11:40:12 UTC) orbiter warning: context deadline exceeded Duration: 5.000805387s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................X........X...........X..X...........] duration: 41.979974354s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:39:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 41.980002298s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [X...X.....XX.XX..XXX...X.X....X..XXXX..X.X.XX.....X.X.X..X.X] duration: 5.001210298s trace: https://jaeger.dev.storj.io/trace/14e98ab02403ae10 FAIL (2025-12-28 11:39:30 UTC) orbiter warning: context deadline exceeded Duration: 5.001235825s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..XXX...X.XX.X..XX..X.......X.......................XX....X.] duration: 5.67309556s trace: https://jaeger.dev.storj.io/trace/2fefc5bfc20bdb01 FAIL (2025-12-28 11:39:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8h9m47.099694332s) [upload=480657]: Uploading segments... Dec 28 11:38:47.228817881 (548h9m47.099900926s) [upload=480657]: Got next segment. Inline: false Dec 28 11:38:47.228860011 (548h9m47.099943056s) [upload=480657]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:38:47.228865792 (548h9m47.099948827s) [upload=480657]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:38:47.473831737 (548h9m47.344914782s) [seg_pos={0 0}, upload=480657]: Begin upload segment... Dec 28 11:38:47.473887965 (548h9m47.34497101s) [seg_pos={0 0}, upload=480657]: Done begin upload segment. Dec 28 11:38:47.473925045 (548h9m47.34500809s) [noise=true , address=45.91.116.5:4297 , piece_id=AFKJWUGSB566MVE64PST3PSWZGK5VA3YOAI7ZUYKSSFGADLAQHAA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.474004907 (548h9m47.345087952s) [noise=true , address=145.79.227.47:4265 , piece_id=POXVT6RXKKWGQJSZZSB4Q733VQXBJGNZBNXWMXFEVYU6S6FUFSYA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.474561437 (548h9m47.345644472s) [noise=true , address=145.79.227.11:4729 , piece_id=THEOLKVVM2B4BS4H3J5EW3VFRAW36NM4Z65FVLBVA5JINHHBVYNQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.474866628 (548h9m47.345949663s) [noise=true , address=45.91.116.11:4118 , piece_id=ZHWNRZJI73UJBC5Z5NZB7LSAO7KRNDT7R7AH2IFZLTLMR2AHVQPA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.475139508 (548h9m47.346222543s) [noise=true , address=45.91.116.14:4190 , piece_id=JY6LQDWYSFGYTZFCBI7XKWPWLC7FP7HNZMY2OKLUUF6ISN7UCBIQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.475427065 (548h9m47.34651011s) [noise=true , address=45.91.116.9:4262 , piece_id=7BQ2CIDXBSY54FLSL6VO3JZC46BEJ3APNNXHLZMG5TPOE3LSPJ4A, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.475719562 (548h9m47.346802607s) [noise=true , address=145.79.227.15:4875 , piece_id=64HMNDX7SUWKB6CDIE44XEOVCL523GP6BT3TK52GUJYCLWCWQMSQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.476142047 (548h9m47.347225092s) [noise=true , address=145.79.227.39:4014 , piece_id=4RQHZIGPG5MSZWSJOXC25FA2GOWRTKVXSKXKT5MQNIQF52JVZURQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.476600019 (548h9m47.347683064s) [noise=true , address=145.79.227.16:4911 , piece_id=TKJRVZZJHEBVRLEGLKSL6A43PELBQ5T2VSC6EKV4I77ISCP4KCKA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.477869055 (548h9m47.34895209s) [noise=true , address=145.79.227.52:4447 , piece_id=26WZQBGHPBUMCEXYGAZULOGI2A2FE46LDL42ERWXUOO3BLPUSQUQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.479013264 (548h9m47.350096299s) [noise=true , address=145.79.227.26:4655 , piece_id=3EURLMKAKAXAV5E5WGP62QNP6N34FXSZVIICDES2IBH7URBI5XOQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.481790542 (548h9m47.352873587s) [noise=true , address=145.79.227.2:4114 , piece_id=WJRWUD35OW3S4NKXDWUT3F2X34JG6CWUGXDOKIUDNXEWTVAYB6JA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.482069744 (548h9m47.353152779s) [noise=true , address=45.91.116.4:4939 , piece_id=4IST6DJ5O3ORIO6BHOGFEURKSKAPPLXFNZPTDF35RAGOTFTAKWAQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.486092984 (548h9m47.357176019s) [noise=true , address=145.79.227.45:4231 , piece_id=H3DBZR6ENJUOFX7D3PXLW2SIY3LYHJAD2EI4BEEC2R4G5TA3CF7A, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.489275004 (548h9m47.360358049s) [noise=true , address=45.91.116.13:4156 , piece_id=XNBPVP6ITRD2XUUD7SFWW5DCY2AT5JMGG3WUX5SJS4GGHFDWJ3LQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.491105800 (548h9m47.362188845s) [noise=true , address=145.79.227.14:4839 , piece_id=5X62CRU2CW2MV4G5GQGUO4J4B2PY7YI6VQFQ7SXLG2C4R4Z2IPZQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:49.204036963 (548h9m49.075120008s) [upload=480657]: Next returned nil segment Dec 28 11:38:49.204082359 (548h9m49.075165395s) [upload=480657]: Waiting for error group managing segments... Dec 28 11:38:49.481600209 (548h9m49.352683244s) [noise=true , address=145.79.227.15:4875 , piece_id=64HMNDX7SUWKB6CDIE44XEOVCL523GP6BT3TK52GUJYCLWCWQMSQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.483491170 (548h9m49.354574215s) [noise=true , address=145.79.227.11:4729 , piece_id=THEOLKVVM2B4BS4H3J5EW3VFRAW36NM4Z65FVLBVA5JINHHBVYNQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.493301911 (548h9m49.364384946s) [noise=true , address=145.79.227.45:4231 , piece_id=H3DBZR6ENJUOFX7D3PXLW2SIY3LYHJAD2EI4BEEC2R4G5TA3CF7A, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.495779899 (548h9m49.366862944s) [noise=true , address=145.79.227.2:4114 , piece_id=WJRWUD35OW3S4NKXDWUT3F2X34JG6CWUGXDOKIUDNXEWTVAYB6JA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.498460213 (548h9m49.369543258s) [noise=true , address=145.79.227.14:4839 , piece_id=5X62CRU2CW2MV4G5GQGUO4J4B2PY7YI6VQFQ7SXLG2C4R4Z2IPZQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.499105783 (548h9m49.370188818s) [noise=true , address=145.79.227.26:4655 , piece_id=3EURLMKAKAXAV5E5WGP62QNP6N34FXSZVIICDES2IBH7URBI5XOQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.499901318 (548h9m49.370984363s) [noise=true , address=145.79.227.16:4911 , piece_id=TKJRVZZJHEBVRLEGLKSL6A43PELBQ5T2VSC6EKV4I77ISCP4KCKA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.507835054 (548h9m49.378918099s) [noise=true , address=145.79.227.39:4014 , piece_id=4RQHZIGPG5MSZWSJOXC25FA2GOWRTKVXSKXKT5MQNIQF52JVZURQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.516648407 (548h9m49.387731442s) [noise=true , address=145.79.227.47:4265 , piece_id=POXVT6RXKKWGQJSZZSB4Q733VQXBJGNZBNXWMXFEVYU6S6FUFSYA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.518784324 (548h9m49.389867359s) [noise=true , address=145.79.227.52:4447 , piece_id=26WZQBGHPBUMCEXYGAZULOGI2A2FE46LDL42ERWXUOO3BLPUSQUQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.731298456 (548h9m49.602381501s) [noise=true , address=45.91.116.4:4939 , piece_id=4IST6DJ5O3ORIO6BHOGFEURKSKAPPLXFNZPTDF35RAGOTFTAKWAQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.749791860 (548h9m49.620874895s) [noise=true , address=45.91.116.14:4190 , piece_id=JY6LQDWYSFGYTZFCBI7XKWPWLC7FP7HNZMY2OKLUUF6ISN7UCBIQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.762986425 (548h9m49.63406947s) [noise=true , address=45.91.116.13:4156 , piece_id=XNBPVP6ITRD2XUUD7SFWW5DCY2AT5JMGG3WUX5SJS4GGHFDWJ3LQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.763007395 (548h9m49.63409043s) [seg_pos={0 0}, upload=480657]: Segment reached optimal threshold of 13 pieces. Dec 28 11:38:49.763213838 (548h9m49.634296883s) [noise=true , address=45.91.116.9:4262 , piece_id=7BQ2CIDXBSY54FLSL6VO3JZC46BEJ3APNNXHLZMG5TPOE3LSPJ4A, seg_pos={0 0}, upload=480657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:45844->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:45844->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45844->45.91.116.9:4262: use of closed network connection Dec 28 11:38:49.763220290 (548h9m49.634303325s) [noise=true , address=45.91.116.9:4262 , piece_id=7BQ2CIDXBSY54FLSL6VO3JZC46BEJ3APNNXHLZMG5TPOE3LSPJ4A, seg_pos={0 0}, upload=480657]: Long tail cancellation due to optimal threshold Dec 28 11:38:49.763265466 (548h9m49.634348501s) [noise=true , address=45.91.116.11:4118 , piece_id=ZHWNRZJI73UJBC5Z5NZB7LSAO7KRNDT7R7AH2IFZLTLMR2AHVQPA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:53220->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:53220->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53220->45.91.116.11:4118: use of closed network connection Dec 28 11:38:49.763273912 (548h9m49.634356957s) [noise=true , address=45.91.116.11:4118 , piece_id=ZHWNRZJI73UJBC5Z5NZB7LSAO7KRNDT7R7AH2IFZLTLMR2AHVQPA, seg_pos={0 0}, upload=480657]: Long tail cancellation due to optimal threshold Dec 28 11:38:49.763427144 (548h9m49.634510179s) [noise=true , address=45.91.116.5:4297 , piece_id=AFKJWUGSB566MVE64PST3PSWZGK5VA3YOAI7ZUYKSSFGADLAQHAA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 11:38:49.763434107 (548h9m49.634517142s) [noise=true , address=45.91.116.5:4297 , piece_id=AFKJWUGSB566MVE64PST3PSWZGK5VA3YOAI7ZUYKSSFGADLAQHAA, seg_pos={0 0}, upload=480657]: Long tail cancellation due to optimal threshold Dec 28 11:38:49.763449908 (548h9m49.634532953s) [seg_pos={0 0}, upload=480657]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:38:49.763519551 (548h9m49.634602596s) [upload=480657]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:38:49.763530692 (548h9m49.634613737s) [upload=480657]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:38:50.053154022 (548h9m49.924237057s) [upload=480657]: Done uploading segments... Dec 28 11:38:50.053173539 (548h9m49.924256574s) [upload=480657]: Upload finished. err: <nil> Duration: 5.673116889s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...XXX...X.XX.X..XX..X.......X.......................XX....X] duration: 30.003791985s trace: https://jaeger.dev.storj.io/trace/2fefc5bfc20bdb01 FAIL (2025-12-28 11:39:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8h9m47.099694332s) [upload=480657]: Uploading segments... Dec 28 11:38:47.228817881 (548h9m47.099900926s) [upload=480657]: Got next segment. Inline: false Dec 28 11:38:47.228860011 (548h9m47.099943056s) [upload=480657]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:38:47.228865792 (548h9m47.099948827s) [upload=480657]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:38:47.473831737 (548h9m47.344914782s) [seg_pos={0 0}, upload=480657]: Begin upload segment... Dec 28 11:38:47.473887965 (548h9m47.34497101s) [seg_pos={0 0}, upload=480657]: Done begin upload segment. Dec 28 11:38:47.473925045 (548h9m47.34500809s) [noise=true , address=45.91.116.5:4297 , piece_id=AFKJWUGSB566MVE64PST3PSWZGK5VA3YOAI7ZUYKSSFGADLAQHAA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.474004907 (548h9m47.345087952s) [noise=true , address=145.79.227.47:4265 , piece_id=POXVT6RXKKWGQJSZZSB4Q733VQXBJGNZBNXWMXFEVYU6S6FUFSYA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.474561437 (548h9m47.345644472s) [noise=true , address=145.79.227.11:4729 , piece_id=THEOLKVVM2B4BS4H3J5EW3VFRAW36NM4Z65FVLBVA5JINHHBVYNQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.474866628 (548h9m47.345949663s) [noise=true , address=45.91.116.11:4118 , piece_id=ZHWNRZJI73UJBC5Z5NZB7LSAO7KRNDT7R7AH2IFZLTLMR2AHVQPA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.475139508 (548h9m47.346222543s) [noise=true , address=45.91.116.14:4190 , piece_id=JY6LQDWYSFGYTZFCBI7XKWPWLC7FP7HNZMY2OKLUUF6ISN7UCBIQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.475427065 (548h9m47.34651011s) [noise=true , address=45.91.116.9:4262 , piece_id=7BQ2CIDXBSY54FLSL6VO3JZC46BEJ3APNNXHLZMG5TPOE3LSPJ4A, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.475719562 (548h9m47.346802607s) [noise=true , address=145.79.227.15:4875 , piece_id=64HMNDX7SUWKB6CDIE44XEOVCL523GP6BT3TK52GUJYCLWCWQMSQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.476142047 (548h9m47.347225092s) [noise=true , address=145.79.227.39:4014 , piece_id=4RQHZIGPG5MSZWSJOXC25FA2GOWRTKVXSKXKT5MQNIQF52JVZURQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.476600019 (548h9m47.347683064s) [noise=true , address=145.79.227.16:4911 , piece_id=TKJRVZZJHEBVRLEGLKSL6A43PELBQ5T2VSC6EKV4I77ISCP4KCKA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.477869055 (548h9m47.34895209s) [noise=true , address=145.79.227.52:4447 , piece_id=26WZQBGHPBUMCEXYGAZULOGI2A2FE46LDL42ERWXUOO3BLPUSQUQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.479013264 (548h9m47.350096299s) [noise=true , address=145.79.227.26:4655 , piece_id=3EURLMKAKAXAV5E5WGP62QNP6N34FXSZVIICDES2IBH7URBI5XOQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.481790542 (548h9m47.352873587s) [noise=true , address=145.79.227.2:4114 , piece_id=WJRWUD35OW3S4NKXDWUT3F2X34JG6CWUGXDOKIUDNXEWTVAYB6JA, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.482069744 (548h9m47.353152779s) [noise=true , address=45.91.116.4:4939 , piece_id=4IST6DJ5O3ORIO6BHOGFEURKSKAPPLXFNZPTDF35RAGOTFTAKWAQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.486092984 (548h9m47.357176019s) [noise=true , address=145.79.227.45:4231 , piece_id=H3DBZR6ENJUOFX7D3PXLW2SIY3LYHJAD2EI4BEEC2R4G5TA3CF7A, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.489275004 (548h9m47.360358049s) [noise=true , address=45.91.116.13:4156 , piece_id=XNBPVP6ITRD2XUUD7SFWW5DCY2AT5JMGG3WUX5SJS4GGHFDWJ3LQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:47.491105800 (548h9m47.362188845s) [noise=true , address=145.79.227.14:4839 , piece_id=5X62CRU2CW2MV4G5GQGUO4J4B2PY7YI6VQFQ7SXLG2C4R4Z2IPZQ, seg_pos={0 0}, upload=480657]: Uploading piece... Dec 28 11:38:49.204036963 (548h9m49.075120008s) [upload=480657]: Next returned nil segment Dec 28 11:38:49.204082359 (548h9m49.075165395s) [upload=480657]: Waiting for error group managing segments... Dec 28 11:38:49.481600209 (548h9m49.352683244s) [noise=true , address=145.79.227.15:4875 , piece_id=64HMNDX7SUWKB6CDIE44XEOVCL523GP6BT3TK52GUJYCLWCWQMSQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.483491170 (548h9m49.354574215s) [noise=true , address=145.79.227.11:4729 , piece_id=THEOLKVVM2B4BS4H3J5EW3VFRAW36NM4Z65FVLBVA5JINHHBVYNQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.493301911 (548h9m49.364384946s) [noise=true , address=145.79.227.45:4231 , piece_id=H3DBZR6ENJUOFX7D3PXLW2SIY3LYHJAD2EI4BEEC2R4G5TA3CF7A, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.495779899 (548h9m49.366862944s) [noise=true , address=145.79.227.2:4114 , piece_id=WJRWUD35OW3S4NKXDWUT3F2X34JG6CWUGXDOKIUDNXEWTVAYB6JA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.498460213 (548h9m49.369543258s) [noise=true , address=145.79.227.14:4839 , piece_id=5X62CRU2CW2MV4G5GQGUO4J4B2PY7YI6VQFQ7SXLG2C4R4Z2IPZQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.499105783 (548h9m49.370188818s) [noise=true , address=145.79.227.26:4655 , piece_id=3EURLMKAKAXAV5E5WGP62QNP6N34FXSZVIICDES2IBH7URBI5XOQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.499901318 (548h9m49.370984363s) [noise=true , address=145.79.227.16:4911 , piece_id=TKJRVZZJHEBVRLEGLKSL6A43PELBQ5T2VSC6EKV4I77ISCP4KCKA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.507835054 (548h9m49.378918099s) [noise=true , address=145.79.227.39:4014 , piece_id=4RQHZIGPG5MSZWSJOXC25FA2GOWRTKVXSKXKT5MQNIQF52JVZURQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.516648407 (548h9m49.387731442s) [noise=true , address=145.79.227.47:4265 , piece_id=POXVT6RXKKWGQJSZZSB4Q733VQXBJGNZBNXWMXFEVYU6S6FUFSYA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.518784324 (548h9m49.389867359s) [noise=true , address=145.79.227.52:4447 , piece_id=26WZQBGHPBUMCEXYGAZULOGI2A2FE46LDL42ERWXUOO3BLPUSQUQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.731298456 (548h9m49.602381501s) [noise=true , address=45.91.116.4:4939 , piece_id=4IST6DJ5O3ORIO6BHOGFEURKSKAPPLXFNZPTDF35RAGOTFTAKWAQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.749791860 (548h9m49.620874895s) [noise=true , address=45.91.116.14:4190 , piece_id=JY6LQDWYSFGYTZFCBI7XKWPWLC7FP7HNZMY2OKLUUF6ISN7UCBIQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.762986425 (548h9m49.63406947s) [noise=true , address=45.91.116.13:4156 , piece_id=XNBPVP6ITRD2XUUD7SFWW5DCY2AT5JMGG3WUX5SJS4GGHFDWJ3LQ, seg_pos={0 0}, upload=480657]: Done uploading piece. err: <nil> Dec 28 11:38:49.763007395 (548h9m49.63409043s) [seg_pos={0 0}, upload=480657]: Segment reached optimal threshold of 13 pieces. Dec 28 11:38:49.763213838 (548h9m49.634296883s) [noise=true , address=45.91.116.9:4262 , piece_id=7BQ2CIDXBSY54FLSL6VO3JZC46BEJ3APNNXHLZMG5TPOE3LSPJ4A, seg_pos={0 0}, upload=480657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:45844->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:45844->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45844->45.91.116.9:4262: use of closed network connection Dec 28 11:38:49.763220290 (548h9m49.634303325s) [noise=true , address=45.91.116.9:4262 , piece_id=7BQ2CIDXBSY54FLSL6VO3JZC46BEJ3APNNXHLZMG5TPOE3LSPJ4A, seg_pos={0 0}, upload=480657]: Long tail cancellation due to optimal threshold Dec 28 11:38:49.763265466 (548h9m49.634348501s) [noise=true , address=45.91.116.11:4118 , piece_id=ZHWNRZJI73UJBC5Z5NZB7LSAO7KRNDT7R7AH2IFZLTLMR2AHVQPA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:53220->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:53220->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53220->45.91.116.11:4118: use of closed network connection Dec 28 11:38:49.763273912 (548h9m49.634356957s) [noise=true , address=45.91.116.11:4118 , piece_id=ZHWNRZJI73UJBC5Z5NZB7LSAO7KRNDT7R7AH2IFZLTLMR2AHVQPA, seg_pos={0 0}, upload=480657]: Long tail cancellation due to optimal threshold Dec 28 11:38:49.763427144 (548h9m49.634510179s) [noise=true , address=45.91.116.5:4297 , piece_id=AFKJWUGSB566MVE64PST3PSWZGK5VA3YOAI7ZUYKSSFGADLAQHAA, seg_pos={0 0}, upload=480657]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 11:38:49.763434107 (548h9m49.634517142s) [noise=true , address=45.91.116.5:4297 , piece_id=AFKJWUGSB566MVE64PST3PSWZGK5VA3YOAI7ZUYKSSFGADLAQHAA, seg_pos={0 0}, upload=480657]: Long tail cancellation due to optimal threshold Dec 28 11:38:49.763449908 (548h9m49.634532953s) [seg_pos={0 0}, upload=480657]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:38:49.763519551 (548h9m49.634602596s) [upload=480657]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:38:49.763530692 (548h9m49.634613737s) [upload=480657]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:38:50.053154022 (548h9m49.924237057s) [upload=480657]: Done uploading segments... Dec 28 11:38:50.053173539 (548h9m49.924256574s) [upload=480657]: Upload finished. err: <nil> Duration: 30.003812815s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................X........X...........X..X..........] duration: 39.511544013s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:38:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 39.511578016s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.X...X.....XX.XX..XXX...X.X....X..XXXX..X.X.XX.....X.X.X..X.] duration: 3.492238766s trace: https://jaeger.dev.storj.io/trace/1de298e8089fbea9 FAIL (2025-12-28 11:38:25 UTC) orbiter warning: context deadline exceeded Duration: 3.492263915s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X...XXX...X.XX.X..XX..X.......X.......................XX....] duration: 5.892720028s trace: https://jaeger.dev.storj.io/trace/6f703d28b1929c52 FAIL (2025-12-28 11:37:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 11:34:32.717979221 (548h5m32.589062256s) [upload=480597]: Starting upload Dec 28 11:34:32.718034726 (548h5m32.589117761s) [upload=480597]: Done starting upload Dec 28 11:34:32.718137753 (548h5m32.589220788s) [upload=480597]: Uploading segments... Dec 28 11:34:32.718242222 (548h5m32.589325267s) [upload=480597]: Got next segment. Inline: false Dec 28 11:34:32.718352251 (548h5m32.589435286s) [upload=480597]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:34:32.718359746 (548h5m32.589442791s) [upload=480597]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:34:32.971048456 (548h5m32.842131501s) [seg_pos={0 0}, upload=480597]: Begin upload segment... Dec 28 11:34:32.971094775 (548h5m32.84217781s) [seg_pos={0 0}, upload=480597]: Done begin upload segment. Dec 28 11:34:32.971122387 (548h5m32.842205432s) [noise=true , address=45.91.116.10:4081 , piece_id=D4WL3ZCAQ5D65HDHOZ37T7RUZKRIK4PLHCQKNLIETHW3XJI2QT7A, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.971315174 (548h5m32.842398219s) [noise=true , address=145.79.227.5:4221 , piece_id=6WFBTJHDE5TSPVHAV2ZTWTE2IBO3JOJDUM64SKO7C37R5IREW4EQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.971972636 (548h5m32.843055681s) [noise=true , address=45.91.116.6:4973 , piece_id=2J2YIKX5ONI3MV4R5POJXU4UA3U73PYWIEFXF2ZO5DKAABEXD6QA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.972383228 (548h5m32.843466263s) [noise=true , address=145.79.227.17:4970 , piece_id=ZYRS4LNRDBFV67SXPMOWHITAIY26IWAJYO4V3PUM7R6P4KWFRKPQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.972768462 (548h5m32.843851497s) [noise=true , address=45.91.116.12:4228 , piece_id=MAKKLTLELGFI2ILAQ2HSX7DHY5VD24CU75BLX2Q627RGL3UQMBQA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.973144468 (548h5m32.844227503s) [noise=true , address=145.79.227.3:4150 , piece_id=OMVP5HXU7JNGWQN5FDDWDXY25OMV36WJL6AOAPK4LKUEWE3I5WNA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.973501487 (548h5m32.844584522s) [noise=true , address=145.79.227.50:4375 , piece_id=LGUOKYMYOIT23TEJQHWGOE6ANOE4XWV3GUGS7UNRXFDIAKZ2SNMA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.973881802 (548h5m32.844964847s) [noise=true , address=45.91.116.11:4120 , piece_id=Y2GZFQPBNRSNNV32QPSXQKNXA6AF4SJLITJLHVBQN5ZFKAI2ANVQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.973921016 (548h5m32.845004061s) [noise=true , address=145.79.227.13:4802 , piece_id=JVWJXW2SOCW3ITHSYVYX7MVLSTKNJBPH3PGAXJ7TYUTFL2OMZQVQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.974492073 (548h5m32.845575118s) [noise=true , address=145.79.227.15:4874 , piece_id=TP6NCLTJIHHLYIRH4XCTRLCH3R6E3V5GCAFARULXKZWRA3O75HEA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.974890192 (548h5m32.845973237s) [noise=true , address=45.91.116.13:4153 , piece_id=TNCDU6FYKLKFMRE3XMQCN7D2VG5VQXDZK2YEGZSGBR7IBES2BOSQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.975273341 (548h5m32.846356377s) [noise=true , address=145.79.227.18:4007 , piece_id=6USMJDFJGIYXRWU5CV4ANCWWPIAISNAUGMJMN7ILF6ZQVFWTHFCA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.975661581 (548h5m32.846744616s) [noise=true , address=145.79.227.16:4909 , piece_id=UMUYPAVHMN7CSTNDVVC7OMKWO4QMBAMEEGQKQU7WC2XF7DSATKMQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.975974486 (548h5m32.847057521s) [noise=true , address=145.79.227.51:4411 , piece_id=LPGALNKEINII3D3OWG3T4BA7DWYVK2KJ7W6SMVIUPYCWA5RMIYKA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.976262475 (548h5m32.84734551s) [noise=true , address=145.79.227.7:4295 , piece_id=YCIYYCA4YTT3ZBXV56GL7HRN7VRG7IK757WS6I46R2ZC33XVDXHQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.976484358 (548h5m32.847567403s) [noise=true , address=77.90.31.21:4878 , piece_id=SSCIB6BIOGQ6IKYYCEVSXMXLYOSERAXABWIGPWYGF6OC3LHZJJSQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:34.685547355 (548h5m34.5566304s) [upload=480597]: Next returned nil segment Dec 28 11:34:34.685565679 (548h5m34.556648724s) [upload=480597]: Waiting for error group managing segments... Dec 28 11:34:34.978426004 (548h5m34.849509039s) [noise=true , address=145.79.227.17:4970 , piece_id=ZYRS4LNRDBFV67SXPMOWHITAIY26IWAJYO4V3PUM7R6P4KWFRKPQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.978883014 (548h5m34.849966049s) [noise=true , address=145.79.227.15:4874 , piece_id=TP6NCLTJIHHLYIRH4XCTRLCH3R6E3V5GCAFARULXKZWRA3O75HEA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.978939592 (548h5m34.850022627s) [noise=true , address=145.79.227.5:4221 , piece_id=6WFBTJHDE5TSPVHAV2ZTWTE2IBO3JOJDUM64SKO7C37R5IREW4EQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.980593361 (548h5m34.851676406s) [noise=true , address=145.79.227.16:4909 , piece_id=UMUYPAVHMN7CSTNDVVC7OMKWO4QMBAMEEGQKQU7WC2XF7DSATKMQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.981122900 (548h5m34.852205935s) [noise=true , address=145.79.227.7:4295 , piece_id=YCIYYCA4YTT3ZBXV56GL7HRN7VRG7IK757WS6I46R2ZC33XVDXHQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.987647213 (548h5m34.858730248s) [noise=true , address=145.79.227.3:4150 , piece_id=OMVP5HXU7JNGWQN5FDDWDXY25OMV36WJL6AOAPK4LKUEWE3I5WNA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.987901146 (548h5m34.858984181s) [noise=true , address=145.79.227.18:4007 , piece_id=6USMJDFJGIYXRWU5CV4ANCWWPIAISNAUGMJMN7ILF6ZQVFWTHFCA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.990895628 (548h5m34.861978663s) [noise=true , address=145.79.227.50:4375 , piece_id=LGUOKYMYOIT23TEJQHWGOE6ANOE4XWV3GUGS7UNRXFDIAKZ2SNMA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.996206201 (548h5m34.867289246s) [noise=true , address=145.79.227.13:4802 , piece_id=JVWJXW2SOCW3ITHSYVYX7MVLSTKNJBPH3PGAXJ7TYUTFL2OMZQVQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.996219997 (548h5m34.867303042s) [noise=true , address=145.79.227.51:4411 , piece_id=LPGALNKEINII3D3OWG3T4BA7DWYVK2KJ7W6SMVIUPYCWA5RMIYKA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:35.185925637 (548h5m35.057008672s) [noise=true , address=77.90.31.21:4878 , piece_id=SSCIB6BIOGQ6IKYYCEVSXMXLYOSERAXABWIGPWYGF6OC3LHZJJSQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:35.204791852 (548h5m35.075874917s) [noise=true , address=45.91.116.6:4973 , piece_id=2J2YIKX5ONI3MV4R5POJXU4UA3U73PYWIEFXF2ZO5DKAABEXD6QA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:35.272032800 (548h5m35.143115835s) [noise=true , address=45.91.116.11:4120 , piece_id=Y2GZFQPBNRSNNV32QPSXQKNXA6AF4SJLITJLHVBQN5ZFKAI2ANVQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:35.272053630 (548h5m35.143136675s) [seg_pos={0 0}, upload=480597]: Segment reached optimal threshold of 13 pieces. Dec 28 11:34:35.272313985 (548h5m35.14339703s) [noise=true , address=45.91.116.13:4153 , piece_id=TNCDU6FYKLKFMRE3XMQCN7D2VG5VQXDZK2YEGZSGBR7IBES2BOSQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 28 11:34:35.272322852 (548h5m35.143405887s) [noise=true , address=45.91.116.13:4153 , piece_id=TNCDU6FYKLKFMRE3XMQCN7D2VG5VQXDZK2YEGZSGBR7IBES2BOSQ, seg_pos={0 0}, upload=480597]: Long tail cancellation due to optimal threshold Dec 28 11:34:35.272548011 (548h5m35.143631056s) [noise=true , address=45.91.116.10:4081 , piece_id=D4WL3ZCAQ5D65HDHOZ37T7RUZKRIK4PLHCQKNLIETHW3XJI2QT7A, seg_pos={0 0}, upload=480597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 11:34:35.272554263 (548h5m35.143637298s) [noise=true , address=45.91.116.10:4081 , piece_id=D4WL3ZCAQ5D65HDHOZ37T7RUZKRIK4PLHCQKNLIETHW3XJI2QT7A, seg_pos={0 0}, upload=480597]: Long tail cancellation due to optimal threshold Dec 28 11:34:35.272612814 (548h5m35.143695869s) [noise=true , address=45.91.116.12:4228 , piece_id=MAKKLTLELGFI2ILAQ2HSX7DHY5VD24CU75BLX2Q627RGL3UQMBQA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 11:34:35.272655736 (548h5m35.143738781s) [noise=true , address=45.91.116.12:4228 , piece_id=MAKKLTLELGFI2ILAQ2HSX7DHY5VD24CU75BLX2Q627RGL3UQMBQA, seg_pos={0 0}, upload=480597]: Long tail cancellation due to optimal threshold Dec 28 11:34:35.272690582 (548h5m35.143773627s) [seg_pos={0 0}, upload=480597]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:34:35.272858011 (548h5m35.143941046s) [upload=480597]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:34:35.272866517 (548h5m35.143949562s) [upload=480597]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:34:35.564025180 (548h5m35.435108215s) [upload=480597]: Done uploading segments... Dec 28 11:34:35.564047874 (548h5m35.435130909s) [upload=480597]: Upload finished. err: <nil> Duration: 5.892763811s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [X.X...X.....XX.XX..XXX...X.X....X..XXXX..X.X.XX.....X.X.X..X] duration: 5.001587435s trace: https://jaeger.dev.storj.io/trace/1de298e8089fbea9 FAIL (2025-12-28 11:37:51 UTC) orbiter warning: context deadline exceeded Duration: 5.001610578s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................X........X...........X..X.........] duration: 42.631809081s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:37:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 42.631870897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X...XXX...X.XX.X..XX..X.......X.......................XX...] duration: 6.052937168s trace: https://jaeger.dev.storj.io/trace/6f703d28b1929c52 FAIL (2025-12-28 11:37:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 11:34:32.717979221 (548h5m32.589062256s) [upload=480597]: Starting upload Dec 28 11:34:32.718034726 (548h5m32.589117761s) [upload=480597]: Done starting upload Dec 28 11:34:32.718137753 (548h5m32.589220788s) [upload=480597]: Uploading segments... Dec 28 11:34:32.718242222 (548h5m32.589325267s) [upload=480597]: Got next segment. Inline: false Dec 28 11:34:32.718352251 (548h5m32.589435286s) [upload=480597]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:34:32.718359746 (548h5m32.589442791s) [upload=480597]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:34:32.971048456 (548h5m32.842131501s) [seg_pos={0 0}, upload=480597]: Begin upload segment... Dec 28 11:34:32.971094775 (548h5m32.84217781s) [seg_pos={0 0}, upload=480597]: Done begin upload segment. Dec 28 11:34:32.971122387 (548h5m32.842205432s) [noise=true , address=45.91.116.10:4081 , piece_id=D4WL3ZCAQ5D65HDHOZ37T7RUZKRIK4PLHCQKNLIETHW3XJI2QT7A, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.971315174 (548h5m32.842398219s) [noise=true , address=145.79.227.5:4221 , piece_id=6WFBTJHDE5TSPVHAV2ZTWTE2IBO3JOJDUM64SKO7C37R5IREW4EQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.971972636 (548h5m32.843055681s) [noise=true , address=45.91.116.6:4973 , piece_id=2J2YIKX5ONI3MV4R5POJXU4UA3U73PYWIEFXF2ZO5DKAABEXD6QA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.972383228 (548h5m32.843466263s) [noise=true , address=145.79.227.17:4970 , piece_id=ZYRS4LNRDBFV67SXPMOWHITAIY26IWAJYO4V3PUM7R6P4KWFRKPQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.972768462 (548h5m32.843851497s) [noise=true , address=45.91.116.12:4228 , piece_id=MAKKLTLELGFI2ILAQ2HSX7DHY5VD24CU75BLX2Q627RGL3UQMBQA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.973144468 (548h5m32.844227503s) [noise=true , address=145.79.227.3:4150 , piece_id=OMVP5HXU7JNGWQN5FDDWDXY25OMV36WJL6AOAPK4LKUEWE3I5WNA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.973501487 (548h5m32.844584522s) [noise=true , address=145.79.227.50:4375 , piece_id=LGUOKYMYOIT23TEJQHWGOE6ANOE4XWV3GUGS7UNRXFDIAKZ2SNMA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.973881802 (548h5m32.844964847s) [noise=true , address=45.91.116.11:4120 , piece_id=Y2GZFQPBNRSNNV32QPSXQKNXA6AF4SJLITJLHVBQN5ZFKAI2ANVQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.973921016 (548h5m32.845004061s) [noise=true , address=145.79.227.13:4802 , piece_id=JVWJXW2SOCW3ITHSYVYX7MVLSTKNJBPH3PGAXJ7TYUTFL2OMZQVQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.974492073 (548h5m32.845575118s) [noise=true , address=145.79.227.15:4874 , piece_id=TP6NCLTJIHHLYIRH4XCTRLCH3R6E3V5GCAFARULXKZWRA3O75HEA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.974890192 (548h5m32.845973237s) [noise=true , address=45.91.116.13:4153 , piece_id=TNCDU6FYKLKFMRE3XMQCN7D2VG5VQXDZK2YEGZSGBR7IBES2BOSQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.975273341 (548h5m32.846356377s) [noise=true , address=145.79.227.18:4007 , piece_id=6USMJDFJGIYXRWU5CV4ANCWWPIAISNAUGMJMN7ILF6ZQVFWTHFCA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.975661581 (548h5m32.846744616s) [noise=true , address=145.79.227.16:4909 , piece_id=UMUYPAVHMN7CSTNDVVC7OMKWO4QMBAMEEGQKQU7WC2XF7DSATKMQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.975974486 (548h5m32.847057521s) [noise=true , address=145.79.227.51:4411 , piece_id=LPGALNKEINII3D3OWG3T4BA7DWYVK2KJ7W6SMVIUPYCWA5RMIYKA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.976262475 (548h5m32.84734551s) [noise=true , address=145.79.227.7:4295 , piece_id=YCIYYCA4YTT3ZBXV56GL7HRN7VRG7IK757WS6I46R2ZC33XVDXHQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.976484358 (548h5m32.847567403s) [noise=true , address=77.90.31.21:4878 , piece_id=SSCIB6BIOGQ6IKYYCEVSXMXLYOSERAXABWIGPWYGF6OC3LHZJJSQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:34.685547355 (548h5m34.5566304s) [upload=480597]: Next returned nil segment Dec 28 11:34:34.685565679 (548h5m34.556648724s) [upload=480597]: Waiting for error group managing segments... Dec 28 11:34:34.978426004 (548h5m34.849509039s) [noise=true , address=145.79.227.17:4970 , piece_id=ZYRS4LNRDBFV67SXPMOWHITAIY26IWAJYO4V3PUM7R6P4KWFRKPQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.978883014 (548h5m34.849966049s) [noise=true , address=145.79.227.15:4874 , piece_id=TP6NCLTJIHHLYIRH4XCTRLCH3R6E3V5GCAFARULXKZWRA3O75HEA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.978939592 (548h5m34.850022627s) [noise=true , address=145.79.227.5:4221 , piece_id=6WFBTJHDE5TSPVHAV2ZTWTE2IBO3JOJDUM64SKO7C37R5IREW4EQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.980593361 (548h5m34.851676406s) [noise=true , address=145.79.227.16:4909 , piece_id=UMUYPAVHMN7CSTNDVVC7OMKWO4QMBAMEEGQKQU7WC2XF7DSATKMQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.981122900 (548h5m34.852205935s) [noise=true , address=145.79.227.7:4295 , piece_id=YCIYYCA4YTT3ZBXV56GL7HRN7VRG7IK757WS6I46R2ZC33XVDXHQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.987647213 (548h5m34.858730248s) [noise=true , address=145.79.227.3:4150 , piece_id=OMVP5HXU7JNGWQN5FDDWDXY25OMV36WJL6AOAPK4LKUEWE3I5WNA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.987901146 (548h5m34.858984181s) [noise=true , address=145.79.227.18:4007 , piece_id=6USMJDFJGIYXRWU5CV4ANCWWPIAISNAUGMJMN7ILF6ZQVFWTHFCA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.990895628 (548h5m34.861978663s) [noise=true , address=145.79.227.50:4375 , piece_id=LGUOKYMYOIT23TEJQHWGOE6ANOE4XWV3GUGS7UNRXFDIAKZ2SNMA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.996206201 (548h5m34.867289246s) [noise=true , address=145.79.227.13:4802 , piece_id=JVWJXW2SOCW3ITHSYVYX7MVLSTKNJBPH3PGAXJ7TYUTFL2OMZQVQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.996219997 (548h5m34.867303042s) [noise=true , address=145.79.227.51:4411 , piece_id=LPGALNKEINII3D3OWG3T4BA7DWYVK2KJ7W6SMVIUPYCWA5RMIYKA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:35.185925637 (548h5m35.057008672s) [noise=true , address=77.90.31.21:4878 , piece_id=SSCIB6BIOGQ6IKYYCEVSXMXLYOSERAXABWIGPWYGF6OC3LHZJJSQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:35.204791852 (548h5m35.075874917s) [noise=true , address=45.91.116.6:4973 , piece_id=2J2YIKX5ONI3MV4R5POJXU4UA3U73PYWIEFXF2ZO5DKAABEXD6QA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:35.272032800 (548h5m35.143115835s) [noise=true , address=45.91.116.11:4120 , piece_id=Y2GZFQPBNRSNNV32QPSXQKNXA6AF4SJLITJLHVBQN5ZFKAI2ANVQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:35.272053630 (548h5m35.143136675s) [seg_pos={0 0}, upload=480597]: Segment reached optimal threshold of 13 pieces. Dec 28 11:34:35.272313985 (548h5m35.14339703s) [noise=true , address=45.91.116.13:4153 , piece_id=TNCDU6FYKLKFMRE3XMQCN7D2VG5VQXDZK2YEGZSGBR7IBES2BOSQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 28 11:34:35.272322852 (548h5m35.143405887s) [noise=true , address=45.91.116.13:4153 , piece_id=TNCDU6FYKLKFMRE3XMQCN7D2VG5VQXDZK2YEGZSGBR7IBES2BOSQ, seg_pos={0 0}, upload=480597]: Long tail cancellation due to optimal threshold Dec 28 11:34:35.272548011 (548h5m35.143631056s) [noise=true , address=45.91.116.10:4081 , piece_id=D4WL3ZCAQ5D65HDHOZ37T7RUZKRIK4PLHCQKNLIETHW3XJI2QT7A, seg_pos={0 0}, upload=480597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 11:34:35.272554263 (548h5m35.143637298s) [noise=true , address=45.91.116.10:4081 , piece_id=D4WL3ZCAQ5D65HDHOZ37T7RUZKRIK4PLHCQKNLIETHW3XJI2QT7A, seg_pos={0 0}, upload=480597]: Long tail cancellation due to optimal threshold Dec 28 11:34:35.272612814 (548h5m35.143695869s) [noise=true , address=45.91.116.12:4228 , piece_id=MAKKLTLELGFI2ILAQ2HSX7DHY5VD24CU75BLX2Q627RGL3UQMBQA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 11:34:35.272655736 (548h5m35.143738781s) [noise=true , address=45.91.116.12:4228 , piece_id=MAKKLTLELGFI2ILAQ2HSX7DHY5VD24CU75BLX2Q627RGL3UQMBQA, seg_pos={0 0}, upload=480597]: Long tail cancellation due to optimal threshold Dec 28 11:34:35.272690582 (548h5m35.143773627s) [seg_pos={0 0}, upload=480597]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:34:35.272858011 (548h5m35.143941046s) [upload=480597]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:34:35.272866517 (548h5m35.143949562s) [upload=480597]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:34:35.564025180 (548h5m35.435108215s) [upload=480597]: Done uploading segments... Dec 28 11:34:35.564047874 (548h5m35.435130909s) [upload=480597]: Upload finished. err: <nil> Duration: 6.052962787s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.X.X...X.....XX.XX..XXX...X.X....X..XXXX..X.X.XX.....X.X.X..] duration: 3.426413876s trace: https://jaeger.dev.storj.io/trace/1da8830a29eebaa FAIL (2025-12-28 11:37:08 UTC) orbiter warning: context deadline exceeded Duration: 3.426446179s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [..X.X...X.....XX.XX..XXX...X.X....X..XXXX..X.X.XX.....X.X.X.] duration: 5.000977355s trace: https://jaeger.dev.storj.io/trace/1da8830a29eebaa FAIL (2025-12-28 11:36:30 UTC) orbiter warning: context deadline exceeded Duration: 5.001002152s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X...XXX...X.XX.X..XX..X.......X.......................XX..] duration: 5.673231355s trace: https://jaeger.dev.storj.io/trace/6f703d28b1929c52 FAIL (2025-12-28 11:36:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 11:34:32.717979221 (548h5m32.589062256s) [upload=480597]: Starting upload Dec 28 11:34:32.718034726 (548h5m32.589117761s) [upload=480597]: Done starting upload Dec 28 11:34:32.718137753 (548h5m32.589220788s) [upload=480597]: Uploading segments... Dec 28 11:34:32.718242222 (548h5m32.589325267s) [upload=480597]: Got next segment. Inline: false Dec 28 11:34:32.718352251 (548h5m32.589435286s) [upload=480597]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:34:32.718359746 (548h5m32.589442791s) [upload=480597]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:34:32.971048456 (548h5m32.842131501s) [seg_pos={0 0}, upload=480597]: Begin upload segment... Dec 28 11:34:32.971094775 (548h5m32.84217781s) [seg_pos={0 0}, upload=480597]: Done begin upload segment. Dec 28 11:34:32.971122387 (548h5m32.842205432s) [noise=true , address=45.91.116.10:4081 , piece_id=D4WL3ZCAQ5D65HDHOZ37T7RUZKRIK4PLHCQKNLIETHW3XJI2QT7A, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.971315174 (548h5m32.842398219s) [noise=true , address=145.79.227.5:4221 , piece_id=6WFBTJHDE5TSPVHAV2ZTWTE2IBO3JOJDUM64SKO7C37R5IREW4EQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.971972636 (548h5m32.843055681s) [noise=true , address=45.91.116.6:4973 , piece_id=2J2YIKX5ONI3MV4R5POJXU4UA3U73PYWIEFXF2ZO5DKAABEXD6QA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.972383228 (548h5m32.843466263s) [noise=true , address=145.79.227.17:4970 , piece_id=ZYRS4LNRDBFV67SXPMOWHITAIY26IWAJYO4V3PUM7R6P4KWFRKPQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.972768462 (548h5m32.843851497s) [noise=true , address=45.91.116.12:4228 , piece_id=MAKKLTLELGFI2ILAQ2HSX7DHY5VD24CU75BLX2Q627RGL3UQMBQA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.973144468 (548h5m32.844227503s) [noise=true , address=145.79.227.3:4150 , piece_id=OMVP5HXU7JNGWQN5FDDWDXY25OMV36WJL6AOAPK4LKUEWE3I5WNA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.973501487 (548h5m32.844584522s) [noise=true , address=145.79.227.50:4375 , piece_id=LGUOKYMYOIT23TEJQHWGOE6ANOE4XWV3GUGS7UNRXFDIAKZ2SNMA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.973881802 (548h5m32.844964847s) [noise=true , address=45.91.116.11:4120 , piece_id=Y2GZFQPBNRSNNV32QPSXQKNXA6AF4SJLITJLHVBQN5ZFKAI2ANVQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.973921016 (548h5m32.845004061s) [noise=true , address=145.79.227.13:4802 , piece_id=JVWJXW2SOCW3ITHSYVYX7MVLSTKNJBPH3PGAXJ7TYUTFL2OMZQVQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.974492073 (548h5m32.845575118s) [noise=true , address=145.79.227.15:4874 , piece_id=TP6NCLTJIHHLYIRH4XCTRLCH3R6E3V5GCAFARULXKZWRA3O75HEA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.974890192 (548h5m32.845973237s) [noise=true , address=45.91.116.13:4153 , piece_id=TNCDU6FYKLKFMRE3XMQCN7D2VG5VQXDZK2YEGZSGBR7IBES2BOSQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.975273341 (548h5m32.846356377s) [noise=true , address=145.79.227.18:4007 , piece_id=6USMJDFJGIYXRWU5CV4ANCWWPIAISNAUGMJMN7ILF6ZQVFWTHFCA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.975661581 (548h5m32.846744616s) [noise=true , address=145.79.227.16:4909 , piece_id=UMUYPAVHMN7CSTNDVVC7OMKWO4QMBAMEEGQKQU7WC2XF7DSATKMQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.975974486 (548h5m32.847057521s) [noise=true , address=145.79.227.51:4411 , piece_id=LPGALNKEINII3D3OWG3T4BA7DWYVK2KJ7W6SMVIUPYCWA5RMIYKA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.976262475 (548h5m32.84734551s) [noise=true , address=145.79.227.7:4295 , piece_id=YCIYYCA4YTT3ZBXV56GL7HRN7VRG7IK757WS6I46R2ZC33XVDXHQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.976484358 (548h5m32.847567403s) [noise=true , address=77.90.31.21:4878 , piece_id=SSCIB6BIOGQ6IKYYCEVSXMXLYOSERAXABWIGPWYGF6OC3LHZJJSQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:34.685547355 (548h5m34.5566304s) [upload=480597]: Next returned nil segment Dec 28 11:34:34.685565679 (548h5m34.556648724s) [upload=480597]: Waiting for error group managing segments... Dec 28 11:34:34.978426004 (548h5m34.849509039s) [noise=true , address=145.79.227.17:4970 , piece_id=ZYRS4LNRDBFV67SXPMOWHITAIY26IWAJYO4V3PUM7R6P4KWFRKPQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.978883014 (548h5m34.849966049s) [noise=true , address=145.79.227.15:4874 , piece_id=TP6NCLTJIHHLYIRH4XCTRLCH3R6E3V5GCAFARULXKZWRA3O75HEA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.978939592 (548h5m34.850022627s) [noise=true , address=145.79.227.5:4221 , piece_id=6WFBTJHDE5TSPVHAV2ZTWTE2IBO3JOJDUM64SKO7C37R5IREW4EQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.980593361 (548h5m34.851676406s) [noise=true , address=145.79.227.16:4909 , piece_id=UMUYPAVHMN7CSTNDVVC7OMKWO4QMBAMEEGQKQU7WC2XF7DSATKMQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.981122900 (548h5m34.852205935s) [noise=true , address=145.79.227.7:4295 , piece_id=YCIYYCA4YTT3ZBXV56GL7HRN7VRG7IK757WS6I46R2ZC33XVDXHQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.987647213 (548h5m34.858730248s) [noise=true , address=145.79.227.3:4150 , piece_id=OMVP5HXU7JNGWQN5FDDWDXY25OMV36WJL6AOAPK4LKUEWE3I5WNA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.987901146 (548h5m34.858984181s) [noise=true , address=145.79.227.18:4007 , piece_id=6USMJDFJGIYXRWU5CV4ANCWWPIAISNAUGMJMN7ILF6ZQVFWTHFCA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.990895628 (548h5m34.861978663s) [noise=true , address=145.79.227.50:4375 , piece_id=LGUOKYMYOIT23TEJQHWGOE6ANOE4XWV3GUGS7UNRXFDIAKZ2SNMA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.996206201 (548h5m34.867289246s) [noise=true , address=145.79.227.13:4802 , piece_id=JVWJXW2SOCW3ITHSYVYX7MVLSTKNJBPH3PGAXJ7TYUTFL2OMZQVQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.996219997 (548h5m34.867303042s) [noise=true , address=145.79.227.51:4411 , piece_id=LPGALNKEINII3D3OWG3T4BA7DWYVK2KJ7W6SMVIUPYCWA5RMIYKA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:35.185925637 (548h5m35.057008672s) [noise=true , address=77.90.31.21:4878 , piece_id=SSCIB6BIOGQ6IKYYCEVSXMXLYOSERAXABWIGPWYGF6OC3LHZJJSQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:35.204791852 (548h5m35.075874917s) [noise=true , address=45.91.116.6:4973 , piece_id=2J2YIKX5ONI3MV4R5POJXU4UA3U73PYWIEFXF2ZO5DKAABEXD6QA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:35.272032800 (548h5m35.143115835s) [noise=true , address=45.91.116.11:4120 , piece_id=Y2GZFQPBNRSNNV32QPSXQKNXA6AF4SJLITJLHVBQN5ZFKAI2ANVQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:35.272053630 (548h5m35.143136675s) [seg_pos={0 0}, upload=480597]: Segment reached optimal threshold of 13 pieces. Dec 28 11:34:35.272313985 (548h5m35.14339703s) [noise=true , address=45.91.116.13:4153 , piece_id=TNCDU6FYKLKFMRE3XMQCN7D2VG5VQXDZK2YEGZSGBR7IBES2BOSQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 28 11:34:35.272322852 (548h5m35.143405887s) [noise=true , address=45.91.116.13:4153 , piece_id=TNCDU6FYKLKFMRE3XMQCN7D2VG5VQXDZK2YEGZSGBR7IBES2BOSQ, seg_pos={0 0}, upload=480597]: Long tail cancellation due to optimal threshold Dec 28 11:34:35.272548011 (548h5m35.143631056s) [noise=true , address=45.91.116.10:4081 , piece_id=D4WL3ZCAQ5D65HDHOZ37T7RUZKRIK4PLHCQKNLIETHW3XJI2QT7A, seg_pos={0 0}, upload=480597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 11:34:35.272554263 (548h5m35.143637298s) [noise=true , address=45.91.116.10:4081 , piece_id=D4WL3ZCAQ5D65HDHOZ37T7RUZKRIK4PLHCQKNLIETHW3XJI2QT7A, seg_pos={0 0}, upload=480597]: Long tail cancellation due to optimal threshold Dec 28 11:34:35.272612814 (548h5m35.143695869s) [noise=true , address=45.91.116.12:4228 , piece_id=MAKKLTLELGFI2ILAQ2HSX7DHY5VD24CU75BLX2Q627RGL3UQMBQA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 11:34:35.272655736 (548h5m35.143738781s) [noise=true , address=45.91.116.12:4228 , piece_id=MAKKLTLELGFI2ILAQ2HSX7DHY5VD24CU75BLX2Q627RGL3UQMBQA, seg_pos={0 0}, upload=480597]: Long tail cancellation due to optimal threshold Dec 28 11:34:35.272690582 (548h5m35.143773627s) [seg_pos={0 0}, upload=480597]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:34:35.272858011 (548h5m35.143941046s) [upload=480597]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:34:35.272866517 (548h5m35.143949562s) [upload=480597]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:34:35.564025180 (548h5m35.435108215s) [upload=480597]: Done uploading segments... Dec 28 11:34:35.564047874 (548h5m35.435130909s) [upload=480597]: Upload finished. err: <nil> Duration: 5.673260792s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................X........X...........X..X........] duration: 20.687593742s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:36:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 20.687609432s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................X........X...........X..X.......] duration: 42.364412876s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:36:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 42.364429629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...X...XXX...X.XX.X..XX..X.......X.......................XX.] duration: 5.677089702s trace: https://jaeger.dev.storj.io/trace/6f703d28b1929c52 FAIL (2025-12-28 11:35:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 11:34:32.717979221 (548h5m32.589062256s) [upload=480597]: Starting upload Dec 28 11:34:32.718034726 (548h5m32.589117761s) [upload=480597]: Done starting upload Dec 28 11:34:32.718137753 (548h5m32.589220788s) [upload=480597]: Uploading segments... Dec 28 11:34:32.718242222 (548h5m32.589325267s) [upload=480597]: Got next segment. Inline: false Dec 28 11:34:32.718352251 (548h5m32.589435286s) [upload=480597]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:34:32.718359746 (548h5m32.589442791s) [upload=480597]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:34:32.971048456 (548h5m32.842131501s) [seg_pos={0 0}, upload=480597]: Begin upload segment... Dec 28 11:34:32.971094775 (548h5m32.84217781s) [seg_pos={0 0}, upload=480597]: Done begin upload segment. Dec 28 11:34:32.971122387 (548h5m32.842205432s) [noise=true , address=45.91.116.10:4081 , piece_id=D4WL3ZCAQ5D65HDHOZ37T7RUZKRIK4PLHCQKNLIETHW3XJI2QT7A, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.971315174 (548h5m32.842398219s) [noise=true , address=145.79.227.5:4221 , piece_id=6WFBTJHDE5TSPVHAV2ZTWTE2IBO3JOJDUM64SKO7C37R5IREW4EQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.971972636 (548h5m32.843055681s) [noise=true , address=45.91.116.6:4973 , piece_id=2J2YIKX5ONI3MV4R5POJXU4UA3U73PYWIEFXF2ZO5DKAABEXD6QA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.972383228 (548h5m32.843466263s) [noise=true , address=145.79.227.17:4970 , piece_id=ZYRS4LNRDBFV67SXPMOWHITAIY26IWAJYO4V3PUM7R6P4KWFRKPQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.972768462 (548h5m32.843851497s) [noise=true , address=45.91.116.12:4228 , piece_id=MAKKLTLELGFI2ILAQ2HSX7DHY5VD24CU75BLX2Q627RGL3UQMBQA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.973144468 (548h5m32.844227503s) [noise=true , address=145.79.227.3:4150 , piece_id=OMVP5HXU7JNGWQN5FDDWDXY25OMV36WJL6AOAPK4LKUEWE3I5WNA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.973501487 (548h5m32.844584522s) [noise=true , address=145.79.227.50:4375 , piece_id=LGUOKYMYOIT23TEJQHWGOE6ANOE4XWV3GUGS7UNRXFDIAKZ2SNMA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.973881802 (548h5m32.844964847s) [noise=true , address=45.91.116.11:4120 , piece_id=Y2GZFQPBNRSNNV32QPSXQKNXA6AF4SJLITJLHVBQN5ZFKAI2ANVQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.973921016 (548h5m32.845004061s) [noise=true , address=145.79.227.13:4802 , piece_id=JVWJXW2SOCW3ITHSYVYX7MVLSTKNJBPH3PGAXJ7TYUTFL2OMZQVQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.974492073 (548h5m32.845575118s) [noise=true , address=145.79.227.15:4874 , piece_id=TP6NCLTJIHHLYIRH4XCTRLCH3R6E3V5GCAFARULXKZWRA3O75HEA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.974890192 (548h5m32.845973237s) [noise=true , address=45.91.116.13:4153 , piece_id=TNCDU6FYKLKFMRE3XMQCN7D2VG5VQXDZK2YEGZSGBR7IBES2BOSQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.975273341 (548h5m32.846356377s) [noise=true , address=145.79.227.18:4007 , piece_id=6USMJDFJGIYXRWU5CV4ANCWWPIAISNAUGMJMN7ILF6ZQVFWTHFCA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.975661581 (548h5m32.846744616s) [noise=true , address=145.79.227.16:4909 , piece_id=UMUYPAVHMN7CSTNDVVC7OMKWO4QMBAMEEGQKQU7WC2XF7DSATKMQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.975974486 (548h5m32.847057521s) [noise=true , address=145.79.227.51:4411 , piece_id=LPGALNKEINII3D3OWG3T4BA7DWYVK2KJ7W6SMVIUPYCWA5RMIYKA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.976262475 (548h5m32.84734551s) [noise=true , address=145.79.227.7:4295 , piece_id=YCIYYCA4YTT3ZBXV56GL7HRN7VRG7IK757WS6I46R2ZC33XVDXHQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.976484358 (548h5m32.847567403s) [noise=true , address=77.90.31.21:4878 , piece_id=SSCIB6BIOGQ6IKYYCEVSXMXLYOSERAXABWIGPWYGF6OC3LHZJJSQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:34.685547355 (548h5m34.5566304s) [upload=480597]: Next returned nil segment Dec 28 11:34:34.685565679 (548h5m34.556648724s) [upload=480597]: Waiting for error group managing segments... Dec 28 11:34:34.978426004 (548h5m34.849509039s) [noise=true , address=145.79.227.17:4970 , piece_id=ZYRS4LNRDBFV67SXPMOWHITAIY26IWAJYO4V3PUM7R6P4KWFRKPQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.978883014 (548h5m34.849966049s) [noise=true , address=145.79.227.15:4874 , piece_id=TP6NCLTJIHHLYIRH4XCTRLCH3R6E3V5GCAFARULXKZWRA3O75HEA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.978939592 (548h5m34.850022627s) [noise=true , address=145.79.227.5:4221 , piece_id=6WFBTJHDE5TSPVHAV2ZTWTE2IBO3JOJDUM64SKO7C37R5IREW4EQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.980593361 (548h5m34.851676406s) [noise=true , address=145.79.227.16:4909 , piece_id=UMUYPAVHMN7CSTNDVVC7OMKWO4QMBAMEEGQKQU7WC2XF7DSATKMQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.981122900 (548h5m34.852205935s) [noise=true , address=145.79.227.7:4295 , piece_id=YCIYYCA4YTT3ZBXV56GL7HRN7VRG7IK757WS6I46R2ZC33XVDXHQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.987647213 (548h5m34.858730248s) [noise=true , address=145.79.227.3:4150 , piece_id=OMVP5HXU7JNGWQN5FDDWDXY25OMV36WJL6AOAPK4LKUEWE3I5WNA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.987901146 (548h5m34.858984181s) [noise=true , address=145.79.227.18:4007 , piece_id=6USMJDFJGIYXRWU5CV4ANCWWPIAISNAUGMJMN7ILF6ZQVFWTHFCA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.990895628 (548h5m34.861978663s) [noise=true , address=145.79.227.50:4375 , piece_id=LGUOKYMYOIT23TEJQHWGOE6ANOE4XWV3GUGS7UNRXFDIAKZ2SNMA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.996206201 (548h5m34.867289246s) [noise=true , address=145.79.227.13:4802 , piece_id=JVWJXW2SOCW3ITHSYVYX7MVLSTKNJBPH3PGAXJ7TYUTFL2OMZQVQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.996219997 (548h5m34.867303042s) [noise=true , address=145.79.227.51:4411 , piece_id=LPGALNKEINII3D3OWG3T4BA7DWYVK2KJ7W6SMVIUPYCWA5RMIYKA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:35.185925637 (548h5m35.057008672s) [noise=true , address=77.90.31.21:4878 , piece_id=SSCIB6BIOGQ6IKYYCEVSXMXLYOSERAXABWIGPWYGF6OC3LHZJJSQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:35.204791852 (548h5m35.075874917s) [noise=true , address=45.91.116.6:4973 , piece_id=2J2YIKX5ONI3MV4R5POJXU4UA3U73PYWIEFXF2ZO5DKAABEXD6QA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:35.272032800 (548h5m35.143115835s) [noise=true , address=45.91.116.11:4120 , piece_id=Y2GZFQPBNRSNNV32QPSXQKNXA6AF4SJLITJLHVBQN5ZFKAI2ANVQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:35.272053630 (548h5m35.143136675s) [seg_pos={0 0}, upload=480597]: Segment reached optimal threshold of 13 pieces. Dec 28 11:34:35.272313985 (548h5m35.14339703s) [noise=true , address=45.91.116.13:4153 , piece_id=TNCDU6FYKLKFMRE3XMQCN7D2VG5VQXDZK2YEGZSGBR7IBES2BOSQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 28 11:34:35.272322852 (548h5m35.143405887s) [noise=true , address=45.91.116.13:4153 , piece_id=TNCDU6FYKLKFMRE3XMQCN7D2VG5VQXDZK2YEGZSGBR7IBES2BOSQ, seg_pos={0 0}, upload=480597]: Long tail cancellation due to optimal threshold Dec 28 11:34:35.272548011 (548h5m35.143631056s) [noise=true , address=45.91.116.10:4081 , piece_id=D4WL3ZCAQ5D65HDHOZ37T7RUZKRIK4PLHCQKNLIETHW3XJI2QT7A, seg_pos={0 0}, upload=480597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 11:34:35.272554263 (548h5m35.143637298s) [noise=true , address=45.91.116.10:4081 , piece_id=D4WL3ZCAQ5D65HDHOZ37T7RUZKRIK4PLHCQKNLIETHW3XJI2QT7A, seg_pos={0 0}, upload=480597]: Long tail cancellation due to optimal threshold Dec 28 11:34:35.272612814 (548h5m35.143695869s) [noise=true , address=45.91.116.12:4228 , piece_id=MAKKLTLELGFI2ILAQ2HSX7DHY5VD24CU75BLX2Q627RGL3UQMBQA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 11:34:35.272655736 (548h5m35.143738781s) [noise=true , address=45.91.116.12:4228 , piece_id=MAKKLTLELGFI2ILAQ2HSX7DHY5VD24CU75BLX2Q627RGL3UQMBQA, seg_pos={0 0}, upload=480597]: Long tail cancellation due to optimal threshold Dec 28 11:34:35.272690582 (548h5m35.143773627s) [seg_pos={0 0}, upload=480597]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:34:35.272858011 (548h5m35.143941046s) [upload=480597]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:34:35.272866517 (548h5m35.143949562s) [upload=480597]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:34:35.564025180 (548h5m35.435108215s) [upload=480597]: Done uploading segments... Dec 28 11:34:35.564047874 (548h5m35.435130909s) [upload=480597]: Upload finished. err: <nil> Duration: 5.677113357s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [...X.X...X.....XX.XX..XXX...X.X....X..XXXX..X.X.XX.....X.X.X] duration: 5.001036908s trace: https://jaeger.dev.storj.io/trace/1da8830a29eebaa FAIL (2025-12-28 11:35:23 UTC) orbiter warning: context deadline exceeded Duration: 5.001060934s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................X........X...........X..X......] duration: 55.956436827s trace: https://jaeger.dev.storj.io/trace/307d4901d0d5949 FAIL (2025-12-28 11:35:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece... Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments... Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil> Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces. Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments... Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil> Duration: 55.956577685s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X...X...XXX...X.XX.X..XX..X.......X.......................XX] duration: 30.012846536s trace: https://jaeger.dev.storj.io/trace/6f703d28b1929c52 FAIL (2025-12-28 11:35:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 11:34:32.717979221 (548h5m32.589062256s) [upload=480597]: Starting upload Dec 28 11:34:32.718034726 (548h5m32.589117761s) [upload=480597]: Done starting upload Dec 28 11:34:32.718137753 (548h5m32.589220788s) [upload=480597]: Uploading segments... Dec 28 11:34:32.718242222 (548h5m32.589325267s) [upload=480597]: Got next segment. Inline: false Dec 28 11:34:32.718352251 (548h5m32.589435286s) [upload=480597]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 11:34:32.718359746 (548h5m32.589442791s) [upload=480597]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 11:34:32.971048456 (548h5m32.842131501s) [seg_pos={0 0}, upload=480597]: Begin upload segment... Dec 28 11:34:32.971094775 (548h5m32.84217781s) [seg_pos={0 0}, upload=480597]: Done begin upload segment. Dec 28 11:34:32.971122387 (548h5m32.842205432s) [noise=true , address=45.91.116.10:4081 , piece_id=D4WL3ZCAQ5D65HDHOZ37T7RUZKRIK4PLHCQKNLIETHW3XJI2QT7A, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.971315174 (548h5m32.842398219s) [noise=true , address=145.79.227.5:4221 , piece_id=6WFBTJHDE5TSPVHAV2ZTWTE2IBO3JOJDUM64SKO7C37R5IREW4EQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.971972636 (548h5m32.843055681s) [noise=true , address=45.91.116.6:4973 , piece_id=2J2YIKX5ONI3MV4R5POJXU4UA3U73PYWIEFXF2ZO5DKAABEXD6QA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.972383228 (548h5m32.843466263s) [noise=true , address=145.79.227.17:4970 , piece_id=ZYRS4LNRDBFV67SXPMOWHITAIY26IWAJYO4V3PUM7R6P4KWFRKPQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.972768462 (548h5m32.843851497s) [noise=true , address=45.91.116.12:4228 , piece_id=MAKKLTLELGFI2ILAQ2HSX7DHY5VD24CU75BLX2Q627RGL3UQMBQA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.973144468 (548h5m32.844227503s) [noise=true , address=145.79.227.3:4150 , piece_id=OMVP5HXU7JNGWQN5FDDWDXY25OMV36WJL6AOAPK4LKUEWE3I5WNA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.973501487 (548h5m32.844584522s) [noise=true , address=145.79.227.50:4375 , piece_id=LGUOKYMYOIT23TEJQHWGOE6ANOE4XWV3GUGS7UNRXFDIAKZ2SNMA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.973881802 (548h5m32.844964847s) [noise=true , address=45.91.116.11:4120 , piece_id=Y2GZFQPBNRSNNV32QPSXQKNXA6AF4SJLITJLHVBQN5ZFKAI2ANVQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.973921016 (548h5m32.845004061s) [noise=true , address=145.79.227.13:4802 , piece_id=JVWJXW2SOCW3ITHSYVYX7MVLSTKNJBPH3PGAXJ7TYUTFL2OMZQVQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.974492073 (548h5m32.845575118s) [noise=true , address=145.79.227.15:4874 , piece_id=TP6NCLTJIHHLYIRH4XCTRLCH3R6E3V5GCAFARULXKZWRA3O75HEA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.974890192 (548h5m32.845973237s) [noise=true , address=45.91.116.13:4153 , piece_id=TNCDU6FYKLKFMRE3XMQCN7D2VG5VQXDZK2YEGZSGBR7IBES2BOSQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.975273341 (548h5m32.846356377s) [noise=true , address=145.79.227.18:4007 , piece_id=6USMJDFJGIYXRWU5CV4ANCWWPIAISNAUGMJMN7ILF6ZQVFWTHFCA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.975661581 (548h5m32.846744616s) [noise=true , address=145.79.227.16:4909 , piece_id=UMUYPAVHMN7CSTNDVVC7OMKWO4QMBAMEEGQKQU7WC2XF7DSATKMQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.975974486 (548h5m32.847057521s) [noise=true , address=145.79.227.51:4411 , piece_id=LPGALNKEINII3D3OWG3T4BA7DWYVK2KJ7W6SMVIUPYCWA5RMIYKA, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.976262475 (548h5m32.84734551s) [noise=true , address=145.79.227.7:4295 , piece_id=YCIYYCA4YTT3ZBXV56GL7HRN7VRG7IK757WS6I46R2ZC33XVDXHQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:32.976484358 (548h5m32.847567403s) [noise=true , address=77.90.31.21:4878 , piece_id=SSCIB6BIOGQ6IKYYCEVSXMXLYOSERAXABWIGPWYGF6OC3LHZJJSQ, seg_pos={0 0}, upload=480597]: Uploading piece... Dec 28 11:34:34.685547355 (548h5m34.5566304s) [upload=480597]: Next returned nil segment Dec 28 11:34:34.685565679 (548h5m34.556648724s) [upload=480597]: Waiting for error group managing segments... Dec 28 11:34:34.978426004 (548h5m34.849509039s) [noise=true , address=145.79.227.17:4970 , piece_id=ZYRS4LNRDBFV67SXPMOWHITAIY26IWAJYO4V3PUM7R6P4KWFRKPQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.978883014 (548h5m34.849966049s) [noise=true , address=145.79.227.15:4874 , piece_id=TP6NCLTJIHHLYIRH4XCTRLCH3R6E3V5GCAFARULXKZWRA3O75HEA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.978939592 (548h5m34.850022627s) [noise=true , address=145.79.227.5:4221 , piece_id=6WFBTJHDE5TSPVHAV2ZTWTE2IBO3JOJDUM64SKO7C37R5IREW4EQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.980593361 (548h5m34.851676406s) [noise=true , address=145.79.227.16:4909 , piece_id=UMUYPAVHMN7CSTNDVVC7OMKWO4QMBAMEEGQKQU7WC2XF7DSATKMQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.981122900 (548h5m34.852205935s) [noise=true , address=145.79.227.7:4295 , piece_id=YCIYYCA4YTT3ZBXV56GL7HRN7VRG7IK757WS6I46R2ZC33XVDXHQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.987647213 (548h5m34.858730248s) [noise=true , address=145.79.227.3:4150 , piece_id=OMVP5HXU7JNGWQN5FDDWDXY25OMV36WJL6AOAPK4LKUEWE3I5WNA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.987901146 (548h5m34.858984181s) [noise=true , address=145.79.227.18:4007 , piece_id=6USMJDFJGIYXRWU5CV4ANCWWPIAISNAUGMJMN7ILF6ZQVFWTHFCA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.990895628 (548h5m34.861978663s) [noise=true , address=145.79.227.50:4375 , piece_id=LGUOKYMYOIT23TEJQHWGOE6ANOE4XWV3GUGS7UNRXFDIAKZ2SNMA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.996206201 (548h5m34.867289246s) [noise=true , address=145.79.227.13:4802 , piece_id=JVWJXW2SOCW3ITHSYVYX7MVLSTKNJBPH3PGAXJ7TYUTFL2OMZQVQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:34.996219997 (548h5m34.867303042s) [noise=true , address=145.79.227.51:4411 , piece_id=LPGALNKEINII3D3OWG3T4BA7DWYVK2KJ7W6SMVIUPYCWA5RMIYKA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:35.185925637 (548h5m35.057008672s) [noise=true , address=77.90.31.21:4878 , piece_id=SSCIB6BIOGQ6IKYYCEVSXMXLYOSERAXABWIGPWYGF6OC3LHZJJSQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:35.204791852 (548h5m35.075874917s) [noise=true , address=45.91.116.6:4973 , piece_id=2J2YIKX5ONI3MV4R5POJXU4UA3U73PYWIEFXF2ZO5DKAABEXD6QA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:35.272032800 (548h5m35.143115835s) [noise=true , address=45.91.116.11:4120 , piece_id=Y2GZFQPBNRSNNV32QPSXQKNXA6AF4SJLITJLHVBQN5ZFKAI2ANVQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: <nil> Dec 28 11:34:35.272053630 (548h5m35.143136675s) [seg_pos={0 0}, upload=480597]: Segment reached optimal threshold of 13 pieces. Dec 28 11:34:35.272313985 (548h5m35.14339703s) [noise=true , address=45.91.116.13:4153 , piece_id=TNCDU6FYKLKFMRE3XMQCN7D2VG5VQXDZK2YEGZSGBR7IBES2BOSQ, seg_pos={0 0}, upload=480597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 28 11:34:35.272322852 (548h5m35.143405887s) [noise=true , address=45.91.116.13:4153 , piece_id=TNCDU6FYKLKFMRE3XMQCN7D2VG5VQXDZK2YEGZSGBR7IBES2BOSQ, seg_pos={0 0}, upload=480597]: Long tail cancellation due to optimal threshold Dec 28 11:34:35.272548011 (548h5m35.143631056s) [noise=true , address=45.91.116.10:4081 , piece_id=D4WL3ZCAQ5D65HDHOZ37T7RUZKRIK4PLHCQKNLIETHW3XJI2QT7A, seg_pos={0 0}, upload=480597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 11:34:35.272554263 (548h5m35.143637298s) [noise=true , address=45.91.116.10:4081 , piece_id=D4WL3ZCAQ5D65HDHOZ37T7RUZKRIK4PLHCQKNLIETHW3XJI2QT7A, seg_pos={0 0}, upload=480597]: Long tail cancellation due to optimal threshold Dec 28 11:34:35.272612814 (548h5m35.143695869s) [noise=true , address=45.91.116.12:4228 , piece_id=MAKKLTLELGFI2ILAQ2HSX7DHY5VD24CU75BLX2Q627RGL3UQMBQA, seg_pos={0 0}, upload=480597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 11:34:35.272655736 (548h5m35.143738781s) [noise=true , address=45.91.116.12:4228 , piece_id=MAKKLTLELGFI2ILAQ2HSX7DHY5VD24CU75BLX2Q627RGL3UQMBQA, seg_pos={0 0}, upload=480597]: Long tail cancellation due to optimal threshold Dec 28 11:34:35.272690582 (548h5m35.143773627s) [seg_pos={0 0}, upload=480597]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 11:34:35.272858011 (548h5m35.143941046s) [upload=480597]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 11:34:35.272866517 (548h5m35.143949562s) [upload=480597]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 11:34:35.564025180 (548h5m35.435108215s) [upload=480597]: Done uploading segments... Dec 28 11:34:35.564047874 (548h5m35.435130909s) [upload=480597]: Upload finished. err: <nil> Duration: 30.012862896s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [....X.X...X.....XX.XX..XXX...X.X....X..XXXX..X.X.XX.....X.X.] duration: 3.433406211s trace: https://jaeger.dev.storj.io/trace/4f8bf70751cdb26 FAIL (2025-12-28 11:34:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.433430827s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)