Orbiter History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 3.522067125s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:42:27 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.522094567s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...XX.....] duration: 6.397087453s trace: https://jaeger.dev.storj.io/trace/785dcaf51e874964 FAIL (2025-12-28 18:42:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA) context deadline exceeded ======LOGS====== 9m49.795105852s) [upload=486719]: Uploading segments... Dec 28 18:38:49.924182661 (555h9m49.795265696s) [upload=486719]: Got next segment. Inline: false Dec 28 18:38:49.924242615 (555h9m49.79532566s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:38:49.924247405 (555h9m49.79533045s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:38:50.170791617 (555h9m50.041874682s) [seg_pos={0 0}, upload=486719]: Begin upload segment... Dec 28 18:38:50.170857693 (555h9m50.041940748s) [seg_pos={0 0}, upload=486719]: Done begin upload segment. Dec 28 18:38:50.170919190 (555h9m50.042002225s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.170984374 (555h9m50.04206742s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171606730 (555h9m50.042689765s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171837629 (555h9m50.042920664s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172032620 (555h9m50.043115666s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172435748 (555h9m50.043518784s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172785054 (555h9m50.043868099s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173070988 (555h9m50.044154033s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173379245 (555h9m50.04446228s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173593743 (555h9m50.044676778s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173951284 (555h9m50.045034319s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174139122 (555h9m50.045222157s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174493948 (555h9m50.045576983s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174752059 (555h9m50.045835094s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175082959 (555h9m50.046165994s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175143824 (555h9m50.046226859s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:51.887343999 (555h9m51.758427034s) [upload=486719]: Next returned nil segment Dec 28 18:38:51.887358918 (555h9m51.758441963s) [upload=486719]: Waiting for error group managing segments... Dec 28 18:38:52.172766574 (555h9m52.043849609s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.176871520 (555h9m52.047954565s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.182783407 (555h9m52.053866452s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.189914355 (555h9m52.06099739s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.193182439 (555h9m52.064265474s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.194753169 (555h9m52.065836204s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.195031610 (555h9m52.066114645s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.198391939 (555h9m52.069474974s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.209163559 (555h9m52.080246604s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.402848580 (555h9m52.273931625s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.449323697 (555h9m52.320406742s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.520405688 (555h9m52.391488733s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655345046 (555h9m52.526428082s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655373009 (555h9m52.526456044s) [seg_pos={0 0}, upload=486719]: Segment reached optimal threshold of 13 pieces. Dec 28 18:38:52.655557942 (555h9m52.526640977s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection Dec 28 18:38:52.655563672 (555h9m52.526646708s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655657632 (555h9m52.526740667s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection Dec 28 18:38:52.655664174 (555h9m52.526747209s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655775666 (555h9m52.526858972s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 18:38:52.655798770 (555h9m52.526881805s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655860497 (555h9m52.526943542s) [seg_pos={0 0}, upload=486719]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:38:52.656024640 (555h9m52.527107675s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:38:52.656042815 (555h9m52.52712587s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:38:52.951505516 (555h9m52.822588561s) [upload=486719]: Done uploading segments... Dec 28 18:38:52.951531356 (555h9m52.822614401s) [upload=486719]: Upload finished. err: <nil> Duration: 6.397113192s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.............X.X........................X.................] duration: 35.31266352s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:41:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 35.312695762s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 3.257122605s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:41:20 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.257152514s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...XX....] duration: 7.817087565s trace: https://jaeger.dev.storj.io/trace/785dcaf51e874964 FAIL (2025-12-28 18:41:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA) context deadline exceeded ======LOGS====== 9m49.795105852s) [upload=486719]: Uploading segments... Dec 28 18:38:49.924182661 (555h9m49.795265696s) [upload=486719]: Got next segment. Inline: false Dec 28 18:38:49.924242615 (555h9m49.79532566s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:38:49.924247405 (555h9m49.79533045s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:38:50.170791617 (555h9m50.041874682s) [seg_pos={0 0}, upload=486719]: Begin upload segment... Dec 28 18:38:50.170857693 (555h9m50.041940748s) [seg_pos={0 0}, upload=486719]: Done begin upload segment. Dec 28 18:38:50.170919190 (555h9m50.042002225s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.170984374 (555h9m50.04206742s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171606730 (555h9m50.042689765s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171837629 (555h9m50.042920664s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172032620 (555h9m50.043115666s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172435748 (555h9m50.043518784s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172785054 (555h9m50.043868099s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173070988 (555h9m50.044154033s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173379245 (555h9m50.04446228s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173593743 (555h9m50.044676778s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173951284 (555h9m50.045034319s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174139122 (555h9m50.045222157s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174493948 (555h9m50.045576983s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174752059 (555h9m50.045835094s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175082959 (555h9m50.046165994s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175143824 (555h9m50.046226859s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:51.887343999 (555h9m51.758427034s) [upload=486719]: Next returned nil segment Dec 28 18:38:51.887358918 (555h9m51.758441963s) [upload=486719]: Waiting for error group managing segments... Dec 28 18:38:52.172766574 (555h9m52.043849609s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.176871520 (555h9m52.047954565s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.182783407 (555h9m52.053866452s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.189914355 (555h9m52.06099739s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.193182439 (555h9m52.064265474s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.194753169 (555h9m52.065836204s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.195031610 (555h9m52.066114645s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.198391939 (555h9m52.069474974s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.209163559 (555h9m52.080246604s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.402848580 (555h9m52.273931625s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.449323697 (555h9m52.320406742s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.520405688 (555h9m52.391488733s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655345046 (555h9m52.526428082s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655373009 (555h9m52.526456044s) [seg_pos={0 0}, upload=486719]: Segment reached optimal threshold of 13 pieces. Dec 28 18:38:52.655557942 (555h9m52.526640977s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection Dec 28 18:38:52.655563672 (555h9m52.526646708s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655657632 (555h9m52.526740667s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection Dec 28 18:38:52.655664174 (555h9m52.526747209s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655775666 (555h9m52.526858972s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 18:38:52.655798770 (555h9m52.526881805s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655860497 (555h9m52.526943542s) [seg_pos={0 0}, upload=486719]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:38:52.656024640 (555h9m52.527107675s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:38:52.656042815 (555h9m52.52712587s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:38:52.951505516 (555h9m52.822588561s) [upload=486719]: Done uploading segments... Dec 28 18:38:52.951531356 (555h9m52.822614401s) [upload=486719]: Upload finished. err: <nil> Duration: 7.817120047s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...XX...] duration: 5.874268183s trace: https://jaeger.dev.storj.io/trace/785dcaf51e874964 FAIL (2025-12-28 18:40:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA) context deadline exceeded ======LOGS====== 9m49.795105852s) [upload=486719]: Uploading segments... Dec 28 18:38:49.924182661 (555h9m49.795265696s) [upload=486719]: Got next segment. Inline: false Dec 28 18:38:49.924242615 (555h9m49.79532566s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:38:49.924247405 (555h9m49.79533045s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:38:50.170791617 (555h9m50.041874682s) [seg_pos={0 0}, upload=486719]: Begin upload segment... Dec 28 18:38:50.170857693 (555h9m50.041940748s) [seg_pos={0 0}, upload=486719]: Done begin upload segment. Dec 28 18:38:50.170919190 (555h9m50.042002225s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.170984374 (555h9m50.04206742s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171606730 (555h9m50.042689765s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171837629 (555h9m50.042920664s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172032620 (555h9m50.043115666s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172435748 (555h9m50.043518784s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172785054 (555h9m50.043868099s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173070988 (555h9m50.044154033s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173379245 (555h9m50.04446228s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173593743 (555h9m50.044676778s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173951284 (555h9m50.045034319s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174139122 (555h9m50.045222157s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174493948 (555h9m50.045576983s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174752059 (555h9m50.045835094s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175082959 (555h9m50.046165994s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175143824 (555h9m50.046226859s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:51.887343999 (555h9m51.758427034s) [upload=486719]: Next returned nil segment Dec 28 18:38:51.887358918 (555h9m51.758441963s) [upload=486719]: Waiting for error group managing segments... Dec 28 18:38:52.172766574 (555h9m52.043849609s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.176871520 (555h9m52.047954565s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.182783407 (555h9m52.053866452s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.189914355 (555h9m52.06099739s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.193182439 (555h9m52.064265474s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.194753169 (555h9m52.065836204s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.195031610 (555h9m52.066114645s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.198391939 (555h9m52.069474974s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.209163559 (555h9m52.080246604s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.402848580 (555h9m52.273931625s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.449323697 (555h9m52.320406742s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.520405688 (555h9m52.391488733s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655345046 (555h9m52.526428082s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655373009 (555h9m52.526456044s) [seg_pos={0 0}, upload=486719]: Segment reached optimal threshold of 13 pieces. Dec 28 18:38:52.655557942 (555h9m52.526640977s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection Dec 28 18:38:52.655563672 (555h9m52.526646708s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655657632 (555h9m52.526740667s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection Dec 28 18:38:52.655664174 (555h9m52.526747209s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655775666 (555h9m52.526858972s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 18:38:52.655798770 (555h9m52.526881805s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655860497 (555h9m52.526943542s) [seg_pos={0 0}, upload=486719]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:38:52.656024640 (555h9m52.527107675s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:38:52.656042815 (555h9m52.52712587s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:38:52.951505516 (555h9m52.822588561s) [upload=486719]: Done uploading segments... Dec 28 18:38:52.951531356 (555h9m52.822614401s) [upload=486719]: Upload finished. err: <nil> Duration: 5.874293421s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 3.486170943s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:40:58 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.486200048s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.............X.X........................X................] duration: 46.200190898s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:40:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 46.200210846s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...XX..] duration: 19.778656854s trace: https://jaeger.dev.storj.io/trace/785dcaf51e874964 FAIL (2025-12-28 18:40:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA) context deadline exceeded ======LOGS====== 9m49.795105852s) [upload=486719]: Uploading segments... Dec 28 18:38:49.924182661 (555h9m49.795265696s) [upload=486719]: Got next segment. Inline: false Dec 28 18:38:49.924242615 (555h9m49.79532566s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:38:49.924247405 (555h9m49.79533045s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:38:50.170791617 (555h9m50.041874682s) [seg_pos={0 0}, upload=486719]: Begin upload segment... Dec 28 18:38:50.170857693 (555h9m50.041940748s) [seg_pos={0 0}, upload=486719]: Done begin upload segment. Dec 28 18:38:50.170919190 (555h9m50.042002225s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.170984374 (555h9m50.04206742s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171606730 (555h9m50.042689765s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171837629 (555h9m50.042920664s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172032620 (555h9m50.043115666s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172435748 (555h9m50.043518784s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172785054 (555h9m50.043868099s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173070988 (555h9m50.044154033s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173379245 (555h9m50.04446228s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173593743 (555h9m50.044676778s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173951284 (555h9m50.045034319s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174139122 (555h9m50.045222157s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174493948 (555h9m50.045576983s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174752059 (555h9m50.045835094s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175082959 (555h9m50.046165994s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175143824 (555h9m50.046226859s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:51.887343999 (555h9m51.758427034s) [upload=486719]: Next returned nil segment Dec 28 18:38:51.887358918 (555h9m51.758441963s) [upload=486719]: Waiting for error group managing segments... Dec 28 18:38:52.172766574 (555h9m52.043849609s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.176871520 (555h9m52.047954565s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.182783407 (555h9m52.053866452s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.189914355 (555h9m52.06099739s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.193182439 (555h9m52.064265474s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.194753169 (555h9m52.065836204s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.195031610 (555h9m52.066114645s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.198391939 (555h9m52.069474974s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.209163559 (555h9m52.080246604s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.402848580 (555h9m52.273931625s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.449323697 (555h9m52.320406742s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.520405688 (555h9m52.391488733s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655345046 (555h9m52.526428082s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655373009 (555h9m52.526456044s) [seg_pos={0 0}, upload=486719]: Segment reached optimal threshold of 13 pieces. Dec 28 18:38:52.655557942 (555h9m52.526640977s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection Dec 28 18:38:52.655563672 (555h9m52.526646708s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655657632 (555h9m52.526740667s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection Dec 28 18:38:52.655664174 (555h9m52.526747209s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655775666 (555h9m52.526858972s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 18:38:52.655798770 (555h9m52.526881805s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655860497 (555h9m52.526943542s) [seg_pos={0 0}, upload=486719]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:38:52.656024640 (555h9m52.527107675s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:38:52.656042815 (555h9m52.52712587s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:38:52.951505516 (555h9m52.822588561s) [upload=486719]: Done uploading segments... Dec 28 18:38:52.951531356 (555h9m52.822614401s) [upload=486719]: Upload finished. err: <nil> Duration: 19.778695909s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 3.42082354s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:40:07 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.420861232s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.............X.X........................X...............] duration: 27.567127762s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:40:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 27.567274682s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...XX.] duration: 6.474394087s trace: https://jaeger.dev.storj.io/trace/785dcaf51e874964 FAIL (2025-12-28 18:39:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA) context deadline exceeded ======LOGS====== 9m49.795105852s) [upload=486719]: Uploading segments... Dec 28 18:38:49.924182661 (555h9m49.795265696s) [upload=486719]: Got next segment. Inline: false Dec 28 18:38:49.924242615 (555h9m49.79532566s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:38:49.924247405 (555h9m49.79533045s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:38:50.170791617 (555h9m50.041874682s) [seg_pos={0 0}, upload=486719]: Begin upload segment... Dec 28 18:38:50.170857693 (555h9m50.041940748s) [seg_pos={0 0}, upload=486719]: Done begin upload segment. Dec 28 18:38:50.170919190 (555h9m50.042002225s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.170984374 (555h9m50.04206742s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171606730 (555h9m50.042689765s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171837629 (555h9m50.042920664s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172032620 (555h9m50.043115666s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172435748 (555h9m50.043518784s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172785054 (555h9m50.043868099s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173070988 (555h9m50.044154033s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173379245 (555h9m50.04446228s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173593743 (555h9m50.044676778s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173951284 (555h9m50.045034319s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174139122 (555h9m50.045222157s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174493948 (555h9m50.045576983s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174752059 (555h9m50.045835094s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175082959 (555h9m50.046165994s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175143824 (555h9m50.046226859s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:51.887343999 (555h9m51.758427034s) [upload=486719]: Next returned nil segment Dec 28 18:38:51.887358918 (555h9m51.758441963s) [upload=486719]: Waiting for error group managing segments... Dec 28 18:38:52.172766574 (555h9m52.043849609s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.176871520 (555h9m52.047954565s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.182783407 (555h9m52.053866452s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.189914355 (555h9m52.06099739s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.193182439 (555h9m52.064265474s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.194753169 (555h9m52.065836204s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.195031610 (555h9m52.066114645s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.198391939 (555h9m52.069474974s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.209163559 (555h9m52.080246604s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.402848580 (555h9m52.273931625s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.449323697 (555h9m52.320406742s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.520405688 (555h9m52.391488733s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655345046 (555h9m52.526428082s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655373009 (555h9m52.526456044s) [seg_pos={0 0}, upload=486719]: Segment reached optimal threshold of 13 pieces. Dec 28 18:38:52.655557942 (555h9m52.526640977s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection Dec 28 18:38:52.655563672 (555h9m52.526646708s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655657632 (555h9m52.526740667s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection Dec 28 18:38:52.655664174 (555h9m52.526747209s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655775666 (555h9m52.526858972s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 18:38:52.655798770 (555h9m52.526881805s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655860497 (555h9m52.526943542s) [seg_pos={0 0}, upload=486719]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:38:52.656024640 (555h9m52.527107675s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:38:52.656042815 (555h9m52.52712587s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:38:52.951505516 (555h9m52.822588561s) [upload=486719]: Done uploading segments... Dec 28 18:38:52.951531356 (555h9m52.822614401s) [upload=486719]: Upload finished. err: <nil> Duration: 6.47442231s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.............X.X........................X..............] duration: 18.829213187s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:39:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 18.829239257s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...XX] duration: 30.002671516s trace: https://jaeger.dev.storj.io/trace/785dcaf51e874964 FAIL (2025-12-28 18:39:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA) context deadline exceeded ======LOGS====== 9m49.795105852s) [upload=486719]: Uploading segments... Dec 28 18:38:49.924182661 (555h9m49.795265696s) [upload=486719]: Got next segment. Inline: false Dec 28 18:38:49.924242615 (555h9m49.79532566s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:38:49.924247405 (555h9m49.79533045s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:38:50.170791617 (555h9m50.041874682s) [seg_pos={0 0}, upload=486719]: Begin upload segment... Dec 28 18:38:50.170857693 (555h9m50.041940748s) [seg_pos={0 0}, upload=486719]: Done begin upload segment. Dec 28 18:38:50.170919190 (555h9m50.042002225s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.170984374 (555h9m50.04206742s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171606730 (555h9m50.042689765s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.171837629 (555h9m50.042920664s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172032620 (555h9m50.043115666s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172435748 (555h9m50.043518784s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.172785054 (555h9m50.043868099s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173070988 (555h9m50.044154033s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173379245 (555h9m50.04446228s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173593743 (555h9m50.044676778s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.173951284 (555h9m50.045034319s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174139122 (555h9m50.045222157s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174493948 (555h9m50.045576983s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.174752059 (555h9m50.045835094s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175082959 (555h9m50.046165994s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:50.175143824 (555h9m50.046226859s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Uploading piece... Dec 28 18:38:51.887343999 (555h9m51.758427034s) [upload=486719]: Next returned nil segment Dec 28 18:38:51.887358918 (555h9m51.758441963s) [upload=486719]: Waiting for error group managing segments... Dec 28 18:38:52.172766574 (555h9m52.043849609s) [noise=true , address=145.79.227.46:4603 , piece_id=JFCDYUXU6BOPZ45TYODMY5BOFS6YRZW7V4TMJTW7YDPK5BJLMTGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.176871520 (555h9m52.047954565s) [noise=true , address=145.79.227.15:4876 , piece_id=PZ4V6ITXVMNRPSIV3BWYVEPTTAMD6IKGJMSGLF22O6HO7WEY5EXQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.182783407 (555h9m52.053866452s) [noise=true , address=145.79.227.37:4565 , piece_id=WFR2T2RAA5MQBM4BZLKH3IGJO6WGHRYV7Y452GRWVP6XAYA3HC6Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.189914355 (555h9m52.06099739s) [noise=true , address=145.79.227.26:4653 , piece_id=PJDOZZ3U47HVXYJIIWXX5XKFOUKLAPOZ6G32MCMYUA5TTLFWNTYA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.193182439 (555h9m52.064265474s) [noise=true , address=145.79.227.50:4374 , piece_id=SGSBYBA63MRY2BWUO2NNHYX6XZXYT5HF2JHDWYBQCVGYYJ2YJ26A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.194753169 (555h9m52.065836204s) [noise=true , address=145.79.227.38:4978 , piece_id=FSKOXY2ONT3OKULTEIGYOJJHRRRE6WSCWBZNP4623PT7EO4J3KEQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.195031610 (555h9m52.066114645s) [noise=true , address=145.79.227.12:4765 , piece_id=LAV4QBZUFLZP4CR4YD7D7DN3K3ZD3YNLAOJURSMRKRYX2U5JS23Q, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.198391939 (555h9m52.069474974s) [noise=true , address=145.79.227.4:4187 , piece_id=PMG47QU6NRQTJPPTWYNB4Y5GZJUM7N6CJFFMQM74CPH6Z2UEMFGA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.209163559 (555h9m52.080246604s) [noise=true , address=145.79.227.2:4114 , piece_id=2WBYK5RHMVQYTLBUNFIMETOHKSVT7JTJTILSDMK6CQNWV7M3GGIA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.402848580 (555h9m52.273931625s) [noise=true , address=145.79.227.47:4265 , piece_id=FLVWOVGDIJT3J4ZEXVRUYCM2X2VIQE5EC6ZW3B3PN44UC2RPDUOA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.449323697 (555h9m52.320406742s) [noise=true , address=45.91.116.13:4155 , piece_id=H44BFVDNT2UUABDJUDTPGYBGGFH2DI7AU2GLMMOE54D6IAIMKIXA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.520405688 (555h9m52.391488733s) [noise=true , address=77.90.31.12:4589 , piece_id=YJXQ3NO2S3BEKYNFDF2BAXZHVGMGU5YCHCNG6DLWCVBXPQ5TLI7A, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655345046 (555h9m52.526428082s) [noise=true , address=77.90.31.13:4627 , piece_id=SSJSW7XOK6MXGLHQA72JPG27P3BVFXPMWKUMTSXSS7ZGZ6U72VFA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: <nil> Dec 28 18:38:52.655373009 (555h9m52.526456044s) [seg_pos={0 0}, upload=486719]: Segment reached optimal threshold of 13 pieces. Dec 28 18:38:52.655557942 (555h9m52.526640977s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39268->45.91.116.8:4047: use of closed network connection Dec 28 18:38:52.655563672 (555h9m52.526646708s) [noise=true , address=45.91.116.8:4047 , piece_id=PUG4SU5DMNPBJZIO4JAE3HG3BQ3AYA3CPS4GSGM6XFXSLWOXGSKQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655657632 (555h9m52.526740667s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55744->145.79.227.48:4302: use of closed network connection Dec 28 18:38:52.655664174 (555h9m52.526747209s) [noise=true , address=145.79.227.48:4302 , piece_id=MSOW3EKUYEJAWXDVEPUNK4AHBJDAHHDLAI2R55NPA7W3QOLY34YQ, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655775666 (555h9m52.526858972s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 18:38:52.655798770 (555h9m52.526881805s) [noise=true , address=77.90.31.21:4878 , piece_id=WX6VV3KSILSFTSCDD3F3MWQD7ECIJYVDPBNZLXHN57RRQXRYC6SA, seg_pos={0 0}, upload=486719]: Long tail cancellation due to optimal threshold Dec 28 18:38:52.655860497 (555h9m52.526943542s) [seg_pos={0 0}, upload=486719]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:38:52.656024640 (555h9m52.527107675s) [upload=486719]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:38:52.656042815 (555h9m52.52712587s) [upload=486719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:38:52.951505516 (555h9m52.822588561s) [upload=486719]: Done uploading segments... Dec 28 18:38:52.951531356 (555h9m52.822614401s) [upload=486719]: Upload finished. err: <nil> Duration: 30.002693608s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 3.390520938s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:39:04 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.390551416s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 3.516215072s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:38:40 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.516241894s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.............X.X........................X.............] duration: 33.628723708s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:38:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 33.62873975s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...X] duration: 30.002813589s trace: https://jaeger.dev.storj.io/trace/4fbed146ec31b60f FAIL (2025-12-28 18:38:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: O45R525ETFGWT43PDWDUJV5VYPIKXUDHDHR4JKAKPOWZ6BRUGULA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: UIPUC3NJRS4S6BAASNOF7E4LUAMJFCJUT76FRPAMWJH6PXD344VA) context deadline exceeded ======LOGS====== Dec 28 18:37:47.952924132 (555h8m47.824007167s) [upload=486703]: Starting upload Dec 28 18:37:47.952996239 (555h8m47.824079274s) [upload=486703]: Done starting upload Dec 28 18:37:47.953044731 (555h8m47.824127766s) [upload=486703]: Uploading segments... Dec 28 18:37:47.953210126 (555h8m47.824293171s) [upload=486703]: Got next segment. Inline: false Dec 28 18:37:47.953242157 (555h8m47.824325192s) [upload=486703]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:37:47.953247898 (555h8m47.824330933s) [upload=486703]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:37:48.203667200 (555h8m48.074750265s) [seg_pos={0 0}, upload=486703]: Begin upload segment... Dec 28 18:37:48.203720922 (555h8m48.074803957s) [seg_pos={0 0}, upload=486703]: Done begin upload segment. Dec 28 18:37:48.203734067 (555h8m48.074817112s) [noise=true , address=77.90.31.11:4556 , piece_id=JZTPKCAMECPIJKWWXJCANZRXUCMIDRELAUXZTF537ETNSBOJMRMQ, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.203749928 (555h8m48.074832963s) [noise=true , address=45.91.116.5:4298 , piece_id=O45R525ETFGWT43PDWDUJV5VYPIKXUDHDHR4JKAKPOWZ6BRUGULA, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.203916956 (555h8m48.074999991s) [noise=true , address=145.79.227.36:4519 , piece_id=LHQKDQ6XHMOZFOU66MMMLKBF2W5R6WY7UQ6WLAUKPQOM4KNYFGGQ, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.204281420 (555h8m48.075364455s) [noise=true , address=145.79.227.16:4911 , piece_id=HCWHTXYBAT2S76PVNDAW7VZD2G5BOVRVVI7S57KO52VE2PYYUBVA, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.204616819 (555h8m48.075699854s) [noise=true , address=145.79.227.29:4942 , piece_id=32IV2553K5LM4IBPODY4PC36VY4AWDWYMBGJC6S3VC54K5DRUQZA, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.204884868 (555h8m48.075967903s) [noise=true , address=145.79.227.2:4113 , piece_id=LG6UO4VALLB4IMLBZSE44ST3SZUYMVF6XMQ3QDS5I752INHUXO4A, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.205146527 (555h8m48.076229562s) [noise=true , address=45.91.116.10:4081 , piece_id=SPBBD6TRT5KLITNFB4THJS5JGALVCIGGNBJNF57XIPPAQ4VWTX6A, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.205410911 (555h8m48.076493946s) [noise=true , address=145.79.227.9:4366 , piece_id=7VDYG6UIXNXBD636HAM253FHQKOC5W32GHFLQ3CNSR2JZSF3T6MQ, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.205723767 (555h8m48.076806812s) [noise=true , address=145.79.227.46:4602 , piece_id=MUBSLMB7UXCE23B6RB7ZTVXRXT4MHKZZ3R4X23BKCCC5NNDKD2HQ, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.206052121 (555h8m48.077135156s) [noise=true , address=145.79.227.3:4151 , piece_id=DZYAVTVZR6KUYTBIGT3ZOU3IHP6AZ7K4A7TYQR3EN7LSGSQLIL3Q, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.206368544 (555h8m48.077451589s) [noise=true , address=145.79.227.50:4374 , piece_id=FREM4FLXUJVAWIRRBB5NE4YSXKJ6QIUDVKHTCMLUEEAKPAKSJSVQ, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.206793653 (555h8m48.077876688s) [noise=true , address=145.79.227.7:4295 , piece_id=S7V6JQCIWML56ZHSI6Y7FUE7H4L5HOYI3ZB3KECQ4MQHEN6HAE2Q, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.207779020 (555h8m48.078862055s) [noise=true , address=145.79.227.19:4078 , piece_id=42ESUDAV7R45YMJGTWYFVFTW36AOJN42DXL6JYV75BS7AETRPP6A, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.208788222 (555h8m48.079871257s) [noise=true , address=145.79.227.30:4761 , piece_id=NXTV5NYP7NCS3CKKZLHNCEBKVOWGMWNE7S7ONXVZUVR6TXL3IBLQ, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.209870072 (555h8m48.080953107s) [noise=true , address=45.91.116.14:4191 , piece_id=UIPUC3NJRS4S6BAASNOF7E4LUAMJFCJUT76FRPAMWJH6PXD344VA, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:48.211042934 (555h8m48.082125969s) [noise=true , address=45.91.116.6:4975 , piece_id=3B24J7HKP323UDIOCACO5BLGP6BLDJZLF76IQQRBNL4SJ6T62HEA, seg_pos={0 0}, upload=486703]: Uploading piece... Dec 28 18:37:49.825604830 (555h8m49.696687875s) [upload=486703]: Next returned nil segment Dec 28 18:37:49.825713468 (555h8m49.696796533s) [upload=486703]: Waiting for error group managing segments... Dec 28 18:37:50.202777853 (555h8m50.073860898s) [noise=true , address=145.79.227.46:4602 , piece_id=MUBSLMB7UXCE23B6RB7ZTVXRXT4MHKZZ3R4X23BKCCC5NNDKD2HQ, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.213864273 (555h8m50.084947308s) [noise=true , address=145.79.227.3:4151 , piece_id=DZYAVTVZR6KUYTBIGT3ZOU3IHP6AZ7K4A7TYQR3EN7LSGSQLIL3Q, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.214459827 (555h8m50.085542872s) [noise=true , address=145.79.227.2:4113 , piece_id=LG6UO4VALLB4IMLBZSE44ST3SZUYMVF6XMQ3QDS5I752INHUXO4A, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.223913449 (555h8m50.094996484s) [noise=true , address=145.79.227.19:4078 , piece_id=42ESUDAV7R45YMJGTWYFVFTW36AOJN42DXL6JYV75BS7AETRPP6A, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.226410564 (555h8m50.097493609s) [noise=true , address=145.79.227.7:4295 , piece_id=S7V6JQCIWML56ZHSI6Y7FUE7H4L5HOYI3ZB3KECQ4MQHEN6HAE2Q, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.227179488 (555h8m50.098262523s) [noise=true , address=145.79.227.9:4366 , piece_id=7VDYG6UIXNXBD636HAM253FHQKOC5W32GHFLQ3CNSR2JZSF3T6MQ, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.231591910 (555h8m50.102674965s) [noise=true , address=145.79.227.36:4519 , piece_id=LHQKDQ6XHMOZFOU66MMMLKBF2W5R6WY7UQ6WLAUKPQOM4KNYFGGQ, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.233960269 (555h8m50.105043314s) [noise=true , address=145.79.227.50:4374 , piece_id=FREM4FLXUJVAWIRRBB5NE4YSXKJ6QIUDVKHTCMLUEEAKPAKSJSVQ, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.269409902 (555h8m50.140492937s) [noise=true , address=145.79.227.16:4911 , piece_id=HCWHTXYBAT2S76PVNDAW7VZD2G5BOVRVVI7S57KO52VE2PYYUBVA, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.475175736 (555h8m50.346258771s) [noise=true , address=77.90.31.11:4556 , piece_id=JZTPKCAMECPIJKWWXJCANZRXUCMIDRELAUXZTF537ETNSBOJMRMQ, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.499048384 (555h8m50.370131419s) [noise=true , address=45.91.116.14:4191 , piece_id=UIPUC3NJRS4S6BAASNOF7E4LUAMJFCJUT76FRPAMWJH6PXD344VA, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.754699638 (555h8m50.625782673s) [noise=true , address=45.91.116.5:4298 , piece_id=O45R525ETFGWT43PDWDUJV5VYPIKXUDHDHR4JKAKPOWZ6BRUGULA, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.799090025 (555h8m50.67017306s) [noise=true , address=45.91.116.10:4081 , piece_id=SPBBD6TRT5KLITNFB4THJS5JGALVCIGGNBJNF57XIPPAQ4VWTX6A, seg_pos={0 0}, upload=486703]: Done uploading piece. err: <nil> Dec 28 18:37:50.799110284 (555h8m50.670193319s) [seg_pos={0 0}, upload=486703]: Segment reached optimal threshold of 13 pieces. Dec 28 18:37:50.799322979 (555h8m50.670406014s) [noise=true , address=145.79.227.29:4942 , piece_id=32IV2553K5LM4IBPODY4PC36VY4AWDWYMBGJC6S3VC54K5DRUQZA, seg_pos={0 0}, upload=486703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 28 18:37:50.799332867 (555h8m50.670415942s) [noise=true , address=145.79.227.29:4942 , piece_id=32IV2553K5LM4IBPODY4PC36VY4AWDWYMBGJC6S3VC54K5DRUQZA, seg_pos={0 0}, upload=486703]: Long tail cancellation due to optimal threshold Dec 28 18:37:50.799460621 (555h8m50.670543656s) [noise=true , address=145.79.227.30:4761 , piece_id=NXTV5NYP7NCS3CKKZLHNCEBKVOWGMWNE7S7ONXVZUVR6TXL3IBLQ, seg_pos={0 0}, upload=486703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 28 18:37:50.799466852 (555h8m50.670549887s) [noise=true , address=145.79.227.30:4761 , piece_id=NXTV5NYP7NCS3CKKZLHNCEBKVOWGMWNE7S7ONXVZUVR6TXL3IBLQ, seg_pos={0 0}, upload=486703]: Long tail cancellation due to optimal threshold Dec 28 18:37:50.799714804 (555h8m50.670797869s) [noise=true , address=45.91.116.6:4975 , piece_id=3B24J7HKP323UDIOCACO5BLGP6BLDJZLF76IQQRBNL4SJ6T62HEA, seg_pos={0 0}, upload=486703]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 28 18:37:50.799734803 (555h8m50.670817858s) [noise=true , address=45.91.116.6:4975 , piece_id=3B24J7HKP323UDIOCACO5BLGP6BLDJZLF76IQQRBNL4SJ6T62HEA, seg_pos={0 0}, upload=486703]: Long tail cancellation due to optimal threshold Dec 28 18:37:50.799762596 (555h8m50.670845631s) [seg_pos={0 0}, upload=486703]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:37:50.799889106 (555h8m50.670972151s) [upload=486703]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:37:50.799896490 (555h8m50.670979535s) [upload=486703]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:37:51.098664356 (555h8m50.969747401s) [upload=486703]: Done uploading segments... Dec 28 18:37:51.098690967 (555h8m50.969774002s) [upload=486703]: Upload finished. err: <nil> Duration: 30.002836231s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.............X.X........................X............] duration: 34.010076533s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:38:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 34.01010568s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 3.495097099s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:37:55 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.495125944s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X...] duration: 14.336671485s trace: https://jaeger.dev.storj.io/trace/76f49c76a8e1da20 FAIL (2025-12-28 18:37:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: MELX67YAXCOOIXHLPA3HWG6EA2GRKUOM5QXVEIAJ7BS5PMEXD5WA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: FYEIMUXCYS6QIY3BRDU2IZAGXTDNOXO2ENVSKZ2YRL46PUKXSUIA) context deadline exceeded ======LOGS====== c 28 18:33:58.722754987 (555h4m58.593838022s) [upload=486649]: Uploading segments... Dec 28 18:33:58.722808368 (555h4m58.593891403s) [upload=486649]: Got next segment. Inline: false Dec 28 18:33:58.722877549 (555h4m58.593960584s) [upload=486649]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:33:58.722882610 (555h4m58.593965645s) [upload=486649]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:33:58.968658119 (555h4m58.839741164s) [seg_pos={0 0}, upload=486649]: Begin upload segment... Dec 28 18:33:58.968718123 (555h4m58.839801158s) [seg_pos={0 0}, upload=486649]: Done begin upload segment. Dec 28 18:33:58.968748762 (555h4m58.839831797s) [noise=true , address=77.90.31.19:4842 , piece_id=FYEIMUXCYS6QIY3BRDU2IZAGXTDNOXO2ENVSKZ2YRL46PUKXSUIA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.968781634 (555h4m58.839864679s) [noise=true , address=77.90.31.12:4590 , piece_id=MSFP5QZQGVNJSZL7EHTRABC2ROWUQ4L22UGGHF6LCNG33WKABWQQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.969214989 (555h4m58.840298034s) [noise=true , address=145.79.227.52:4446 , piece_id=AOEDOIZO2ZVTYHDAHFACYKORLRIGGNF36M5BMUBTML7GMGCYLB5Q, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.969636361 (555h4m58.840719396s) [noise=true , address=77.90.31.13:4629 , piece_id=MELX67YAXCOOIXHLPA3HWG6EA2GRKUOM5QXVEIAJ7BS5PMEXD5WA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.970158345 (555h4m58.84124139s) [noise=true , address=145.79.227.12:4765 , piece_id=OSCF4NIIBQOL4UFJTALLXNCWCCNJXA2E5C357BIR4AT6457TINSQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.970701309 (555h4m58.841784345s) [noise=true , address=145.79.227.6:4257 , piece_id=3JDEIUV6BCME6EUKHKT2LLRMPQCGRRB6MBSMJ3KVFD73LH737WMA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.971126038 (555h4m58.842209073s) [noise=true , address=145.79.227.13:4804 , piece_id=3ZKXUQNJSUOXL6MPEXDFPP4DOKM7JTLBWQH4MY6PUPPK6S5L34VQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.971570705 (555h4m58.84265374s) [noise=true , address=145.79.227.19:4078 , piece_id=VQRDD6ESLBAUNSOY3F4C36NEC5FDS4HWDQQ3HHE6QFYCN3DVTG5Q, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.972646433 (555h4m58.843729478s) [noise=true , address=145.79.227.47:4265 , piece_id=SG55NEYJEBCK2KBFZ7U3PLQOUK7P5DO6Z5DRQ2DUWX2QYUZJUYFA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.972911328 (555h4m58.843994373s) [noise=true , address=145.79.227.45:4231 , piece_id=J2DCYUTGYDQJ7FGVNGEWZQA2JNJ7IBV2NDGPFPUVBNYLM4QHMD5Q, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.973974452 (555h4m58.845057487s) [noise=true , address=77.90.31.18:4808 , piece_id=NHM3KRMS6EGNCCE26C6FBZEW3TDMP4FXN5WE47PY6ODRE27LRLKQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.975212530 (555h4m58.846295565s) [noise=true , address=145.79.227.21:4475 , piece_id=GGBKTA4ICTXO3BWP5MYKWKOWMSCMKRVDWGV2TNWWBNOYCARKF35Q, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.978087724 (555h4m58.849170769s) [noise=true , address=145.79.227.7:4294 , piece_id=AOGDE3GEOSHGLZBF764HP6TFYEHRSGG4L2BFOQFUJE7UWOOAVRJA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.981692148 (555h4m58.852775193s) [noise=true , address=45.91.116.4:4939 , piece_id=RKDLUQITGFAHQV4AE2TCPTNCNAYARVZ25MF7JZLZZE4LPUQMVN3A, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.983781126 (555h4m58.854864161s) [noise=true , address=145.79.227.28:4727 , piece_id=RDEWPTFGENDP5VNTWQ36JS3VDB62UIZRO5T3EZYMKQ3PRXLNPIRQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.987752909 (555h4m58.858835954s) [noise=true , address=145.79.227.15:4875 , piece_id=KSZC4UZW5KMDXRSSRRKVWATYXERGXCJAK6MLPNXCW2F5FBLERONA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:34:00.690994828 (555h5m0.562077863s) [upload=486649]: Next returned nil segment Dec 28 18:34:00.691008914 (555h5m0.562091949s) [upload=486649]: Waiting for error group managing segments... Dec 28 18:34:00.977746422 (555h5m0.848829457s) [noise=true , address=145.79.227.12:4765 , piece_id=OSCF4NIIBQOL4UFJTALLXNCWCCNJXA2E5C357BIR4AT6457TINSQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.978844533 (555h5m0.849927568s) [noise=true , address=145.79.227.52:4446 , piece_id=AOEDOIZO2ZVTYHDAHFACYKORLRIGGNF36M5BMUBTML7GMGCYLB5Q, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.986731873 (555h5m0.857814918s) [noise=true , address=145.79.227.21:4475 , piece_id=GGBKTA4ICTXO3BWP5MYKWKOWMSCMKRVDWGV2TNWWBNOYCARKF35Q, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.987946485 (555h5m0.85902953s) [noise=true , address=145.79.227.15:4875 , piece_id=KSZC4UZW5KMDXRSSRRKVWATYXERGXCJAK6MLPNXCW2F5FBLERONA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.992659470 (555h5m0.863742505s) [noise=true , address=145.79.227.6:4257 , piece_id=3JDEIUV6BCME6EUKHKT2LLRMPQCGRRB6MBSMJ3KVFD73LH737WMA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.999582272 (555h5m0.870665317s) [noise=true , address=145.79.227.19:4078 , piece_id=VQRDD6ESLBAUNSOY3F4C36NEC5FDS4HWDQQ3HHE6QFYCN3DVTG5Q, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.005606773 (555h5m0.876689829s) [noise=true , address=145.79.227.45:4231 , piece_id=J2DCYUTGYDQJ7FGVNGEWZQA2JNJ7IBV2NDGPFPUVBNYLM4QHMD5Q, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.006067911 (555h5m0.877150956s) [noise=true , address=145.79.227.47:4265 , piece_id=SG55NEYJEBCK2KBFZ7U3PLQOUK7P5DO6Z5DRQ2DUWX2QYUZJUYFA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.011528650 (555h5m0.882611695s) [noise=true , address=145.79.227.28:4727 , piece_id=RDEWPTFGENDP5VNTWQ36JS3VDB62UIZRO5T3EZYMKQ3PRXLNPIRQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.049370617 (555h5m0.920453662s) [noise=true , address=145.79.227.13:4804 , piece_id=3ZKXUQNJSUOXL6MPEXDFPP4DOKM7JTLBWQH4MY6PUPPK6S5L34VQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.140450690 (555h5m1.011533755s) [noise=true , address=77.90.31.13:4629 , piece_id=MELX67YAXCOOIXHLPA3HWG6EA2GRKUOM5QXVEIAJ7BS5PMEXD5WA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.265885880 (555h5m1.136968935s) [noise=true , address=77.90.31.19:4842 , piece_id=FYEIMUXCYS6QIY3BRDU2IZAGXTDNOXO2ENVSKZ2YRL46PUKXSUIA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.266473128 (555h5m1.137556173s) [noise=true , address=77.90.31.12:4590 , piece_id=MSFP5QZQGVNJSZL7EHTRABC2ROWUQ4L22UGGHF6LCNG33WKABWQQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.266497515 (555h5m1.13758056s) [seg_pos={0 0}, upload=486649]: Segment reached optimal threshold of 13 pieces. Dec 28 18:34:01.267022615 (555h5m1.13810566s) [noise=true , address=77.90.31.18:4808 , piece_id=NHM3KRMS6EGNCCE26C6FBZEW3TDMP4FXN5WE47PY6ODRE27LRLKQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:38010->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:38010->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38010->77.90.31.18:4808: use of closed network connection Dec 28 18:34:01.267032714 (555h5m1.138115759s) [noise=true , address=77.90.31.18:4808 , piece_id=NHM3KRMS6EGNCCE26C6FBZEW3TDMP4FXN5WE47PY6ODRE27LRLKQ, seg_pos={0 0}, upload=486649]: Long tail cancellation due to optimal threshold Dec 28 18:34:01.267046881 (555h5m1.138129926s) [noise=true , address=45.91.116.4:4939 , piece_id=RKDLUQITGFAHQV4AE2TCPTNCNAYARVZ25MF7JZLZZE4LPUQMVN3A, seg_pos={0 0}, upload=486649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 18:34:01.267059134 (555h5m1.138142179s) [noise=true , address=45.91.116.4:4939 , piece_id=RKDLUQITGFAHQV4AE2TCPTNCNAYARVZ25MF7JZLZZE4LPUQMVN3A, seg_pos={0 0}, upload=486649]: Long tail cancellation due to optimal threshold Dec 28 18:34:01.267229859 (555h5m1.138312904s) [noise=true , address=145.79.227.7:4294 , piece_id=AOGDE3GEOSHGLZBF764HP6TFYEHRSGG4L2BFOQFUJE7UWOOAVRJA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:49112->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:49112->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->145.79.227.7:4294: use of closed network connection Dec 28 18:34:01.267238606 (555h5m1.138321661s) [noise=true , address=145.79.227.7:4294 , piece_id=AOGDE3GEOSHGLZBF764HP6TFYEHRSGG4L2BFOQFUJE7UWOOAVRJA, seg_pos={0 0}, upload=486649]: Long tail cancellation due to optimal threshold Dec 28 18:34:01.267260468 (555h5m1.138343513s) [seg_pos={0 0}, upload=486649]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:34:01.267375306 (555h5m1.138458351s) [upload=486649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:34:01.267383311 (555h5m1.138466356s) [upload=486649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:34:01.563507573 (555h5m1.434590618s) [upload=486649]: Done uploading segments... Dec 28 18:34:01.563533192 (555h5m1.434616237s) [upload=486649]: Upload finished. err: <nil> Duration: 14.336702553s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.............X.X........................X...........] duration: 39.058597139s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:37:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 39.0586137s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 3.478420489s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:36:58 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.478437251s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 3.395112977s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:36:23 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.395161519s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X..] duration: 5.933485499s trace: https://jaeger.dev.storj.io/trace/76f49c76a8e1da20 FAIL (2025-12-28 18:36:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: MELX67YAXCOOIXHLPA3HWG6EA2GRKUOM5QXVEIAJ7BS5PMEXD5WA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: FYEIMUXCYS6QIY3BRDU2IZAGXTDNOXO2ENVSKZ2YRL46PUKXSUIA) context deadline exceeded ======LOGS====== c 28 18:33:58.722754987 (555h4m58.593838022s) [upload=486649]: Uploading segments... Dec 28 18:33:58.722808368 (555h4m58.593891403s) [upload=486649]: Got next segment. Inline: false Dec 28 18:33:58.722877549 (555h4m58.593960584s) [upload=486649]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:33:58.722882610 (555h4m58.593965645s) [upload=486649]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:33:58.968658119 (555h4m58.839741164s) [seg_pos={0 0}, upload=486649]: Begin upload segment... Dec 28 18:33:58.968718123 (555h4m58.839801158s) [seg_pos={0 0}, upload=486649]: Done begin upload segment. Dec 28 18:33:58.968748762 (555h4m58.839831797s) [noise=true , address=77.90.31.19:4842 , piece_id=FYEIMUXCYS6QIY3BRDU2IZAGXTDNOXO2ENVSKZ2YRL46PUKXSUIA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.968781634 (555h4m58.839864679s) [noise=true , address=77.90.31.12:4590 , piece_id=MSFP5QZQGVNJSZL7EHTRABC2ROWUQ4L22UGGHF6LCNG33WKABWQQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.969214989 (555h4m58.840298034s) [noise=true , address=145.79.227.52:4446 , piece_id=AOEDOIZO2ZVTYHDAHFACYKORLRIGGNF36M5BMUBTML7GMGCYLB5Q, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.969636361 (555h4m58.840719396s) [noise=true , address=77.90.31.13:4629 , piece_id=MELX67YAXCOOIXHLPA3HWG6EA2GRKUOM5QXVEIAJ7BS5PMEXD5WA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.970158345 (555h4m58.84124139s) [noise=true , address=145.79.227.12:4765 , piece_id=OSCF4NIIBQOL4UFJTALLXNCWCCNJXA2E5C357BIR4AT6457TINSQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.970701309 (555h4m58.841784345s) [noise=true , address=145.79.227.6:4257 , piece_id=3JDEIUV6BCME6EUKHKT2LLRMPQCGRRB6MBSMJ3KVFD73LH737WMA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.971126038 (555h4m58.842209073s) [noise=true , address=145.79.227.13:4804 , piece_id=3ZKXUQNJSUOXL6MPEXDFPP4DOKM7JTLBWQH4MY6PUPPK6S5L34VQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.971570705 (555h4m58.84265374s) [noise=true , address=145.79.227.19:4078 , piece_id=VQRDD6ESLBAUNSOY3F4C36NEC5FDS4HWDQQ3HHE6QFYCN3DVTG5Q, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.972646433 (555h4m58.843729478s) [noise=true , address=145.79.227.47:4265 , piece_id=SG55NEYJEBCK2KBFZ7U3PLQOUK7P5DO6Z5DRQ2DUWX2QYUZJUYFA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.972911328 (555h4m58.843994373s) [noise=true , address=145.79.227.45:4231 , piece_id=J2DCYUTGYDQJ7FGVNGEWZQA2JNJ7IBV2NDGPFPUVBNYLM4QHMD5Q, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.973974452 (555h4m58.845057487s) [noise=true , address=77.90.31.18:4808 , piece_id=NHM3KRMS6EGNCCE26C6FBZEW3TDMP4FXN5WE47PY6ODRE27LRLKQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.975212530 (555h4m58.846295565s) [noise=true , address=145.79.227.21:4475 , piece_id=GGBKTA4ICTXO3BWP5MYKWKOWMSCMKRVDWGV2TNWWBNOYCARKF35Q, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.978087724 (555h4m58.849170769s) [noise=true , address=145.79.227.7:4294 , piece_id=AOGDE3GEOSHGLZBF764HP6TFYEHRSGG4L2BFOQFUJE7UWOOAVRJA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.981692148 (555h4m58.852775193s) [noise=true , address=45.91.116.4:4939 , piece_id=RKDLUQITGFAHQV4AE2TCPTNCNAYARVZ25MF7JZLZZE4LPUQMVN3A, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.983781126 (555h4m58.854864161s) [noise=true , address=145.79.227.28:4727 , piece_id=RDEWPTFGENDP5VNTWQ36JS3VDB62UIZRO5T3EZYMKQ3PRXLNPIRQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.987752909 (555h4m58.858835954s) [noise=true , address=145.79.227.15:4875 , piece_id=KSZC4UZW5KMDXRSSRRKVWATYXERGXCJAK6MLPNXCW2F5FBLERONA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:34:00.690994828 (555h5m0.562077863s) [upload=486649]: Next returned nil segment Dec 28 18:34:00.691008914 (555h5m0.562091949s) [upload=486649]: Waiting for error group managing segments... Dec 28 18:34:00.977746422 (555h5m0.848829457s) [noise=true , address=145.79.227.12:4765 , piece_id=OSCF4NIIBQOL4UFJTALLXNCWCCNJXA2E5C357BIR4AT6457TINSQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.978844533 (555h5m0.849927568s) [noise=true , address=145.79.227.52:4446 , piece_id=AOEDOIZO2ZVTYHDAHFACYKORLRIGGNF36M5BMUBTML7GMGCYLB5Q, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.986731873 (555h5m0.857814918s) [noise=true , address=145.79.227.21:4475 , piece_id=GGBKTA4ICTXO3BWP5MYKWKOWMSCMKRVDWGV2TNWWBNOYCARKF35Q, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.987946485 (555h5m0.85902953s) [noise=true , address=145.79.227.15:4875 , piece_id=KSZC4UZW5KMDXRSSRRKVWATYXERGXCJAK6MLPNXCW2F5FBLERONA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.992659470 (555h5m0.863742505s) [noise=true , address=145.79.227.6:4257 , piece_id=3JDEIUV6BCME6EUKHKT2LLRMPQCGRRB6MBSMJ3KVFD73LH737WMA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.999582272 (555h5m0.870665317s) [noise=true , address=145.79.227.19:4078 , piece_id=VQRDD6ESLBAUNSOY3F4C36NEC5FDS4HWDQQ3HHE6QFYCN3DVTG5Q, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.005606773 (555h5m0.876689829s) [noise=true , address=145.79.227.45:4231 , piece_id=J2DCYUTGYDQJ7FGVNGEWZQA2JNJ7IBV2NDGPFPUVBNYLM4QHMD5Q, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.006067911 (555h5m0.877150956s) [noise=true , address=145.79.227.47:4265 , piece_id=SG55NEYJEBCK2KBFZ7U3PLQOUK7P5DO6Z5DRQ2DUWX2QYUZJUYFA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.011528650 (555h5m0.882611695s) [noise=true , address=145.79.227.28:4727 , piece_id=RDEWPTFGENDP5VNTWQ36JS3VDB62UIZRO5T3EZYMKQ3PRXLNPIRQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.049370617 (555h5m0.920453662s) [noise=true , address=145.79.227.13:4804 , piece_id=3ZKXUQNJSUOXL6MPEXDFPP4DOKM7JTLBWQH4MY6PUPPK6S5L34VQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.140450690 (555h5m1.011533755s) [noise=true , address=77.90.31.13:4629 , piece_id=MELX67YAXCOOIXHLPA3HWG6EA2GRKUOM5QXVEIAJ7BS5PMEXD5WA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.265885880 (555h5m1.136968935s) [noise=true , address=77.90.31.19:4842 , piece_id=FYEIMUXCYS6QIY3BRDU2IZAGXTDNOXO2ENVSKZ2YRL46PUKXSUIA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.266473128 (555h5m1.137556173s) [noise=true , address=77.90.31.12:4590 , piece_id=MSFP5QZQGVNJSZL7EHTRABC2ROWUQ4L22UGGHF6LCNG33WKABWQQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.266497515 (555h5m1.13758056s) [seg_pos={0 0}, upload=486649]: Segment reached optimal threshold of 13 pieces. Dec 28 18:34:01.267022615 (555h5m1.13810566s) [noise=true , address=77.90.31.18:4808 , piece_id=NHM3KRMS6EGNCCE26C6FBZEW3TDMP4FXN5WE47PY6ODRE27LRLKQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:38010->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:38010->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38010->77.90.31.18:4808: use of closed network connection Dec 28 18:34:01.267032714 (555h5m1.138115759s) [noise=true , address=77.90.31.18:4808 , piece_id=NHM3KRMS6EGNCCE26C6FBZEW3TDMP4FXN5WE47PY6ODRE27LRLKQ, seg_pos={0 0}, upload=486649]: Long tail cancellation due to optimal threshold Dec 28 18:34:01.267046881 (555h5m1.138129926s) [noise=true , address=45.91.116.4:4939 , piece_id=RKDLUQITGFAHQV4AE2TCPTNCNAYARVZ25MF7JZLZZE4LPUQMVN3A, seg_pos={0 0}, upload=486649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 18:34:01.267059134 (555h5m1.138142179s) [noise=true , address=45.91.116.4:4939 , piece_id=RKDLUQITGFAHQV4AE2TCPTNCNAYARVZ25MF7JZLZZE4LPUQMVN3A, seg_pos={0 0}, upload=486649]: Long tail cancellation due to optimal threshold Dec 28 18:34:01.267229859 (555h5m1.138312904s) [noise=true , address=145.79.227.7:4294 , piece_id=AOGDE3GEOSHGLZBF764HP6TFYEHRSGG4L2BFOQFUJE7UWOOAVRJA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:49112->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:49112->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->145.79.227.7:4294: use of closed network connection Dec 28 18:34:01.267238606 (555h5m1.138321661s) [noise=true , address=145.79.227.7:4294 , piece_id=AOGDE3GEOSHGLZBF764HP6TFYEHRSGG4L2BFOQFUJE7UWOOAVRJA, seg_pos={0 0}, upload=486649]: Long tail cancellation due to optimal threshold Dec 28 18:34:01.267260468 (555h5m1.138343513s) [seg_pos={0 0}, upload=486649]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:34:01.267375306 (555h5m1.138458351s) [upload=486649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:34:01.267383311 (555h5m1.138466356s) [upload=486649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:34:01.563507573 (555h5m1.434590618s) [upload=486649]: Done uploading segments... Dec 28 18:34:01.563533192 (555h5m1.434616237s) [upload=486649]: Upload finished. err: <nil> Duration: 5.933514112s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.............X.X........................X..........] duration: 33.316015365s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:35:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 33.316030164s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 3.492808103s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:35:26 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.492894268s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [X........X.............X.X........................X.........] duration: 46.022419347s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:35:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 46.022438052s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XX.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X.] duration: 5.866419794s trace: https://jaeger.dev.storj.io/trace/76f49c76a8e1da20 FAIL (2025-12-28 18:35:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: MELX67YAXCOOIXHLPA3HWG6EA2GRKUOM5QXVEIAJ7BS5PMEXD5WA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: FYEIMUXCYS6QIY3BRDU2IZAGXTDNOXO2ENVSKZ2YRL46PUKXSUIA) context deadline exceeded ======LOGS====== c 28 18:33:58.722754987 (555h4m58.593838022s) [upload=486649]: Uploading segments... Dec 28 18:33:58.722808368 (555h4m58.593891403s) [upload=486649]: Got next segment. Inline: false Dec 28 18:33:58.722877549 (555h4m58.593960584s) [upload=486649]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:33:58.722882610 (555h4m58.593965645s) [upload=486649]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:33:58.968658119 (555h4m58.839741164s) [seg_pos={0 0}, upload=486649]: Begin upload segment... Dec 28 18:33:58.968718123 (555h4m58.839801158s) [seg_pos={0 0}, upload=486649]: Done begin upload segment. Dec 28 18:33:58.968748762 (555h4m58.839831797s) [noise=true , address=77.90.31.19:4842 , piece_id=FYEIMUXCYS6QIY3BRDU2IZAGXTDNOXO2ENVSKZ2YRL46PUKXSUIA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.968781634 (555h4m58.839864679s) [noise=true , address=77.90.31.12:4590 , piece_id=MSFP5QZQGVNJSZL7EHTRABC2ROWUQ4L22UGGHF6LCNG33WKABWQQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.969214989 (555h4m58.840298034s) [noise=true , address=145.79.227.52:4446 , piece_id=AOEDOIZO2ZVTYHDAHFACYKORLRIGGNF36M5BMUBTML7GMGCYLB5Q, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.969636361 (555h4m58.840719396s) [noise=true , address=77.90.31.13:4629 , piece_id=MELX67YAXCOOIXHLPA3HWG6EA2GRKUOM5QXVEIAJ7BS5PMEXD5WA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.970158345 (555h4m58.84124139s) [noise=true , address=145.79.227.12:4765 , piece_id=OSCF4NIIBQOL4UFJTALLXNCWCCNJXA2E5C357BIR4AT6457TINSQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.970701309 (555h4m58.841784345s) [noise=true , address=145.79.227.6:4257 , piece_id=3JDEIUV6BCME6EUKHKT2LLRMPQCGRRB6MBSMJ3KVFD73LH737WMA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.971126038 (555h4m58.842209073s) [noise=true , address=145.79.227.13:4804 , piece_id=3ZKXUQNJSUOXL6MPEXDFPP4DOKM7JTLBWQH4MY6PUPPK6S5L34VQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.971570705 (555h4m58.84265374s) [noise=true , address=145.79.227.19:4078 , piece_id=VQRDD6ESLBAUNSOY3F4C36NEC5FDS4HWDQQ3HHE6QFYCN3DVTG5Q, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.972646433 (555h4m58.843729478s) [noise=true , address=145.79.227.47:4265 , piece_id=SG55NEYJEBCK2KBFZ7U3PLQOUK7P5DO6Z5DRQ2DUWX2QYUZJUYFA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.972911328 (555h4m58.843994373s) [noise=true , address=145.79.227.45:4231 , piece_id=J2DCYUTGYDQJ7FGVNGEWZQA2JNJ7IBV2NDGPFPUVBNYLM4QHMD5Q, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.973974452 (555h4m58.845057487s) [noise=true , address=77.90.31.18:4808 , piece_id=NHM3KRMS6EGNCCE26C6FBZEW3TDMP4FXN5WE47PY6ODRE27LRLKQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.975212530 (555h4m58.846295565s) [noise=true , address=145.79.227.21:4475 , piece_id=GGBKTA4ICTXO3BWP5MYKWKOWMSCMKRVDWGV2TNWWBNOYCARKF35Q, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.978087724 (555h4m58.849170769s) [noise=true , address=145.79.227.7:4294 , piece_id=AOGDE3GEOSHGLZBF764HP6TFYEHRSGG4L2BFOQFUJE7UWOOAVRJA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.981692148 (555h4m58.852775193s) [noise=true , address=45.91.116.4:4939 , piece_id=RKDLUQITGFAHQV4AE2TCPTNCNAYARVZ25MF7JZLZZE4LPUQMVN3A, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.983781126 (555h4m58.854864161s) [noise=true , address=145.79.227.28:4727 , piece_id=RDEWPTFGENDP5VNTWQ36JS3VDB62UIZRO5T3EZYMKQ3PRXLNPIRQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.987752909 (555h4m58.858835954s) [noise=true , address=145.79.227.15:4875 , piece_id=KSZC4UZW5KMDXRSSRRKVWATYXERGXCJAK6MLPNXCW2F5FBLERONA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:34:00.690994828 (555h5m0.562077863s) [upload=486649]: Next returned nil segment Dec 28 18:34:00.691008914 (555h5m0.562091949s) [upload=486649]: Waiting for error group managing segments... Dec 28 18:34:00.977746422 (555h5m0.848829457s) [noise=true , address=145.79.227.12:4765 , piece_id=OSCF4NIIBQOL4UFJTALLXNCWCCNJXA2E5C357BIR4AT6457TINSQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.978844533 (555h5m0.849927568s) [noise=true , address=145.79.227.52:4446 , piece_id=AOEDOIZO2ZVTYHDAHFACYKORLRIGGNF36M5BMUBTML7GMGCYLB5Q, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.986731873 (555h5m0.857814918s) [noise=true , address=145.79.227.21:4475 , piece_id=GGBKTA4ICTXO3BWP5MYKWKOWMSCMKRVDWGV2TNWWBNOYCARKF35Q, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.987946485 (555h5m0.85902953s) [noise=true , address=145.79.227.15:4875 , piece_id=KSZC4UZW5KMDXRSSRRKVWATYXERGXCJAK6MLPNXCW2F5FBLERONA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.992659470 (555h5m0.863742505s) [noise=true , address=145.79.227.6:4257 , piece_id=3JDEIUV6BCME6EUKHKT2LLRMPQCGRRB6MBSMJ3KVFD73LH737WMA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.999582272 (555h5m0.870665317s) [noise=true , address=145.79.227.19:4078 , piece_id=VQRDD6ESLBAUNSOY3F4C36NEC5FDS4HWDQQ3HHE6QFYCN3DVTG5Q, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.005606773 (555h5m0.876689829s) [noise=true , address=145.79.227.45:4231 , piece_id=J2DCYUTGYDQJ7FGVNGEWZQA2JNJ7IBV2NDGPFPUVBNYLM4QHMD5Q, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.006067911 (555h5m0.877150956s) [noise=true , address=145.79.227.47:4265 , piece_id=SG55NEYJEBCK2KBFZ7U3PLQOUK7P5DO6Z5DRQ2DUWX2QYUZJUYFA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.011528650 (555h5m0.882611695s) [noise=true , address=145.79.227.28:4727 , piece_id=RDEWPTFGENDP5VNTWQ36JS3VDB62UIZRO5T3EZYMKQ3PRXLNPIRQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.049370617 (555h5m0.920453662s) [noise=true , address=145.79.227.13:4804 , piece_id=3ZKXUQNJSUOXL6MPEXDFPP4DOKM7JTLBWQH4MY6PUPPK6S5L34VQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.140450690 (555h5m1.011533755s) [noise=true , address=77.90.31.13:4629 , piece_id=MELX67YAXCOOIXHLPA3HWG6EA2GRKUOM5QXVEIAJ7BS5PMEXD5WA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.265885880 (555h5m1.136968935s) [noise=true , address=77.90.31.19:4842 , piece_id=FYEIMUXCYS6QIY3BRDU2IZAGXTDNOXO2ENVSKZ2YRL46PUKXSUIA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.266473128 (555h5m1.137556173s) [noise=true , address=77.90.31.12:4590 , piece_id=MSFP5QZQGVNJSZL7EHTRABC2ROWUQ4L22UGGHF6LCNG33WKABWQQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.266497515 (555h5m1.13758056s) [seg_pos={0 0}, upload=486649]: Segment reached optimal threshold of 13 pieces. Dec 28 18:34:01.267022615 (555h5m1.13810566s) [noise=true , address=77.90.31.18:4808 , piece_id=NHM3KRMS6EGNCCE26C6FBZEW3TDMP4FXN5WE47PY6ODRE27LRLKQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:38010->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:38010->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38010->77.90.31.18:4808: use of closed network connection Dec 28 18:34:01.267032714 (555h5m1.138115759s) [noise=true , address=77.90.31.18:4808 , piece_id=NHM3KRMS6EGNCCE26C6FBZEW3TDMP4FXN5WE47PY6ODRE27LRLKQ, seg_pos={0 0}, upload=486649]: Long tail cancellation due to optimal threshold Dec 28 18:34:01.267046881 (555h5m1.138129926s) [noise=true , address=45.91.116.4:4939 , piece_id=RKDLUQITGFAHQV4AE2TCPTNCNAYARVZ25MF7JZLZZE4LPUQMVN3A, seg_pos={0 0}, upload=486649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 18:34:01.267059134 (555h5m1.138142179s) [noise=true , address=45.91.116.4:4939 , piece_id=RKDLUQITGFAHQV4AE2TCPTNCNAYARVZ25MF7JZLZZE4LPUQMVN3A, seg_pos={0 0}, upload=486649]: Long tail cancellation due to optimal threshold Dec 28 18:34:01.267229859 (555h5m1.138312904s) [noise=true , address=145.79.227.7:4294 , piece_id=AOGDE3GEOSHGLZBF764HP6TFYEHRSGG4L2BFOQFUJE7UWOOAVRJA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:49112->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:49112->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->145.79.227.7:4294: use of closed network connection Dec 28 18:34:01.267238606 (555h5m1.138321661s) [noise=true , address=145.79.227.7:4294 , piece_id=AOGDE3GEOSHGLZBF764HP6TFYEHRSGG4L2BFOQFUJE7UWOOAVRJA, seg_pos={0 0}, upload=486649]: Long tail cancellation due to optimal threshold Dec 28 18:34:01.267260468 (555h5m1.138343513s) [seg_pos={0 0}, upload=486649]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:34:01.267375306 (555h5m1.138458351s) [upload=486649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:34:01.267383311 (555h5m1.138466356s) [upload=486649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:34:01.563507573 (555h5m1.434590618s) [upload=486649]: Done uploading segments... Dec 28 18:34:01.563533192 (555h5m1.434616237s) [upload=486649]: Upload finished. err: <nil> Duration: 5.866457907s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 3.446781092s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:35:07 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.446814704s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 3.052317303s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:34:51 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.0523421s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 3.216231092s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:34:35 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.216263244s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..XX.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...X] duration: 30.002410104s trace: https://jaeger.dev.storj.io/trace/76f49c76a8e1da20 FAIL (2025-12-28 18:34:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: MELX67YAXCOOIXHLPA3HWG6EA2GRKUOM5QXVEIAJ7BS5PMEXD5WA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: FYEIMUXCYS6QIY3BRDU2IZAGXTDNOXO2ENVSKZ2YRL46PUKXSUIA) context deadline exceeded ======LOGS====== c 28 18:33:58.722754987 (555h4m58.593838022s) [upload=486649]: Uploading segments... Dec 28 18:33:58.722808368 (555h4m58.593891403s) [upload=486649]: Got next segment. Inline: false Dec 28 18:33:58.722877549 (555h4m58.593960584s) [upload=486649]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:33:58.722882610 (555h4m58.593965645s) [upload=486649]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:33:58.968658119 (555h4m58.839741164s) [seg_pos={0 0}, upload=486649]: Begin upload segment... Dec 28 18:33:58.968718123 (555h4m58.839801158s) [seg_pos={0 0}, upload=486649]: Done begin upload segment. Dec 28 18:33:58.968748762 (555h4m58.839831797s) [noise=true , address=77.90.31.19:4842 , piece_id=FYEIMUXCYS6QIY3BRDU2IZAGXTDNOXO2ENVSKZ2YRL46PUKXSUIA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.968781634 (555h4m58.839864679s) [noise=true , address=77.90.31.12:4590 , piece_id=MSFP5QZQGVNJSZL7EHTRABC2ROWUQ4L22UGGHF6LCNG33WKABWQQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.969214989 (555h4m58.840298034s) [noise=true , address=145.79.227.52:4446 , piece_id=AOEDOIZO2ZVTYHDAHFACYKORLRIGGNF36M5BMUBTML7GMGCYLB5Q, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.969636361 (555h4m58.840719396s) [noise=true , address=77.90.31.13:4629 , piece_id=MELX67YAXCOOIXHLPA3HWG6EA2GRKUOM5QXVEIAJ7BS5PMEXD5WA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.970158345 (555h4m58.84124139s) [noise=true , address=145.79.227.12:4765 , piece_id=OSCF4NIIBQOL4UFJTALLXNCWCCNJXA2E5C357BIR4AT6457TINSQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.970701309 (555h4m58.841784345s) [noise=true , address=145.79.227.6:4257 , piece_id=3JDEIUV6BCME6EUKHKT2LLRMPQCGRRB6MBSMJ3KVFD73LH737WMA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.971126038 (555h4m58.842209073s) [noise=true , address=145.79.227.13:4804 , piece_id=3ZKXUQNJSUOXL6MPEXDFPP4DOKM7JTLBWQH4MY6PUPPK6S5L34VQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.971570705 (555h4m58.84265374s) [noise=true , address=145.79.227.19:4078 , piece_id=VQRDD6ESLBAUNSOY3F4C36NEC5FDS4HWDQQ3HHE6QFYCN3DVTG5Q, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.972646433 (555h4m58.843729478s) [noise=true , address=145.79.227.47:4265 , piece_id=SG55NEYJEBCK2KBFZ7U3PLQOUK7P5DO6Z5DRQ2DUWX2QYUZJUYFA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.972911328 (555h4m58.843994373s) [noise=true , address=145.79.227.45:4231 , piece_id=J2DCYUTGYDQJ7FGVNGEWZQA2JNJ7IBV2NDGPFPUVBNYLM4QHMD5Q, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.973974452 (555h4m58.845057487s) [noise=true , address=77.90.31.18:4808 , piece_id=NHM3KRMS6EGNCCE26C6FBZEW3TDMP4FXN5WE47PY6ODRE27LRLKQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.975212530 (555h4m58.846295565s) [noise=true , address=145.79.227.21:4475 , piece_id=GGBKTA4ICTXO3BWP5MYKWKOWMSCMKRVDWGV2TNWWBNOYCARKF35Q, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.978087724 (555h4m58.849170769s) [noise=true , address=145.79.227.7:4294 , piece_id=AOGDE3GEOSHGLZBF764HP6TFYEHRSGG4L2BFOQFUJE7UWOOAVRJA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.981692148 (555h4m58.852775193s) [noise=true , address=45.91.116.4:4939 , piece_id=RKDLUQITGFAHQV4AE2TCPTNCNAYARVZ25MF7JZLZZE4LPUQMVN3A, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.983781126 (555h4m58.854864161s) [noise=true , address=145.79.227.28:4727 , piece_id=RDEWPTFGENDP5VNTWQ36JS3VDB62UIZRO5T3EZYMKQ3PRXLNPIRQ, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:33:58.987752909 (555h4m58.858835954s) [noise=true , address=145.79.227.15:4875 , piece_id=KSZC4UZW5KMDXRSSRRKVWATYXERGXCJAK6MLPNXCW2F5FBLERONA, seg_pos={0 0}, upload=486649]: Uploading piece... Dec 28 18:34:00.690994828 (555h5m0.562077863s) [upload=486649]: Next returned nil segment Dec 28 18:34:00.691008914 (555h5m0.562091949s) [upload=486649]: Waiting for error group managing segments... Dec 28 18:34:00.977746422 (555h5m0.848829457s) [noise=true , address=145.79.227.12:4765 , piece_id=OSCF4NIIBQOL4UFJTALLXNCWCCNJXA2E5C357BIR4AT6457TINSQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.978844533 (555h5m0.849927568s) [noise=true , address=145.79.227.52:4446 , piece_id=AOEDOIZO2ZVTYHDAHFACYKORLRIGGNF36M5BMUBTML7GMGCYLB5Q, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.986731873 (555h5m0.857814918s) [noise=true , address=145.79.227.21:4475 , piece_id=GGBKTA4ICTXO3BWP5MYKWKOWMSCMKRVDWGV2TNWWBNOYCARKF35Q, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.987946485 (555h5m0.85902953s) [noise=true , address=145.79.227.15:4875 , piece_id=KSZC4UZW5KMDXRSSRRKVWATYXERGXCJAK6MLPNXCW2F5FBLERONA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.992659470 (555h5m0.863742505s) [noise=true , address=145.79.227.6:4257 , piece_id=3JDEIUV6BCME6EUKHKT2LLRMPQCGRRB6MBSMJ3KVFD73LH737WMA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:00.999582272 (555h5m0.870665317s) [noise=true , address=145.79.227.19:4078 , piece_id=VQRDD6ESLBAUNSOY3F4C36NEC5FDS4HWDQQ3HHE6QFYCN3DVTG5Q, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.005606773 (555h5m0.876689829s) [noise=true , address=145.79.227.45:4231 , piece_id=J2DCYUTGYDQJ7FGVNGEWZQA2JNJ7IBV2NDGPFPUVBNYLM4QHMD5Q, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.006067911 (555h5m0.877150956s) [noise=true , address=145.79.227.47:4265 , piece_id=SG55NEYJEBCK2KBFZ7U3PLQOUK7P5DO6Z5DRQ2DUWX2QYUZJUYFA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.011528650 (555h5m0.882611695s) [noise=true , address=145.79.227.28:4727 , piece_id=RDEWPTFGENDP5VNTWQ36JS3VDB62UIZRO5T3EZYMKQ3PRXLNPIRQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.049370617 (555h5m0.920453662s) [noise=true , address=145.79.227.13:4804 , piece_id=3ZKXUQNJSUOXL6MPEXDFPP4DOKM7JTLBWQH4MY6PUPPK6S5L34VQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.140450690 (555h5m1.011533755s) [noise=true , address=77.90.31.13:4629 , piece_id=MELX67YAXCOOIXHLPA3HWG6EA2GRKUOM5QXVEIAJ7BS5PMEXD5WA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.265885880 (555h5m1.136968935s) [noise=true , address=77.90.31.19:4842 , piece_id=FYEIMUXCYS6QIY3BRDU2IZAGXTDNOXO2ENVSKZ2YRL46PUKXSUIA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.266473128 (555h5m1.137556173s) [noise=true , address=77.90.31.12:4590 , piece_id=MSFP5QZQGVNJSZL7EHTRABC2ROWUQ4L22UGGHF6LCNG33WKABWQQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: <nil> Dec 28 18:34:01.266497515 (555h5m1.13758056s) [seg_pos={0 0}, upload=486649]: Segment reached optimal threshold of 13 pieces. Dec 28 18:34:01.267022615 (555h5m1.13810566s) [noise=true , address=77.90.31.18:4808 , piece_id=NHM3KRMS6EGNCCE26C6FBZEW3TDMP4FXN5WE47PY6ODRE27LRLKQ, seg_pos={0 0}, upload=486649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:38010->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:38010->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38010->77.90.31.18:4808: use of closed network connection Dec 28 18:34:01.267032714 (555h5m1.138115759s) [noise=true , address=77.90.31.18:4808 , piece_id=NHM3KRMS6EGNCCE26C6FBZEW3TDMP4FXN5WE47PY6ODRE27LRLKQ, seg_pos={0 0}, upload=486649]: Long tail cancellation due to optimal threshold Dec 28 18:34:01.267046881 (555h5m1.138129926s) [noise=true , address=45.91.116.4:4939 , piece_id=RKDLUQITGFAHQV4AE2TCPTNCNAYARVZ25MF7JZLZZE4LPUQMVN3A, seg_pos={0 0}, upload=486649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 18:34:01.267059134 (555h5m1.138142179s) [noise=true , address=45.91.116.4:4939 , piece_id=RKDLUQITGFAHQV4AE2TCPTNCNAYARVZ25MF7JZLZZE4LPUQMVN3A, seg_pos={0 0}, upload=486649]: Long tail cancellation due to optimal threshold Dec 28 18:34:01.267229859 (555h5m1.138312904s) [noise=true , address=145.79.227.7:4294 , piece_id=AOGDE3GEOSHGLZBF764HP6TFYEHRSGG4L2BFOQFUJE7UWOOAVRJA, seg_pos={0 0}, upload=486649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:49112->145.79.227.7:4294: use of closed network connection; write tcp 188.166.225.170:49112->145.79.227.7:4294: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->145.79.227.7:4294: use of closed network connection Dec 28 18:34:01.267238606 (555h5m1.138321661s) [noise=true , address=145.79.227.7:4294 , piece_id=AOGDE3GEOSHGLZBF764HP6TFYEHRSGG4L2BFOQFUJE7UWOOAVRJA, seg_pos={0 0}, upload=486649]: Long tail cancellation due to optimal threshold Dec 28 18:34:01.267260468 (555h5m1.138343513s) [seg_pos={0 0}, upload=486649]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:34:01.267375306 (555h5m1.138458351s) [upload=486649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:34:01.267383311 (555h5m1.138466356s) [upload=486649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:34:01.563507573 (555h5m1.434590618s) [upload=486649]: Done uploading segments... Dec 28 18:34:01.563533192 (555h5m1.434616237s) [upload=486649]: Upload finished. err: <nil> Duration: 30.002432016s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X........X.............X.X........................X........] duration: 53.194196951s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:34:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 53.194221017s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 3.2546237s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:34:07 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.254684314s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 3.950555435s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:33:18 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.950584179s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X........X.............X.X........................X.......] duration: 32.399738674s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:33:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 32.399763751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X..XX.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X...] duration: 5.87199008s trace: https://jaeger.dev.storj.io/trace/291c428b844a0474 FAIL (2025-12-28 18:32:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: SPJHYU4QFIGCNOUXJRHC6PJ77NPAUX7PXKIRJPYRKQL2VEGOYRPQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: N336ELTV7WYFZGP6LSXGWQ4VA5FPZYZCOR6MYOIU7J3MVVIP5IRQ) context deadline exceeded ======LOGS====== 1m23.977394577s) [upload=486602]: Uploading segments... Dec 28 18:30:24.106371696 (555h1m23.977454741s) [upload=486602]: Got next segment. Inline: false Dec 28 18:30:24.106401292 (555h1m23.977484337s) [upload=486602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:30:24.106405650 (555h1m23.977488696s) [upload=486602]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:30:24.356211154 (555h1m24.227294199s) [seg_pos={0 0}, upload=486602]: Begin upload segment... Dec 28 18:30:24.356307267 (555h1m24.227390302s) [seg_pos={0 0}, upload=486602]: Done begin upload segment. Dec 28 18:30:24.356344198 (555h1m24.227427243s) [noise=true , address=145.79.227.48:4301 , piece_id=KAUC6B4SNHGYVLVNJCKAXZ2ZOX23ORZJGN45PVCZ2FJ7WWA2DYNQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.362040965 (555h1m24.23312401s) [noise=true , address=45.91.116.10:4084 , piece_id=HP6KY3HIJWTJ2BGLYUMI6MQOTJQOZOATMMTLOEIIE7AIWLI6IUIA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.363959479 (555h1m24.235042514s) [noise=true , address=145.79.227.3:4150 , piece_id=IXI6YYWC3VZDXB6GQIOF4PNAXEQXT4EQRZ3GXUREEUNPNTXHXHUQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.365049704 (555h1m24.236132749s) [noise=true , address=145.79.227.37:4566 , piece_id=7JECM4YX64MUNTNVNBL3GDEOXBAJYTWTLI3AAQE27YBSIUJSZG6Q, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.366012729 (555h1m24.237095774s) [noise=true , address=77.90.31.15:4697 , piece_id=B2HTKHQKFOOO6MRIDY33MNQDLR3QHWNIIFTXK4W44ZBFFJQHQWFA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.366082111 (555h1m24.237165146s) [noise=true , address=77.90.31.14:4663 , piece_id=N336ELTV7WYFZGP6LSXGWQ4VA5FPZYZCOR6MYOIU7J3MVVIP5IRQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.366707802 (555h1m24.237790847s) [noise=true , address=145.79.227.13:4803 , piece_id=WGD4HUTUX2OCQNIO3FNY4PR5AJEEJ7X2XM4PYTSZFKYDINNI44PA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.366805537 (555h1m24.237888582s) [noise=true , address=145.79.227.7:4295 , piece_id=764M3EY6NZQHJ2VNHYXP5QFGVCICQU6ACQO7KK7TKZFWCGLW7YZA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.367179119 (555h1m24.238262164s) [noise=true , address=145.79.227.39:4015 , piece_id=KJDFSCM4R2QCQE4AXNMWZ3S2WDYXXCGX2AMEQTILBBOXKA4MLTNQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.367317152 (555h1m24.238400187s) [noise=true , address=45.91.116.14:4192 , piece_id=SPJHYU4QFIGCNOUXJRHC6PJ77NPAUX7PXKIRJPYRKQL2VEGOYRPQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.367667419 (555h1m24.238750474s) [noise=true , address=145.79.227.38:4977 , piece_id=MPSYTE7G6SLY52KRFGM6I3ILMVITDYQHWCESTY53X6XEQJJWPDMQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.367832303 (555h1m24.238915338s) [noise=true , address=45.91.116.4:4940 , piece_id=IVFBCBSLTMG7OUQF7HEBQC4VK5PGI3GT5J42MZJP4NAOUF6E4DCA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.368185936 (555h1m24.239268981s) [noise=true , address=145.79.227.52:4447 , piece_id=5JBYMCC34PVNXD7QOAOKLJHYJBRAVGEHWJMSQCYLEOG4KJFJEDPA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.368298510 (555h1m24.239381545s) [noise=true , address=145.79.227.15:4874 , piece_id=FT54ZA3ZN7NZ4LCNVJ2SPOBAYNQ6Q7IGVGRFRE7TJJDKXCPY7Q5A, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.368769107 (555h1m24.239852142s) [noise=true , address=145.79.227.41:4087 , piece_id=LKCKLONESBRTDKDVP4DA2JV3GHAETLT3ADIH7AQDNJZQRFJ65TYA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.369292824 (555h1m24.240375869s) [noise=true , address=145.79.227.45:4231 , piece_id=LAPV3G65R5MDPAWY7FDCFTFJZY4CSEXMNJUF5PPMDG7GISMXSETQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:25.987984716 (555h1m25.859067751s) [upload=486602]: Next returned nil segment Dec 28 18:30:25.987999474 (555h1m25.859082519s) [upload=486602]: Waiting for error group managing segments... Dec 28 18:30:26.371269827 (555h1m26.242352872s) [noise=true , address=145.79.227.39:4015 , piece_id=KJDFSCM4R2QCQE4AXNMWZ3S2WDYXXCGX2AMEQTILBBOXKA4MLTNQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.375273340 (555h1m26.246356375s) [noise=true , address=145.79.227.37:4566 , piece_id=7JECM4YX64MUNTNVNBL3GDEOXBAJYTWTLI3AAQE27YBSIUJSZG6Q, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.377276505 (555h1m26.24835955s) [noise=true , address=145.79.227.7:4295 , piece_id=764M3EY6NZQHJ2VNHYXP5QFGVCICQU6ACQO7KK7TKZFWCGLW7YZA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.385263112 (555h1m26.256346157s) [noise=true , address=145.79.227.38:4977 , piece_id=MPSYTE7G6SLY52KRFGM6I3ILMVITDYQHWCESTY53X6XEQJJWPDMQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.388294895 (555h1m26.25937794s) [noise=true , address=145.79.227.52:4447 , piece_id=5JBYMCC34PVNXD7QOAOKLJHYJBRAVGEHWJMSQCYLEOG4KJFJEDPA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.392247552 (555h1m26.263330597s) [noise=true , address=145.79.227.45:4231 , piece_id=LAPV3G65R5MDPAWY7FDCFTFJZY4CSEXMNJUF5PPMDG7GISMXSETQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.399676858 (555h1m26.270759894s) [noise=true , address=145.79.227.48:4301 , piece_id=KAUC6B4SNHGYVLVNJCKAXZ2ZOX23ORZJGN45PVCZ2FJ7WWA2DYNQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.400307971 (555h1m26.271391016s) [noise=true , address=145.79.227.41:4087 , piece_id=LKCKLONESBRTDKDVP4DA2JV3GHAETLT3ADIH7AQDNJZQRFJ65TYA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.402205133 (555h1m26.273288168s) [noise=true , address=145.79.227.15:4874 , piece_id=FT54ZA3ZN7NZ4LCNVJ2SPOBAYNQ6Q7IGVGRFRE7TJJDKXCPY7Q5A, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.565360970 (555h1m26.436444015s) [noise=true , address=77.90.31.14:4663 , piece_id=N336ELTV7WYFZGP6LSXGWQ4VA5FPZYZCOR6MYOIU7J3MVVIP5IRQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.589487802 (555h1m26.460570847s) [noise=true , address=145.79.227.13:4803 , piece_id=WGD4HUTUX2OCQNIO3FNY4PR5AJEEJ7X2XM4PYTSZFKYDINNI44PA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.630597672 (555h1m26.501680717s) [noise=true , address=45.91.116.14:4192 , piece_id=SPJHYU4QFIGCNOUXJRHC6PJ77NPAUX7PXKIRJPYRKQL2VEGOYRPQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.652758160 (555h1m26.523841205s) [noise=true , address=77.90.31.15:4697 , piece_id=B2HTKHQKFOOO6MRIDY33MNQDLR3QHWNIIFTXK4W44ZBFFJQHQWFA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.652783077 (555h1m26.523866113s) [seg_pos={0 0}, upload=486602]: Segment reached optimal threshold of 13 pieces. Dec 28 18:30:26.653003737 (555h1m26.524086782s) [noise=true , address=145.79.227.3:4150 , piece_id=IXI6YYWC3VZDXB6GQIOF4PNAXEQXT4EQRZ3GXUREEUNPNTXHXHUQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: write tcp 188.166.225.170:53738->145.79.227.3:4150: use of closed network connection; write tcp 188.166.225.170:53738->145.79.227.3:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53738->145.79.227.3:4150: use of closed network connection Dec 28 18:30:26.653014498 (555h1m26.524097534s) [noise=true , address=145.79.227.3:4150 , piece_id=IXI6YYWC3VZDXB6GQIOF4PNAXEQXT4EQRZ3GXUREEUNPNTXHXHUQ, seg_pos={0 0}, upload=486602]: Long tail cancellation due to optimal threshold Dec 28 18:30:26.653025519 (555h1m26.524108564s) [noise=true , address=45.91.116.4:4940 , piece_id=IVFBCBSLTMG7OUQF7HEBQC4VK5PGI3GT5J42MZJP4NAOUF6E4DCA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:57704->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:57704->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57704->45.91.116.4:4940: use of closed network connection Dec 28 18:30:26.653033464 (555h1m26.524116509s) [noise=true , address=45.91.116.4:4940 , piece_id=IVFBCBSLTMG7OUQF7HEBQC4VK5PGI3GT5J42MZJP4NAOUF6E4DCA, seg_pos={0 0}, upload=486602]: Long tail cancellation due to optimal threshold Dec 28 18:30:26.653216292 (555h1m26.524299337s) [noise=true , address=45.91.116.10:4084 , piece_id=HP6KY3HIJWTJ2BGLYUMI6MQOTJQOZOATMMTLOEIIE7AIWLI6IUIA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 28 18:30:26.653225009 (555h1m26.524308074s) [noise=true , address=45.91.116.10:4084 , piece_id=HP6KY3HIJWTJ2BGLYUMI6MQOTJQOZOATMMTLOEIIE7AIWLI6IUIA, seg_pos={0 0}, upload=486602]: Long tail cancellation due to optimal threshold Dec 28 18:30:26.653244837 (555h1m26.524327872s) [seg_pos={0 0}, upload=486602]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:30:26.653328276 (555h1m26.524411321s) [upload=486602]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:30:26.653334898 (555h1m26.524417943s) [upload=486602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:30:26.957763122 (555h1m26.828846167s) [upload=486602]: Done uploading segments... Dec 28 18:30:26.957789813 (555h1m26.828872858s) [upload=486602]: Upload finished. err: <nil> Duration: 5.872019355s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 3.454598407s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:32:37 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.454626009s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X........X.............X.X........................X......] duration: 24.112178018s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:32:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 24.11220443s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X..XX.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X..] duration: 5.725855694s trace: https://jaeger.dev.storj.io/trace/291c428b844a0474 FAIL (2025-12-28 18:32:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: SPJHYU4QFIGCNOUXJRHC6PJ77NPAUX7PXKIRJPYRKQL2VEGOYRPQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: N336ELTV7WYFZGP6LSXGWQ4VA5FPZYZCOR6MYOIU7J3MVVIP5IRQ) context deadline exceeded ======LOGS====== 1m23.977394577s) [upload=486602]: Uploading segments... Dec 28 18:30:24.106371696 (555h1m23.977454741s) [upload=486602]: Got next segment. Inline: false Dec 28 18:30:24.106401292 (555h1m23.977484337s) [upload=486602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:30:24.106405650 (555h1m23.977488696s) [upload=486602]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:30:24.356211154 (555h1m24.227294199s) [seg_pos={0 0}, upload=486602]: Begin upload segment... Dec 28 18:30:24.356307267 (555h1m24.227390302s) [seg_pos={0 0}, upload=486602]: Done begin upload segment. Dec 28 18:30:24.356344198 (555h1m24.227427243s) [noise=true , address=145.79.227.48:4301 , piece_id=KAUC6B4SNHGYVLVNJCKAXZ2ZOX23ORZJGN45PVCZ2FJ7WWA2DYNQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.362040965 (555h1m24.23312401s) [noise=true , address=45.91.116.10:4084 , piece_id=HP6KY3HIJWTJ2BGLYUMI6MQOTJQOZOATMMTLOEIIE7AIWLI6IUIA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.363959479 (555h1m24.235042514s) [noise=true , address=145.79.227.3:4150 , piece_id=IXI6YYWC3VZDXB6GQIOF4PNAXEQXT4EQRZ3GXUREEUNPNTXHXHUQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.365049704 (555h1m24.236132749s) [noise=true , address=145.79.227.37:4566 , piece_id=7JECM4YX64MUNTNVNBL3GDEOXBAJYTWTLI3AAQE27YBSIUJSZG6Q, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.366012729 (555h1m24.237095774s) [noise=true , address=77.90.31.15:4697 , piece_id=B2HTKHQKFOOO6MRIDY33MNQDLR3QHWNIIFTXK4W44ZBFFJQHQWFA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.366082111 (555h1m24.237165146s) [noise=true , address=77.90.31.14:4663 , piece_id=N336ELTV7WYFZGP6LSXGWQ4VA5FPZYZCOR6MYOIU7J3MVVIP5IRQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.366707802 (555h1m24.237790847s) [noise=true , address=145.79.227.13:4803 , piece_id=WGD4HUTUX2OCQNIO3FNY4PR5AJEEJ7X2XM4PYTSZFKYDINNI44PA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.366805537 (555h1m24.237888582s) [noise=true , address=145.79.227.7:4295 , piece_id=764M3EY6NZQHJ2VNHYXP5QFGVCICQU6ACQO7KK7TKZFWCGLW7YZA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.367179119 (555h1m24.238262164s) [noise=true , address=145.79.227.39:4015 , piece_id=KJDFSCM4R2QCQE4AXNMWZ3S2WDYXXCGX2AMEQTILBBOXKA4MLTNQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.367317152 (555h1m24.238400187s) [noise=true , address=45.91.116.14:4192 , piece_id=SPJHYU4QFIGCNOUXJRHC6PJ77NPAUX7PXKIRJPYRKQL2VEGOYRPQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.367667419 (555h1m24.238750474s) [noise=true , address=145.79.227.38:4977 , piece_id=MPSYTE7G6SLY52KRFGM6I3ILMVITDYQHWCESTY53X6XEQJJWPDMQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.367832303 (555h1m24.238915338s) [noise=true , address=45.91.116.4:4940 , piece_id=IVFBCBSLTMG7OUQF7HEBQC4VK5PGI3GT5J42MZJP4NAOUF6E4DCA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.368185936 (555h1m24.239268981s) [noise=true , address=145.79.227.52:4447 , piece_id=5JBYMCC34PVNXD7QOAOKLJHYJBRAVGEHWJMSQCYLEOG4KJFJEDPA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.368298510 (555h1m24.239381545s) [noise=true , address=145.79.227.15:4874 , piece_id=FT54ZA3ZN7NZ4LCNVJ2SPOBAYNQ6Q7IGVGRFRE7TJJDKXCPY7Q5A, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.368769107 (555h1m24.239852142s) [noise=true , address=145.79.227.41:4087 , piece_id=LKCKLONESBRTDKDVP4DA2JV3GHAETLT3ADIH7AQDNJZQRFJ65TYA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.369292824 (555h1m24.240375869s) [noise=true , address=145.79.227.45:4231 , piece_id=LAPV3G65R5MDPAWY7FDCFTFJZY4CSEXMNJUF5PPMDG7GISMXSETQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:25.987984716 (555h1m25.859067751s) [upload=486602]: Next returned nil segment Dec 28 18:30:25.987999474 (555h1m25.859082519s) [upload=486602]: Waiting for error group managing segments... Dec 28 18:30:26.371269827 (555h1m26.242352872s) [noise=true , address=145.79.227.39:4015 , piece_id=KJDFSCM4R2QCQE4AXNMWZ3S2WDYXXCGX2AMEQTILBBOXKA4MLTNQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.375273340 (555h1m26.246356375s) [noise=true , address=145.79.227.37:4566 , piece_id=7JECM4YX64MUNTNVNBL3GDEOXBAJYTWTLI3AAQE27YBSIUJSZG6Q, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.377276505 (555h1m26.24835955s) [noise=true , address=145.79.227.7:4295 , piece_id=764M3EY6NZQHJ2VNHYXP5QFGVCICQU6ACQO7KK7TKZFWCGLW7YZA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.385263112 (555h1m26.256346157s) [noise=true , address=145.79.227.38:4977 , piece_id=MPSYTE7G6SLY52KRFGM6I3ILMVITDYQHWCESTY53X6XEQJJWPDMQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.388294895 (555h1m26.25937794s) [noise=true , address=145.79.227.52:4447 , piece_id=5JBYMCC34PVNXD7QOAOKLJHYJBRAVGEHWJMSQCYLEOG4KJFJEDPA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.392247552 (555h1m26.263330597s) [noise=true , address=145.79.227.45:4231 , piece_id=LAPV3G65R5MDPAWY7FDCFTFJZY4CSEXMNJUF5PPMDG7GISMXSETQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.399676858 (555h1m26.270759894s) [noise=true , address=145.79.227.48:4301 , piece_id=KAUC6B4SNHGYVLVNJCKAXZ2ZOX23ORZJGN45PVCZ2FJ7WWA2DYNQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.400307971 (555h1m26.271391016s) [noise=true , address=145.79.227.41:4087 , piece_id=LKCKLONESBRTDKDVP4DA2JV3GHAETLT3ADIH7AQDNJZQRFJ65TYA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.402205133 (555h1m26.273288168s) [noise=true , address=145.79.227.15:4874 , piece_id=FT54ZA3ZN7NZ4LCNVJ2SPOBAYNQ6Q7IGVGRFRE7TJJDKXCPY7Q5A, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.565360970 (555h1m26.436444015s) [noise=true , address=77.90.31.14:4663 , piece_id=N336ELTV7WYFZGP6LSXGWQ4VA5FPZYZCOR6MYOIU7J3MVVIP5IRQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.589487802 (555h1m26.460570847s) [noise=true , address=145.79.227.13:4803 , piece_id=WGD4HUTUX2OCQNIO3FNY4PR5AJEEJ7X2XM4PYTSZFKYDINNI44PA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.630597672 (555h1m26.501680717s) [noise=true , address=45.91.116.14:4192 , piece_id=SPJHYU4QFIGCNOUXJRHC6PJ77NPAUX7PXKIRJPYRKQL2VEGOYRPQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.652758160 (555h1m26.523841205s) [noise=true , address=77.90.31.15:4697 , piece_id=B2HTKHQKFOOO6MRIDY33MNQDLR3QHWNIIFTXK4W44ZBFFJQHQWFA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.652783077 (555h1m26.523866113s) [seg_pos={0 0}, upload=486602]: Segment reached optimal threshold of 13 pieces. Dec 28 18:30:26.653003737 (555h1m26.524086782s) [noise=true , address=145.79.227.3:4150 , piece_id=IXI6YYWC3VZDXB6GQIOF4PNAXEQXT4EQRZ3GXUREEUNPNTXHXHUQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: write tcp 188.166.225.170:53738->145.79.227.3:4150: use of closed network connection; write tcp 188.166.225.170:53738->145.79.227.3:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53738->145.79.227.3:4150: use of closed network connection Dec 28 18:30:26.653014498 (555h1m26.524097534s) [noise=true , address=145.79.227.3:4150 , piece_id=IXI6YYWC3VZDXB6GQIOF4PNAXEQXT4EQRZ3GXUREEUNPNTXHXHUQ, seg_pos={0 0}, upload=486602]: Long tail cancellation due to optimal threshold Dec 28 18:30:26.653025519 (555h1m26.524108564s) [noise=true , address=45.91.116.4:4940 , piece_id=IVFBCBSLTMG7OUQF7HEBQC4VK5PGI3GT5J42MZJP4NAOUF6E4DCA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:57704->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:57704->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57704->45.91.116.4:4940: use of closed network connection Dec 28 18:30:26.653033464 (555h1m26.524116509s) [noise=true , address=45.91.116.4:4940 , piece_id=IVFBCBSLTMG7OUQF7HEBQC4VK5PGI3GT5J42MZJP4NAOUF6E4DCA, seg_pos={0 0}, upload=486602]: Long tail cancellation due to optimal threshold Dec 28 18:30:26.653216292 (555h1m26.524299337s) [noise=true , address=45.91.116.10:4084 , piece_id=HP6KY3HIJWTJ2BGLYUMI6MQOTJQOZOATMMTLOEIIE7AIWLI6IUIA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 28 18:30:26.653225009 (555h1m26.524308074s) [noise=true , address=45.91.116.10:4084 , piece_id=HP6KY3HIJWTJ2BGLYUMI6MQOTJQOZOATMMTLOEIIE7AIWLI6IUIA, seg_pos={0 0}, upload=486602]: Long tail cancellation due to optimal threshold Dec 28 18:30:26.653244837 (555h1m26.524327872s) [seg_pos={0 0}, upload=486602]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:30:26.653328276 (555h1m26.524411321s) [upload=486602]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:30:26.653334898 (555h1m26.524417943s) [upload=486602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:30:26.957763122 (555h1m26.828846167s) [upload=486602]: Done uploading segments... Dec 28 18:30:26.957789813 (555h1m26.828872858s) [upload=486602]: Upload finished. err: <nil> Duration: 5.725887515s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 3.281314714s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:31:44 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.28134381s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X........X.............X.X........................X.....] duration: 22.416206112s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:31:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 22.416235046s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X..XX.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X.] duration: 6.021520221s trace: https://jaeger.dev.storj.io/trace/291c428b844a0474 FAIL (2025-12-28 18:31:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: SPJHYU4QFIGCNOUXJRHC6PJ77NPAUX7PXKIRJPYRKQL2VEGOYRPQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: N336ELTV7WYFZGP6LSXGWQ4VA5FPZYZCOR6MYOIU7J3MVVIP5IRQ) context deadline exceeded ======LOGS====== 1m23.977394577s) [upload=486602]: Uploading segments... Dec 28 18:30:24.106371696 (555h1m23.977454741s) [upload=486602]: Got next segment. Inline: false Dec 28 18:30:24.106401292 (555h1m23.977484337s) [upload=486602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:30:24.106405650 (555h1m23.977488696s) [upload=486602]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:30:24.356211154 (555h1m24.227294199s) [seg_pos={0 0}, upload=486602]: Begin upload segment... Dec 28 18:30:24.356307267 (555h1m24.227390302s) [seg_pos={0 0}, upload=486602]: Done begin upload segment. Dec 28 18:30:24.356344198 (555h1m24.227427243s) [noise=true , address=145.79.227.48:4301 , piece_id=KAUC6B4SNHGYVLVNJCKAXZ2ZOX23ORZJGN45PVCZ2FJ7WWA2DYNQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.362040965 (555h1m24.23312401s) [noise=true , address=45.91.116.10:4084 , piece_id=HP6KY3HIJWTJ2BGLYUMI6MQOTJQOZOATMMTLOEIIE7AIWLI6IUIA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.363959479 (555h1m24.235042514s) [noise=true , address=145.79.227.3:4150 , piece_id=IXI6YYWC3VZDXB6GQIOF4PNAXEQXT4EQRZ3GXUREEUNPNTXHXHUQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.365049704 (555h1m24.236132749s) [noise=true , address=145.79.227.37:4566 , piece_id=7JECM4YX64MUNTNVNBL3GDEOXBAJYTWTLI3AAQE27YBSIUJSZG6Q, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.366012729 (555h1m24.237095774s) [noise=true , address=77.90.31.15:4697 , piece_id=B2HTKHQKFOOO6MRIDY33MNQDLR3QHWNIIFTXK4W44ZBFFJQHQWFA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.366082111 (555h1m24.237165146s) [noise=true , address=77.90.31.14:4663 , piece_id=N336ELTV7WYFZGP6LSXGWQ4VA5FPZYZCOR6MYOIU7J3MVVIP5IRQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.366707802 (555h1m24.237790847s) [noise=true , address=145.79.227.13:4803 , piece_id=WGD4HUTUX2OCQNIO3FNY4PR5AJEEJ7X2XM4PYTSZFKYDINNI44PA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.366805537 (555h1m24.237888582s) [noise=true , address=145.79.227.7:4295 , piece_id=764M3EY6NZQHJ2VNHYXP5QFGVCICQU6ACQO7KK7TKZFWCGLW7YZA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.367179119 (555h1m24.238262164s) [noise=true , address=145.79.227.39:4015 , piece_id=KJDFSCM4R2QCQE4AXNMWZ3S2WDYXXCGX2AMEQTILBBOXKA4MLTNQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.367317152 (555h1m24.238400187s) [noise=true , address=45.91.116.14:4192 , piece_id=SPJHYU4QFIGCNOUXJRHC6PJ77NPAUX7PXKIRJPYRKQL2VEGOYRPQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.367667419 (555h1m24.238750474s) [noise=true , address=145.79.227.38:4977 , piece_id=MPSYTE7G6SLY52KRFGM6I3ILMVITDYQHWCESTY53X6XEQJJWPDMQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.367832303 (555h1m24.238915338s) [noise=true , address=45.91.116.4:4940 , piece_id=IVFBCBSLTMG7OUQF7HEBQC4VK5PGI3GT5J42MZJP4NAOUF6E4DCA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.368185936 (555h1m24.239268981s) [noise=true , address=145.79.227.52:4447 , piece_id=5JBYMCC34PVNXD7QOAOKLJHYJBRAVGEHWJMSQCYLEOG4KJFJEDPA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.368298510 (555h1m24.239381545s) [noise=true , address=145.79.227.15:4874 , piece_id=FT54ZA3ZN7NZ4LCNVJ2SPOBAYNQ6Q7IGVGRFRE7TJJDKXCPY7Q5A, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.368769107 (555h1m24.239852142s) [noise=true , address=145.79.227.41:4087 , piece_id=LKCKLONESBRTDKDVP4DA2JV3GHAETLT3ADIH7AQDNJZQRFJ65TYA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.369292824 (555h1m24.240375869s) [noise=true , address=145.79.227.45:4231 , piece_id=LAPV3G65R5MDPAWY7FDCFTFJZY4CSEXMNJUF5PPMDG7GISMXSETQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:25.987984716 (555h1m25.859067751s) [upload=486602]: Next returned nil segment Dec 28 18:30:25.987999474 (555h1m25.859082519s) [upload=486602]: Waiting for error group managing segments... Dec 28 18:30:26.371269827 (555h1m26.242352872s) [noise=true , address=145.79.227.39:4015 , piece_id=KJDFSCM4R2QCQE4AXNMWZ3S2WDYXXCGX2AMEQTILBBOXKA4MLTNQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.375273340 (555h1m26.246356375s) [noise=true , address=145.79.227.37:4566 , piece_id=7JECM4YX64MUNTNVNBL3GDEOXBAJYTWTLI3AAQE27YBSIUJSZG6Q, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.377276505 (555h1m26.24835955s) [noise=true , address=145.79.227.7:4295 , piece_id=764M3EY6NZQHJ2VNHYXP5QFGVCICQU6ACQO7KK7TKZFWCGLW7YZA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.385263112 (555h1m26.256346157s) [noise=true , address=145.79.227.38:4977 , piece_id=MPSYTE7G6SLY52KRFGM6I3ILMVITDYQHWCESTY53X6XEQJJWPDMQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.388294895 (555h1m26.25937794s) [noise=true , address=145.79.227.52:4447 , piece_id=5JBYMCC34PVNXD7QOAOKLJHYJBRAVGEHWJMSQCYLEOG4KJFJEDPA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.392247552 (555h1m26.263330597s) [noise=true , address=145.79.227.45:4231 , piece_id=LAPV3G65R5MDPAWY7FDCFTFJZY4CSEXMNJUF5PPMDG7GISMXSETQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.399676858 (555h1m26.270759894s) [noise=true , address=145.79.227.48:4301 , piece_id=KAUC6B4SNHGYVLVNJCKAXZ2ZOX23ORZJGN45PVCZ2FJ7WWA2DYNQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.400307971 (555h1m26.271391016s) [noise=true , address=145.79.227.41:4087 , piece_id=LKCKLONESBRTDKDVP4DA2JV3GHAETLT3ADIH7AQDNJZQRFJ65TYA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.402205133 (555h1m26.273288168s) [noise=true , address=145.79.227.15:4874 , piece_id=FT54ZA3ZN7NZ4LCNVJ2SPOBAYNQ6Q7IGVGRFRE7TJJDKXCPY7Q5A, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.565360970 (555h1m26.436444015s) [noise=true , address=77.90.31.14:4663 , piece_id=N336ELTV7WYFZGP6LSXGWQ4VA5FPZYZCOR6MYOIU7J3MVVIP5IRQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.589487802 (555h1m26.460570847s) [noise=true , address=145.79.227.13:4803 , piece_id=WGD4HUTUX2OCQNIO3FNY4PR5AJEEJ7X2XM4PYTSZFKYDINNI44PA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.630597672 (555h1m26.501680717s) [noise=true , address=45.91.116.14:4192 , piece_id=SPJHYU4QFIGCNOUXJRHC6PJ77NPAUX7PXKIRJPYRKQL2VEGOYRPQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.652758160 (555h1m26.523841205s) [noise=true , address=77.90.31.15:4697 , piece_id=B2HTKHQKFOOO6MRIDY33MNQDLR3QHWNIIFTXK4W44ZBFFJQHQWFA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.652783077 (555h1m26.523866113s) [seg_pos={0 0}, upload=486602]: Segment reached optimal threshold of 13 pieces. Dec 28 18:30:26.653003737 (555h1m26.524086782s) [noise=true , address=145.79.227.3:4150 , piece_id=IXI6YYWC3VZDXB6GQIOF4PNAXEQXT4EQRZ3GXUREEUNPNTXHXHUQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: write tcp 188.166.225.170:53738->145.79.227.3:4150: use of closed network connection; write tcp 188.166.225.170:53738->145.79.227.3:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53738->145.79.227.3:4150: use of closed network connection Dec 28 18:30:26.653014498 (555h1m26.524097534s) [noise=true , address=145.79.227.3:4150 , piece_id=IXI6YYWC3VZDXB6GQIOF4PNAXEQXT4EQRZ3GXUREEUNPNTXHXHUQ, seg_pos={0 0}, upload=486602]: Long tail cancellation due to optimal threshold Dec 28 18:30:26.653025519 (555h1m26.524108564s) [noise=true , address=45.91.116.4:4940 , piece_id=IVFBCBSLTMG7OUQF7HEBQC4VK5PGI3GT5J42MZJP4NAOUF6E4DCA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:57704->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:57704->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57704->45.91.116.4:4940: use of closed network connection Dec 28 18:30:26.653033464 (555h1m26.524116509s) [noise=true , address=45.91.116.4:4940 , piece_id=IVFBCBSLTMG7OUQF7HEBQC4VK5PGI3GT5J42MZJP4NAOUF6E4DCA, seg_pos={0 0}, upload=486602]: Long tail cancellation due to optimal threshold Dec 28 18:30:26.653216292 (555h1m26.524299337s) [noise=true , address=45.91.116.10:4084 , piece_id=HP6KY3HIJWTJ2BGLYUMI6MQOTJQOZOATMMTLOEIIE7AIWLI6IUIA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 28 18:30:26.653225009 (555h1m26.524308074s) [noise=true , address=45.91.116.10:4084 , piece_id=HP6KY3HIJWTJ2BGLYUMI6MQOTJQOZOATMMTLOEIIE7AIWLI6IUIA, seg_pos={0 0}, upload=486602]: Long tail cancellation due to optimal threshold Dec 28 18:30:26.653244837 (555h1m26.524327872s) [seg_pos={0 0}, upload=486602]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:30:26.653328276 (555h1m26.524411321s) [upload=486602]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:30:26.653334898 (555h1m26.524417943s) [upload=486602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:30:26.957763122 (555h1m26.828846167s) [upload=486602]: Done uploading segments... Dec 28 18:30:26.957789813 (555h1m26.828872858s) [upload=486602]: Upload finished. err: <nil> Duration: 6.02156686s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X........X.............X.X........................X....] duration: 26.529835468s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:31:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 26.529855787s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 3.514239903s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:31:00 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.514258559s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [...X..XX.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X] duration: 30.002742206s trace: https://jaeger.dev.storj.io/trace/291c428b844a0474 FAIL (2025-12-28 18:30:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: SPJHYU4QFIGCNOUXJRHC6PJ77NPAUX7PXKIRJPYRKQL2VEGOYRPQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: N336ELTV7WYFZGP6LSXGWQ4VA5FPZYZCOR6MYOIU7J3MVVIP5IRQ) context deadline exceeded ======LOGS====== 1m23.977394577s) [upload=486602]: Uploading segments... Dec 28 18:30:24.106371696 (555h1m23.977454741s) [upload=486602]: Got next segment. Inline: false Dec 28 18:30:24.106401292 (555h1m23.977484337s) [upload=486602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:30:24.106405650 (555h1m23.977488696s) [upload=486602]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:30:24.356211154 (555h1m24.227294199s) [seg_pos={0 0}, upload=486602]: Begin upload segment... Dec 28 18:30:24.356307267 (555h1m24.227390302s) [seg_pos={0 0}, upload=486602]: Done begin upload segment. Dec 28 18:30:24.356344198 (555h1m24.227427243s) [noise=true , address=145.79.227.48:4301 , piece_id=KAUC6B4SNHGYVLVNJCKAXZ2ZOX23ORZJGN45PVCZ2FJ7WWA2DYNQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.362040965 (555h1m24.23312401s) [noise=true , address=45.91.116.10:4084 , piece_id=HP6KY3HIJWTJ2BGLYUMI6MQOTJQOZOATMMTLOEIIE7AIWLI6IUIA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.363959479 (555h1m24.235042514s) [noise=true , address=145.79.227.3:4150 , piece_id=IXI6YYWC3VZDXB6GQIOF4PNAXEQXT4EQRZ3GXUREEUNPNTXHXHUQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.365049704 (555h1m24.236132749s) [noise=true , address=145.79.227.37:4566 , piece_id=7JECM4YX64MUNTNVNBL3GDEOXBAJYTWTLI3AAQE27YBSIUJSZG6Q, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.366012729 (555h1m24.237095774s) [noise=true , address=77.90.31.15:4697 , piece_id=B2HTKHQKFOOO6MRIDY33MNQDLR3QHWNIIFTXK4W44ZBFFJQHQWFA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.366082111 (555h1m24.237165146s) [noise=true , address=77.90.31.14:4663 , piece_id=N336ELTV7WYFZGP6LSXGWQ4VA5FPZYZCOR6MYOIU7J3MVVIP5IRQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.366707802 (555h1m24.237790847s) [noise=true , address=145.79.227.13:4803 , piece_id=WGD4HUTUX2OCQNIO3FNY4PR5AJEEJ7X2XM4PYTSZFKYDINNI44PA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.366805537 (555h1m24.237888582s) [noise=true , address=145.79.227.7:4295 , piece_id=764M3EY6NZQHJ2VNHYXP5QFGVCICQU6ACQO7KK7TKZFWCGLW7YZA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.367179119 (555h1m24.238262164s) [noise=true , address=145.79.227.39:4015 , piece_id=KJDFSCM4R2QCQE4AXNMWZ3S2WDYXXCGX2AMEQTILBBOXKA4MLTNQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.367317152 (555h1m24.238400187s) [noise=true , address=45.91.116.14:4192 , piece_id=SPJHYU4QFIGCNOUXJRHC6PJ77NPAUX7PXKIRJPYRKQL2VEGOYRPQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.367667419 (555h1m24.238750474s) [noise=true , address=145.79.227.38:4977 , piece_id=MPSYTE7G6SLY52KRFGM6I3ILMVITDYQHWCESTY53X6XEQJJWPDMQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.367832303 (555h1m24.238915338s) [noise=true , address=45.91.116.4:4940 , piece_id=IVFBCBSLTMG7OUQF7HEBQC4VK5PGI3GT5J42MZJP4NAOUF6E4DCA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.368185936 (555h1m24.239268981s) [noise=true , address=145.79.227.52:4447 , piece_id=5JBYMCC34PVNXD7QOAOKLJHYJBRAVGEHWJMSQCYLEOG4KJFJEDPA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.368298510 (555h1m24.239381545s) [noise=true , address=145.79.227.15:4874 , piece_id=FT54ZA3ZN7NZ4LCNVJ2SPOBAYNQ6Q7IGVGRFRE7TJJDKXCPY7Q5A, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.368769107 (555h1m24.239852142s) [noise=true , address=145.79.227.41:4087 , piece_id=LKCKLONESBRTDKDVP4DA2JV3GHAETLT3ADIH7AQDNJZQRFJ65TYA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.369292824 (555h1m24.240375869s) [noise=true , address=145.79.227.45:4231 , piece_id=LAPV3G65R5MDPAWY7FDCFTFJZY4CSEXMNJUF5PPMDG7GISMXSETQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:25.987984716 (555h1m25.859067751s) [upload=486602]: Next returned nil segment Dec 28 18:30:25.987999474 (555h1m25.859082519s) [upload=486602]: Waiting for error group managing segments... Dec 28 18:30:26.371269827 (555h1m26.242352872s) [noise=true , address=145.79.227.39:4015 , piece_id=KJDFSCM4R2QCQE4AXNMWZ3S2WDYXXCGX2AMEQTILBBOXKA4MLTNQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.375273340 (555h1m26.246356375s) [noise=true , address=145.79.227.37:4566 , piece_id=7JECM4YX64MUNTNVNBL3GDEOXBAJYTWTLI3AAQE27YBSIUJSZG6Q, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.377276505 (555h1m26.24835955s) [noise=true , address=145.79.227.7:4295 , piece_id=764M3EY6NZQHJ2VNHYXP5QFGVCICQU6ACQO7KK7TKZFWCGLW7YZA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.385263112 (555h1m26.256346157s) [noise=true , address=145.79.227.38:4977 , piece_id=MPSYTE7G6SLY52KRFGM6I3ILMVITDYQHWCESTY53X6XEQJJWPDMQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.388294895 (555h1m26.25937794s) [noise=true , address=145.79.227.52:4447 , piece_id=5JBYMCC34PVNXD7QOAOKLJHYJBRAVGEHWJMSQCYLEOG4KJFJEDPA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.392247552 (555h1m26.263330597s) [noise=true , address=145.79.227.45:4231 , piece_id=LAPV3G65R5MDPAWY7FDCFTFJZY4CSEXMNJUF5PPMDG7GISMXSETQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.399676858 (555h1m26.270759894s) [noise=true , address=145.79.227.48:4301 , piece_id=KAUC6B4SNHGYVLVNJCKAXZ2ZOX23ORZJGN45PVCZ2FJ7WWA2DYNQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.400307971 (555h1m26.271391016s) [noise=true , address=145.79.227.41:4087 , piece_id=LKCKLONESBRTDKDVP4DA2JV3GHAETLT3ADIH7AQDNJZQRFJ65TYA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.402205133 (555h1m26.273288168s) [noise=true , address=145.79.227.15:4874 , piece_id=FT54ZA3ZN7NZ4LCNVJ2SPOBAYNQ6Q7IGVGRFRE7TJJDKXCPY7Q5A, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.565360970 (555h1m26.436444015s) [noise=true , address=77.90.31.14:4663 , piece_id=N336ELTV7WYFZGP6LSXGWQ4VA5FPZYZCOR6MYOIU7J3MVVIP5IRQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.589487802 (555h1m26.460570847s) [noise=true , address=145.79.227.13:4803 , piece_id=WGD4HUTUX2OCQNIO3FNY4PR5AJEEJ7X2XM4PYTSZFKYDINNI44PA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.630597672 (555h1m26.501680717s) [noise=true , address=45.91.116.14:4192 , piece_id=SPJHYU4QFIGCNOUXJRHC6PJ77NPAUX7PXKIRJPYRKQL2VEGOYRPQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.652758160 (555h1m26.523841205s) [noise=true , address=77.90.31.15:4697 , piece_id=B2HTKHQKFOOO6MRIDY33MNQDLR3QHWNIIFTXK4W44ZBFFJQHQWFA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.652783077 (555h1m26.523866113s) [seg_pos={0 0}, upload=486602]: Segment reached optimal threshold of 13 pieces. Dec 28 18:30:26.653003737 (555h1m26.524086782s) [noise=true , address=145.79.227.3:4150 , piece_id=IXI6YYWC3VZDXB6GQIOF4PNAXEQXT4EQRZ3GXUREEUNPNTXHXHUQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: write tcp 188.166.225.170:53738->145.79.227.3:4150: use of closed network connection; write tcp 188.166.225.170:53738->145.79.227.3:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53738->145.79.227.3:4150: use of closed network connection Dec 28 18:30:26.653014498 (555h1m26.524097534s) [noise=true , address=145.79.227.3:4150 , piece_id=IXI6YYWC3VZDXB6GQIOF4PNAXEQXT4EQRZ3GXUREEUNPNTXHXHUQ, seg_pos={0 0}, upload=486602]: Long tail cancellation due to optimal threshold Dec 28 18:30:26.653025519 (555h1m26.524108564s) [noise=true , address=45.91.116.4:4940 , piece_id=IVFBCBSLTMG7OUQF7HEBQC4VK5PGI3GT5J42MZJP4NAOUF6E4DCA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:57704->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:57704->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57704->45.91.116.4:4940: use of closed network connection Dec 28 18:30:26.653033464 (555h1m26.524116509s) [noise=true , address=45.91.116.4:4940 , piece_id=IVFBCBSLTMG7OUQF7HEBQC4VK5PGI3GT5J42MZJP4NAOUF6E4DCA, seg_pos={0 0}, upload=486602]: Long tail cancellation due to optimal threshold Dec 28 18:30:26.653216292 (555h1m26.524299337s) [noise=true , address=45.91.116.10:4084 , piece_id=HP6KY3HIJWTJ2BGLYUMI6MQOTJQOZOATMMTLOEIIE7AIWLI6IUIA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 28 18:30:26.653225009 (555h1m26.524308074s) [noise=true , address=45.91.116.10:4084 , piece_id=HP6KY3HIJWTJ2BGLYUMI6MQOTJQOZOATMMTLOEIIE7AIWLI6IUIA, seg_pos={0 0}, upload=486602]: Long tail cancellation due to optimal threshold Dec 28 18:30:26.653244837 (555h1m26.524327872s) [seg_pos={0 0}, upload=486602]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:30:26.653328276 (555h1m26.524411321s) [upload=486602]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:30:26.653334898 (555h1m26.524417943s) [upload=486602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:30:26.957763122 (555h1m26.828846167s) [upload=486602]: Done uploading segments... Dec 28 18:30:26.957789813 (555h1m26.828872858s) [upload=486602]: Upload finished. err: <nil> Duration: 30.002771211s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X........X.............X.X........................X...] duration: 40.67328265s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:30:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 40.673314061s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 3.371907608s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:30:09 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.371950841s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....X..XX.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..] duration: 5.97793625s trace: https://jaeger.dev.storj.io/trace/10cce5e50b12fdca FAIL (2025-12-28 18:29:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== context deadline exceeded ======LOGS====== Dec 28 18:28:18.510655659 (554h59m18.381738704s) [upload=486569]: Starting upload Dec 28 18:28:18.510739759 (554h59m18.381822794s) [upload=486569]: Done starting upload Dec 28 18:28:18.510866611 (554h59m18.381949646s) [upload=486569]: Uploading segments... Dec 28 18:28:18.510955830 (554h59m18.382038865s) [upload=486569]: Got next segment. Inline: false Dec 28 18:28:18.510997801 (554h59m18.382080846s) [upload=486569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:28:18.511003402 (554h59m18.382086447s) [upload=486569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:28:18.753609916 (554h59m18.624692951s) [seg_pos={0 0}, upload=486569]: Begin upload segment... Dec 28 18:28:18.753673928 (554h59m18.624756963s) [seg_pos={0 0}, upload=486569]: Done begin upload segment. Dec 28 18:28:18.753698926 (554h59m18.624781971s) [noise=true , address=145.79.227.21:4475 , piece_id=Z3MOXVOSAL3DD6HSQNEMROMTQW2VG4R4IFWNCNADJUGWA6TW4MDQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.753719335 (554h59m18.62480237s) [noise=true , address=45.91.116.9:4263 , piece_id=4OGTBR2P6KTS6OTBE5EFCWID624QSHJQ3WNJNJND4LBZQJC764OA, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.754103286 (554h59m18.625186321s) [noise=true , address=145.79.227.48:4301 , piece_id=UMEGJE2377E7WMTDSDJH6DKLVS4DICR7JFM7GZ7WKHMOF35R2MXQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.754378079 (554h59m18.625461124s) [noise=true , address=45.91.116.5:4298 , piece_id=KWNNNBTDYPBDJNTGVXXFEKDZBTGRM26BMEW27LSQYCRFMVJ6FP7Q, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.754787488 (554h59m18.625870524s) [noise=true , address=145.79.227.35:4482 , piece_id=OLDUYIJ33O2UP7ABG56BEKPX3G76KDU3ZTVMAU4VYABAPGPFVTUA, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.755053716 (554h59m18.626136751s) [noise=true , address=45.91.116.6:4975 , piece_id=C54VHUM3DO4OOVDCSICLR65VDMDVM7UWQ5DYEKFOV6W367ENML2A, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.755319252 (554h59m18.626402287s) [noise=true , address=145.79.227.46:4603 , piece_id=EWGI4MV4KWI6ZPNIKG54GOJ35TM25LV6OKFWDW3EMJH5T4OS7X6Q, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.755587071 (554h59m18.626670106s) [noise=true , address=145.79.227.15:4876 , piece_id=SIKUFW7EZSB5JPYAVVTOOCTOUUBFYZXTUEQBC2AK3GBAVUE3BTNQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.755909364 (554h59m18.6269924s) [noise=true , address=45.91.116.12:4227 , piece_id=HH6FCJIS5OSJDVLNYY5PQJKT4GEVQOVSWU7FQFRG5HOQWVJAQNSQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.757152271 (554h59m18.628235317s) [noise=true , address=145.79.227.51:4409 , piece_id=ZXRRW6NXDE2MGQ7N7QHLBL7XLW3BRBCO4ZBPC7TCU253MCYSHRRA, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.758317020 (554h59m18.629400065s) [noise=true , address=145.79.227.43:4157 , piece_id=DGTK4F7D37ZGGOSWZUT67KRIBJ4G3ZL3FOPNCELGHPP2KL3QP3PQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.759644397 (554h59m18.630727432s) [noise=true , address=145.79.227.41:4087 , piece_id=UJ3KJP327STFFFSM2O2DETOYO5ZYMD4PCSPAS7WB4SKWJKAUJRNQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.761127371 (554h59m18.632210406s) [noise=true , address=145.79.227.11:4732 , piece_id=OLQVBQED5QTKCMRFENXBTQVOXGLBEFJVAOMOITS7MPELN3JAVTVQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.762709013 (554h59m18.633792058s) [noise=true , address=77.90.31.12:4590 , piece_id=IRIW2ZL2JGWIVKWI4ORBZUGXEYSGLKAZQSPNAUNK76WK6QMTTUOQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.763946980 (554h59m18.635030015s) [noise=true , address=77.90.31.15:4697 , piece_id=2ZT44C2RCN4F7LXIHTJPOBZJRKJ5XAX7HOPVNRVCQJXPGTCJRB6Q, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.764949348 (554h59m18.636032394s) [noise=true , address=145.79.227.34:4907 , piece_id=HAE4QTFFUAOJWZXFPJLSQKNDNILZB5H3T5AKRFGM7YSBEG7LYLWQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:20.469372967 (554h59m20.340456002s) [upload=486569]: Next returned nil segment Dec 28 18:28:20.469387253 (554h59m20.340470289s) [upload=486569]: Waiting for error group managing segments... Dec 28 18:28:20.766232138 (554h59m20.637315173s) [noise=true , address=145.79.227.11:4732 , piece_id=OLQVBQED5QTKCMRFENXBTQVOXGLBEFJVAOMOITS7MPELN3JAVTVQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.776862000 (554h59m20.647945035s) [noise=true , address=145.79.227.48:4301 , piece_id=UMEGJE2377E7WMTDSDJH6DKLVS4DICR7JFM7GZ7WKHMOF35R2MXQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.777963156 (554h59m20.649046191s) [noise=true , address=145.79.227.21:4475 , piece_id=Z3MOXVOSAL3DD6HSQNEMROMTQW2VG4R4IFWNCNADJUGWA6TW4MDQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.778884992 (554h59m20.649968027s) [noise=true , address=145.79.227.46:4603 , piece_id=EWGI4MV4KWI6ZPNIKG54GOJ35TM25LV6OKFWDW3EMJH5T4OS7X6Q, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.781995444 (554h59m20.653078479s) [noise=true , address=145.79.227.15:4876 , piece_id=SIKUFW7EZSB5JPYAVVTOOCTOUUBFYZXTUEQBC2AK3GBAVUE3BTNQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.786417855 (554h59m20.65750089s) [noise=true , address=145.79.227.51:4409 , piece_id=ZXRRW6NXDE2MGQ7N7QHLBL7XLW3BRBCO4ZBPC7TCU253MCYSHRRA, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.794969359 (554h59m20.666052394s) [noise=true , address=145.79.227.41:4087 , piece_id=UJ3KJP327STFFFSM2O2DETOYO5ZYMD4PCSPAS7WB4SKWJKAUJRNQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.976239159 (554h59m20.847322194s) [noise=true , address=77.90.31.12:4590 , piece_id=IRIW2ZL2JGWIVKWI4ORBZUGXEYSGLKAZQSPNAUNK76WK6QMTTUOQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.000438899 (554h59m20.871521954s) [noise=true , address=45.91.116.6:4975 , piece_id=C54VHUM3DO4OOVDCSICLR65VDMDVM7UWQ5DYEKFOV6W367ENML2A, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.027118281 (554h59m20.898201316s) [noise=true , address=45.91.116.12:4227 , piece_id=HH6FCJIS5OSJDVLNYY5PQJKT4GEVQOVSWU7FQFRG5HOQWVJAQNSQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.063174649 (554h59m20.934257684s) [noise=true , address=77.90.31.15:4697 , piece_id=2ZT44C2RCN4F7LXIHTJPOBZJRKJ5XAX7HOPVNRVCQJXPGTCJRB6Q, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.167129967 (554h59m21.038213002s) [noise=true , address=45.91.116.5:4298 , piece_id=KWNNNBTDYPBDJNTGVXXFEKDZBTGRM26BMEW27LSQYCRFMVJ6FP7Q, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.739100729 (554h59m21.610183774s) [noise=true , address=45.91.116.9:4263 , piece_id=4OGTBR2P6KTS6OTBE5EFCWID624QSHJQ3WNJNJND4LBZQJC764OA, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.739127150 (554h59m21.610210185s) [seg_pos={0 0}, upload=486569]: Segment reached optimal threshold of 13 pieces. Dec 28 18:28:21.739370012 (554h59m21.610453047s) [noise=true , address=145.79.227.43:4157 , piece_id=DGTK4F7D37ZGGOSWZUT67KRIBJ4G3ZL3FOPNCELGHPP2KL3QP3PQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 28 18:28:21.739378138 (554h59m21.610461173s) [noise=true , address=145.79.227.43:4157 , piece_id=DGTK4F7D37ZGGOSWZUT67KRIBJ4G3ZL3FOPNCELGHPP2KL3QP3PQ, seg_pos={0 0}, upload=486569]: Long tail cancellation due to optimal threshold Dec 28 18:28:21.739603837 (554h59m21.610686872s) [noise=true , address=145.79.227.35:4482 , piece_id=OLDUYIJ33O2UP7ABG56BEKPX3G76KDU3ZTVMAU4VYABAPGPFVTUA, seg_pos={0 0}, upload=486569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 28 18:28:21.739611793 (554h59m21.610694838s) [noise=true , address=145.79.227.35:4482 , piece_id=OLDUYIJ33O2UP7ABG56BEKPX3G76KDU3ZTVMAU4VYABAPGPFVTUA, seg_pos={0 0}, upload=486569]: Long tail cancellation due to optimal threshold Dec 28 18:28:21.739663872 (554h59m21.610746907s) [noise=true , address=145.79.227.34:4907 , piece_id=HAE4QTFFUAOJWZXFPJLSQKNDNILZB5H3T5AKRFGM7YSBEG7LYLWQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 28 18:28:21.739682357 (554h59m21.610765402s) [noise=true , address=145.79.227.34:4907 , piece_id=HAE4QTFFUAOJWZXFPJLSQKNDNILZB5H3T5AKRFGM7YSBEG7LYLWQ, seg_pos={0 0}, upload=486569]: Long tail cancellation due to optimal threshold Dec 28 18:28:21.739723716 (554h59m21.610806761s) [seg_pos={0 0}, upload=486569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:28:21.739847220 (554h59m21.610930265s) [upload=486569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:28:21.739853894 (554h59m21.610936939s) [upload=486569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:28:22.032616163 (554h59m21.903699208s) [upload=486569]: Done uploading segments... Dec 28 18:28:22.032643936 (554h59m21.903727001s) [upload=486569]: Upload finished. err: <nil> Duration: 5.977965746s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)