Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.X...X....X.X.....X.................XX.....X.X......X.X..XX] duration: 30.00387438s trace: https://jaeger.dev.storj.io/trace/1b6d44ecfce121e7 FAIL (2025-12-30 01:38:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 01:37:50.379992278 (586h8m50.251075323s) [upload=513837]: Starting upload Dec 30 01:37:50.380101345 (586h8m50.25118439s) [upload=513837]: Done starting upload Dec 30 01:37:50.380205363 (586h8m50.251288398s) [upload=513837]: Uploading segments... Dec 30 01:37:50.380373043 (586h8m50.251456078s) [upload=513837]: Got next segment. Inline: false Dec 30 01:37:50.380414551 (586h8m50.251497587s) [upload=513837]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:37:50.380420613 (586h8m50.251503658s) [upload=513837]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:37:50.625426289 (586h8m50.496509324s) [seg_pos={0 0}, upload=513837]: Begin upload segment... Dec 30 01:37:50.625474971 (586h8m50.496558006s) [seg_pos={0 0}, upload=513837]: Done begin upload segment. Dec 30 01:37:50.625499919 (586h8m50.496582954s) [noise=true , address=145.79.227.13:4802 , piece_id=66IYTXYJJQOASFN55SNCRKQ6XTEG7RX3U6KZKL6BYZ3LGSG5SFEA, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.625581664 (586h8m50.496664699s) [noise=true , address=45.91.116.7:4012 , piece_id=X4HFHRLPY5N3YESK3FDUV3L7L5ZSAX7HPB34FLZ2VPKJVGPKKOTQ, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.626419940 (586h8m50.497502985s) [noise=true , address=145.79.227.46:4603 , piece_id=BY7QVLNIAIIJYXEZKOIE2HK53WVX5UZAKRNSBBIGUNMMRPXWHY2Q, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.626543516 (586h8m50.497626561s) [noise=true , address=145.79.227.19:4079 , piece_id=5EAL2NGIHFXMYBCBHYJHX5CQC6LCALTAXH5ZVHIKAWSX7BEPLJMQ, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.626725733 (586h8m50.497808768s) [noise=true , address=145.79.227.38:4977 , piece_id=T2HZLMIQZE5GS6ZJ6V527JYBOFS4SXLABDWXQCQQWGXS4UTHDVEQ, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.627100095 (586h8m50.49818313s) [noise=true , address=77.90.31.15:4700 , piece_id=YHWM4LC7OT57QUKZGNBUEYKHZBMSVWFXU7ZL3L462QVDFBCMRTJQ, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.627184977 (586h8m50.498268022s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPEBE2BBZO2EH53A5OY5MINJUKJSCN7BEN2OM5MBMPO7M7MRXKPQ, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.627396790 (586h8m50.498479825s) [noise=true , address=145.79.227.47:4265 , piece_id=BXBESI5LTWBSFLBKTHW5RC4PIDHHHA32THEQ6KS6JIHS3BJTQ5DA, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.627704056 (586h8m50.498787091s) [noise=true , address=145.79.227.45:4230 , piece_id=JQJIWPAQQUS4YTYH4CRN6TOUSIDNUWSAUZGOMRETBQDUTM7QW65Q, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.627759901 (586h8m50.498842936s) [noise=true , address=45.91.116.4:4937 , piece_id=CP2FPGBDKNGXQWG3IHXXT4YEMJE73LLAQ3KSDSAW5Z2DTQ4GG5BA, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.628031499 (586h8m50.499114534s) [noise=true , address=77.90.31.21:4880 , piece_id=W4BACYY6ERKB6JDDH2G4CTRK5BGYLR36WM22SY5UD6XY57H2X5EQ, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.628221511 (586h8m50.499304556s) [noise=true , address=145.79.227.20:4439 , piece_id=FCBPUA2VG6Q6THQ6RK36WUJBVGBV6QXYLZH5BWIRQV6DVOG6KFHA, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.628375484 (586h8m50.499458519s) [noise=true , address=145.79.227.50:4373 , piece_id=MQKL2HRBAA7JPGEEWPMMIVUY2J7WNRTSYRCHU6S55PFDYD52A5XA, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.629076749 (586h8m50.500159804s) [noise=true , address=77.90.31.12:4590 , piece_id=GZH45VSG3Y6JGMRBJKGT4ZQVHODT72WEMEADYF4NM5FEWGYZMKPA, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.629604844 (586h8m50.500687889s) [noise=true , address=145.79.227.15:4875 , piece_id=JYAIPK357MEZYKPE5ZR2BD27PIDPWM5XSMJEXP36DQFS3KVAMZMQ, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:50.630977147 (586h8m50.502060182s) [noise=true , address=77.90.31.16:4735 , piece_id=IQKYV372RCZWSCDZBIQOZTISTUTUXRYEML3YJOE6UBPMQZMEHLHA, seg_pos={0 0}, upload=513837]: Uploading piece... Dec 30 01:37:52.326940740 (586h8m52.198023775s) [upload=513837]: Next returned nil segment Dec 30 01:37:52.326953133 (586h8m52.198036168s) [upload=513837]: Waiting for error group managing segments... Dec 30 01:37:52.599100139 (586h8m52.470183184s) [noise=true , address=145.79.227.13:4802 , piece_id=66IYTXYJJQOASFN55SNCRKQ6XTEG7RX3U6KZKL6BYZ3LGSG5SFEA, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.607723930 (586h8m52.478806985s) [noise=true , address=145.79.227.45:4230 , piece_id=JQJIWPAQQUS4YTYH4CRN6TOUSIDNUWSAUZGOMRETBQDUTM7QW65Q, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.608660193 (586h8m52.479743238s) [noise=true , address=145.79.227.17:4971 , piece_id=ZPEBE2BBZO2EH53A5OY5MINJUKJSCN7BEN2OM5MBMPO7M7MRXKPQ, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.609548224 (586h8m52.480631269s) [noise=true , address=145.79.227.46:4603 , piece_id=BY7QVLNIAIIJYXEZKOIE2HK53WVX5UZAKRNSBBIGUNMMRPXWHY2Q, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.611079500 (586h8m52.482162545s) [noise=true , address=145.79.227.15:4875 , piece_id=JYAIPK357MEZYKPE5ZR2BD27PIDPWM5XSMJEXP36DQFS3KVAMZMQ, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.617429281 (586h8m52.488512326s) [noise=true , address=145.79.227.38:4977 , piece_id=T2HZLMIQZE5GS6ZJ6V527JYBOFS4SXLABDWXQCQQWGXS4UTHDVEQ, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.629767504 (586h8m52.500850539s) [noise=true , address=145.79.227.19:4079 , piece_id=5EAL2NGIHFXMYBCBHYJHX5CQC6LCALTAXH5ZVHIKAWSX7BEPLJMQ, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.644911611 (586h8m52.515994646s) [noise=true , address=145.79.227.20:4439 , piece_id=FCBPUA2VG6Q6THQ6RK36WUJBVGBV6QXYLZH5BWIRQV6DVOG6KFHA, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.704235833 (586h8m52.575318878s) [noise=true , address=145.79.227.47:4265 , piece_id=BXBESI5LTWBSFLBKTHW5RC4PIDHHHA32THEQ6KS6JIHS3BJTQ5DA, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.798030951 (586h8m52.669113996s) [noise=true , address=77.90.31.12:4590 , piece_id=GZH45VSG3Y6JGMRBJKGT4ZQVHODT72WEMEADYF4NM5FEWGYZMKPA, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.813278014 (586h8m52.684361059s) [noise=true , address=77.90.31.21:4880 , piece_id=W4BACYY6ERKB6JDDH2G4CTRK5BGYLR36WM22SY5UD6XY57H2X5EQ, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:52.991422325 (586h8m52.86250537s) [noise=true , address=45.91.116.7:4012 , piece_id=X4HFHRLPY5N3YESK3FDUV3L7L5ZSAX7HPB34FLZ2VPKJVGPKKOTQ, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:53.537779026 (586h8m53.408862061s) [noise=true , address=77.90.31.15:4700 , piece_id=YHWM4LC7OT57QUKZGNBUEYKHZBMSVWFXU7ZL3L462QVDFBCMRTJQ, seg_pos={0 0}, upload=513837]: Done uploading piece. err: <nil> Dec 30 01:37:53.537971512 (586h8m53.409054547s) [seg_pos={0 0}, upload=513837]: Segment reached optimal threshold of 13 pieces. Dec 30 01:37:53.538267857 (586h8m53.409350902s) [noise=true , address=145.79.227.50:4373 , piece_id=MQKL2HRBAA7JPGEEWPMMIVUY2J7WNRTSYRCHU6S55PFDYD52A5XA, seg_pos={0 0}, upload=513837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:42240->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:42240->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42240->145.79.227.50:4373: use of closed network connection Dec 30 01:37:53.538277455 (586h8m53.40936049s) [noise=true , address=145.79.227.50:4373 , piece_id=MQKL2HRBAA7JPGEEWPMMIVUY2J7WNRTSYRCHU6S55PFDYD52A5XA, seg_pos={0 0}, upload=513837]: Long tail cancellation due to optimal threshold Dec 30 01:37:53.538569520 (586h8m53.409652555s) [noise=true , address=45.91.116.4:4937 , piece_id=CP2FPGBDKNGXQWG3IHXXT4YEMJE73LLAQ3KSDSAW5Z2DTQ4GG5BA, seg_pos={0 0}, upload=513837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 30 01:37:53.538578448 (586h8m53.409661483s) [noise=true , address=45.91.116.4:4937 , piece_id=CP2FPGBDKNGXQWG3IHXXT4YEMJE73LLAQ3KSDSAW5Z2DTQ4GG5BA, seg_pos={0 0}, upload=513837]: Long tail cancellation due to optimal threshold Dec 30 01:37:53.538793167 (586h8m53.409876212s) [noise=true , address=77.90.31.16:4735 , piece_id=IQKYV372RCZWSCDZBIQOZTISTUTUXRYEML3YJOE6UBPMQZMEHLHA, seg_pos={0 0}, upload=513837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 30 01:37:53.538801633 (586h8m53.409884678s) [noise=true , address=77.90.31.16:4735 , piece_id=IQKYV372RCZWSCDZBIQOZTISTUTUXRYEML3YJOE6UBPMQZMEHLHA, seg_pos={0 0}, upload=513837]: Long tail cancellation due to optimal threshold Dec 30 01:37:53.538822843 (586h8m53.409905889s) [seg_pos={0 0}, upload=513837]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:37:53.538903917 (586h8m53.409986963s) [upload=513837]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:37:53.538909979 (586h8m53.409993024s) [upload=513837]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:37:53.833385321 (586h8m53.704468367s) [upload=513837]: Done uploading segments... Dec 30 01:37:53.833414106 (586h8m53.704497141s) [upload=513837]: Upload finished. err: <nil> Duration: 30.003897834s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 35.21963719s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:37:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 35.219655836s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.X...X....X.X.....X.................XX.....X.X......X.X..X] duration: 30.002951864s trace: https://jaeger.dev.storj.io/trace/7906bd1eb0e6be1 FAIL (2025-12-30 01:37:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 01:36:40.834951652 (586h7m40.706034687s) [upload=513824]: Starting upload Dec 30 01:36:40.835024431 (586h7m40.706107496s) [upload=513824]: Done starting upload Dec 30 01:36:40.835136515 (586h7m40.70621955s) [upload=513824]: Uploading segments... Dec 30 01:36:40.835375709 (586h7m40.706458745s) [upload=513824]: Got next segment. Inline: false Dec 30 01:36:40.835411458 (586h7m40.706494493s) [upload=513824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:36:40.835416157 (586h7m40.706499202s) [upload=513824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:36:41.081195074 (586h7m40.952278109s) [seg_pos={0 0}, upload=513824]: Begin upload segment... Dec 30 01:36:41.081269215 (586h7m40.95235225s) [seg_pos={0 0}, upload=513824]: Done begin upload segment. Dec 30 01:36:41.081295085 (586h7m40.95237812s) [noise=true , address=45.91.116.7:4009 , piece_id=RO2YR53XK66Y5BOXO7CXM3ZASEP3SGVSEI3YFSB2JIEIMJIGAB4A, seg_pos={0 0}, upload=513824]: Uploading piece... Dec 30 01:36:41.081937037 (586h7m40.953020072s) [noise=true , address=145.79.227.21:4475 , piece_id=7MGKH4ZZDUFCIMRVR3PUJU25U6AYSREQJWFM2WK32Q7TZO66FHZA, seg_pos={0 0}, upload=513824]: Uploading piece... Dec 30 01:36:41.082241367 (586h7m40.953324402s) [noise=true , address=77.90.31.15:4697 , piece_id=2625BAAWURFJ5YBCW3ULWD7HC7H5VZDSKADUTKR5YDSETYDLCTNA, seg_pos={0 0}, upload=513824]: Uploading piece... Dec 30 01:36:41.082698647 (586h7m40.953781682s) [noise=true , address=145.79.227.32:4836 , piece_id=HDEOW54EZNSLSDDKRIDUQGCETVZQYAL2MHPJEPWN4E5VSOH25M4Q, seg_pos={0 0}, upload=513824]: Uploading piece... Dec 30 01:36:41.083200373 (586h7m40.954283418s) [noise=true , address=145.79.227.46:4601 , piece_id=JHPSACZQJYVARK2JENZK4ECOJSIX6JZDFPN6MEEMYOZZU257G52Q, seg_pos={0 0}, upload=513824]: Uploading piece... Dec 30 01:36:41.083870379 (586h7m40.954953414s) [noise=true , address=45.91.116.5:4297 , piece_id=SX7Y5EIPFSZCMVN7AMQVYFNH47VWNMGAYDQAQFR3QG3AWPASRNRA, seg_pos={0 0}, upload=513824]: Uploading piece... Dec 30 01:36:41.084331306 (586h7m40.955414341s) [noise=true , address=145.79.227.28:4725 , piece_id=WCRKNT2R5HI73GDRMYZBP3GTMIWCCBW6ATJJNZ7QCZOCWCUBPXAA, seg_pos={0 0}, upload=513824]: Uploading piece... Dec 30 01:36:41.084689448 (586h7m40.955772493s) [noise=true , address=145.79.227.7:4295 , piece_id=UU7WKYZ67HXE7XWD5MLJLHDN7Y6ASMBF3T4TPUPCXNDY4UOAW4TQ, seg_pos={0 0}, upload=513824]: Uploading piece... Dec 30 01:36:41.084739133 (586h7m40.955822168s) [noise=true , address=145.79.227.5:4221 , piece_id=ZKHTLX6CCGAVJWKSZ55VR7FWQBZFXEI46N6OTW5DSFXBPW5R5YOQ, seg_pos={0 0}, upload=513824]: Uploading piece... Dec 30 01:36:41.085127172 (586h7m40.956210207s) [noise=true , address=145.79.227.52:4446 , piece_id=24CFCQNBNYZS6C4O3PQALJJ7EEXVBPMJE3673AWZPETJQ23MOQGQ, seg_pos={0 0}, upload=513824]: Uploading piece... Dec 30 01:36:41.085307525 (586h7m40.95639057s) [noise=true , address=145.79.227.10:4402 , piece_id=PJNTCQWR2YFH2IYSR3Q4SGJ33DUJI4KPUSSCHJRAL3HDEF3ZHV4Q, seg_pos={0 0}, upload=513824]: Uploading piece... Dec 30 01:36:41.085917307 (586h7m40.957000342s) [noise=true , address=145.79.227.16:4910 , piece_id=TORTCFWCE7Z36O4PTXUJFFLM2Z4VR5DU2RKPDKO5ECMBZXETX2HQ, seg_pos={0 0}, upload=513824]: Uploading piece... Dec 30 01:36:41.086002448 (586h7m40.957085493s) [noise=true , address=145.79.227.11:4729 , piece_id=AWM2KXMFXKN5MNWZACC3PHBY2US5XYOLHWPDHVGPVF7FP3WM4T4A, seg_pos={0 0}, upload=513824]: Uploading piece... Dec 30 01:36:41.086263635 (586h7m40.95734667s) [noise=true , address=145.79.227.38:4977 , piece_id=P35ZE2EGIAMBIGVCDA2FKQVYHQAVYTATCXAMPXCODDDJAL5MDCUA, seg_pos={0 0}, upload=513824]: Uploading piece... Dec 30 01:36:41.086558487 (586h7m40.957641522s) [noise=true , address=45.91.116.8:4048 , piece_id=QIP666GVYHCFRIZD22R7PDV7A4PPT23NPRN5DEAKF7JQIED4XNYA, seg_pos={0 0}, upload=513824]: Uploading piece... Dec 30 01:36:41.086597451 (586h7m40.957680486s) [noise=true , address=145.79.227.48:4302 , piece_id=AXRJQB4AKQKJYXVU47ZNTS77CYLYHKHNQOAKADJO3JZBY65GPC6A, seg_pos={0 0}, upload=513824]: Uploading piece... Dec 30 01:36:42.678684888 (586h7m42.549767923s) [upload=513824]: Next returned nil segment Dec 30 01:36:42.678699075 (586h7m42.54978211s) [upload=513824]: Waiting for error group managing segments... Dec 30 01:36:43.058679982 (586h7m42.929763017s) [noise=true , address=145.79.227.11:4729 , piece_id=AWM2KXMFXKN5MNWZACC3PHBY2US5XYOLHWPDHVGPVF7FP3WM4T4A, seg_pos={0 0}, upload=513824]: Done uploading piece. err: <nil> Dec 30 01:36:43.061571188 (586h7m42.932654233s) [noise=true , address=145.79.227.46:4601 , piece_id=JHPSACZQJYVARK2JENZK4ECOJSIX6JZDFPN6MEEMYOZZU257G52Q, seg_pos={0 0}, upload=513824]: Done uploading piece. err: <nil> Dec 30 01:36:43.074115744 (586h7m42.945198779s) [noise=true , address=145.79.227.32:4836 , piece_id=HDEOW54EZNSLSDDKRIDUQGCETVZQYAL2MHPJEPWN4E5VSOH25M4Q, seg_pos={0 0}, upload=513824]: Done uploading piece. err: <nil> Dec 30 01:36:43.074747136 (586h7m42.945830182s) [noise=true , address=145.79.227.7:4295 , piece_id=UU7WKYZ67HXE7XWD5MLJLHDN7Y6ASMBF3T4TPUPCXNDY4UOAW4TQ, seg_pos={0 0}, upload=513824]: Done uploading piece. err: <nil> Dec 30 01:36:43.076202707 (586h7m42.947285752s) [noise=true , address=145.79.227.10:4402 , piece_id=PJNTCQWR2YFH2IYSR3Q4SGJ33DUJI4KPUSSCHJRAL3HDEF3ZHV4Q, seg_pos={0 0}, upload=513824]: Done uploading piece. err: <nil> Dec 30 01:36:43.080608287 (586h7m42.951691322s) [noise=true , address=145.79.227.21:4475 , piece_id=7MGKH4ZZDUFCIMRVR3PUJU25U6AYSREQJWFM2WK32Q7TZO66FHZA, seg_pos={0 0}, upload=513824]: Done uploading piece. err: <nil> Dec 30 01:36:43.085939558 (586h7m42.957022593s) [noise=true , address=145.79.227.38:4977 , piece_id=P35ZE2EGIAMBIGVCDA2FKQVYHQAVYTATCXAMPXCODDDJAL5MDCUA, seg_pos={0 0}, upload=513824]: Done uploading piece. err: <nil> Dec 30 01:36:43.098998605 (586h7m42.97008164s) [noise=true , address=145.79.227.28:4725 , piece_id=WCRKNT2R5HI73GDRMYZBP3GTMIWCCBW6ATJJNZ7QCZOCWCUBPXAA, seg_pos={0 0}, upload=513824]: Done uploading piece. err: <nil> Dec 30 01:36:43.099121869 (586h7m42.970204914s) [noise=true , address=145.79.227.52:4446 , piece_id=24CFCQNBNYZS6C4O3PQALJJ7EEXVBPMJE3673AWZPETJQ23MOQGQ, seg_pos={0 0}, upload=513824]: Done uploading piece. err: <nil> Dec 30 01:36:43.104102042 (586h7m42.975185087s) [noise=true , address=145.79.227.16:4910 , piece_id=TORTCFWCE7Z36O4PTXUJFFLM2Z4VR5DU2RKPDKO5ECMBZXETX2HQ, seg_pos={0 0}, upload=513824]: Done uploading piece. err: <nil> Dec 30 01:36:43.113442237 (586h7m42.984525282s) [noise=true , address=145.79.227.48:4302 , piece_id=AXRJQB4AKQKJYXVU47ZNTS77CYLYHKHNQOAKADJO3JZBY65GPC6A, seg_pos={0 0}, upload=513824]: Done uploading piece. err: <nil> Dec 30 01:36:43.296882994 (586h7m43.167966039s) [noise=true , address=45.91.116.5:4297 , piece_id=SX7Y5EIPFSZCMVN7AMQVYFNH47VWNMGAYDQAQFR3QG3AWPASRNRA, seg_pos={0 0}, upload=513824]: Done uploading piece. err: <nil> Dec 30 01:36:43.305640690 (586h7m43.176723735s) [noise=true , address=45.91.116.8:4048 , piece_id=QIP666GVYHCFRIZD22R7PDV7A4PPT23NPRN5DEAKF7JQIED4XNYA, seg_pos={0 0}, upload=513824]: Done uploading piece. err: <nil> Dec 30 01:36:43.305665427 (586h7m43.176748462s) [seg_pos={0 0}, upload=513824]: Segment reached optimal threshold of 13 pieces. Dec 30 01:36:43.306182923 (586h7m43.177265958s) [noise=true , address=45.91.116.7:4009 , piece_id=RO2YR53XK66Y5BOXO7CXM3ZASEP3SGVSEI3YFSB2JIEIMJIGAB4A, seg_pos={0 0}, upload=513824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 01:36:43.306193994 (586h7m43.177277029s) [noise=true , address=45.91.116.7:4009 , piece_id=RO2YR53XK66Y5BOXO7CXM3ZASEP3SGVSEI3YFSB2JIEIMJIGAB4A, seg_pos={0 0}, upload=513824]: Long tail cancellation due to optimal threshold Dec 30 01:36:43.306206106 (586h7m43.177289151s) [noise=true , address=145.79.227.5:4221 , piece_id=ZKHTLX6CCGAVJWKSZ55VR7FWQBZFXEI46N6OTW5DSFXBPW5R5YOQ, seg_pos={0 0}, upload=513824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:58016->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:58016->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58016->145.79.227.5:4221: use of closed network connection Dec 30 01:36:43.306213861 (586h7m43.177296896s) [noise=true , address=145.79.227.5:4221 , piece_id=ZKHTLX6CCGAVJWKSZ55VR7FWQBZFXEI46N6OTW5DSFXBPW5R5YOQ, seg_pos={0 0}, upload=513824]: Long tail cancellation due to optimal threshold Dec 30 01:36:43.306422779 (586h7m43.177505814s) [noise=true , address=77.90.31.15:4697 , piece_id=2625BAAWURFJ5YBCW3ULWD7HC7H5VZDSKADUTKR5YDSETYDLCTNA, seg_pos={0 0}, upload=513824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 30 01:36:43.306431777 (586h7m43.177514812s) [noise=true , address=77.90.31.15:4697 , piece_id=2625BAAWURFJ5YBCW3ULWD7HC7H5VZDSKADUTKR5YDSETYDLCTNA, seg_pos={0 0}, upload=513824]: Long tail cancellation due to optimal threshold Dec 30 01:36:43.306455040 (586h7m43.177538075s) [seg_pos={0 0}, upload=513824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:36:43.306542237 (586h7m43.177625282s) [upload=513824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:36:43.306549721 (586h7m43.177632756s) [upload=513824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:36:43.599032850 (586h7m43.470115895s) [upload=513824]: Done uploading segments... Dec 30 01:36:43.599056224 (586h7m43.470139259s) [upload=513824]: Upload finished. err: <nil> Duration: 30.002979727s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 28.50154634s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:36:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 28.501559695s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 29.23605798s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:36:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 29.236075665s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X.X...X....X.X.....X.................XX.....X.X......X.X..] duration: 5.867366758s trace: https://jaeger.dev.storj.io/trace/2a09f9b1206496cd FAIL (2025-12-30 01:36:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: V3CSZOKKW7MNUJ4TJDAHEVSF75JSGOIOMSOLLFURX2ITMZJ5UHRQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: FM32Z47OXYSQXR3PVQOOX5764E4B4IHX2PATPNB3VG34WJZNTHQA) context deadline exceeded ======LOGS====== 2.68503247s) [upload=513796]: Uploading segments... Dec 30 01:34:52.814048624 (586h5m52.685131679s) [upload=513796]: Got next segment. Inline: false Dec 30 01:34:52.814113458 (586h5m52.685196493s) [upload=513796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:34:52.814120321 (586h5m52.685203376s) [upload=513796]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:34:53.060766941 (586h5m52.931849976s) [seg_pos={0 0}, upload=513796]: Begin upload segment... Dec 30 01:34:53.060817417 (586h5m52.931900452s) [seg_pos={0 0}, upload=513796]: Done begin upload segment. Dec 30 01:34:53.060861981 (586h5m52.931945016s) [noise=true , address=45.91.116.8:4045 , piece_id=E5TGZB4TARZ4KR2DKH74G7XHVD7VENPGYBU2I52UFJV4JYMZKIDQ, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.060899894 (586h5m52.931982929s) [noise=true , address=145.79.227.11:4729 , piece_id=26TCLWT3G4OOYQI55KNUFCQT46T5BTT5R43TBD7M2R7ZPUWV7H5Q, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.061019171 (586h5m52.932102206s) [noise=true , address=145.79.227.26:4653 , piece_id=23NKAZO4ND3WXKOJSR4SWHVTMUCWJ2FRKW2ZGLHKDCITRILHBXKA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.061576032 (586h5m52.932659077s) [noise=true , address=145.79.227.42:4122 , piece_id=4IL4HJLEI3GKNDZBEID5AUH57K47CJCAONTLQSRUYBY5WOLTIDAA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.063425213 (586h5m52.934508248s) [noise=true , address=145.79.227.38:4979 , piece_id=XS6Y45OFSXOXZ4SG2TG65EE6I6NSFLOKDPED3BZZJL4QXCLOZJOA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.063972325 (586h5m52.93505536s) [noise=true , address=145.79.227.37:4567 , piece_id=OGKEXSXQP6APGE7H4QPOD2KASV26CYD73MV2NBK36FDOSIB4CEWQ, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.064962710 (586h5m52.936045745s) [noise=true , address=45.91.116.9:4262 , piece_id=ND52YV7EQ2O4VQRONQO7VJ5VYHITSPFZCB5LBBKFQWEPT5K3OYEA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.065706987 (586h5m52.936790032s) [noise=true , address=77.90.31.16:4736 , piece_id=3MMRF5O6NQAPS4DA4VDFS722IPZKG57PWFPZUX5MT3B7PBYK5SMQ, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.066365220 (586h5m52.937448266s) [noise=true , address=145.79.227.28:4725 , piece_id=QI34SC6QSLLW6ZU3KQ2TUJSAAATNGWKDNS6V74LVAWDDKXDNLICA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.068040060 (586h5m52.939123095s) [noise=true , address=45.91.116.13:4154 , piece_id=V3CSZOKKW7MNUJ4TJDAHEVSF75JSGOIOMSOLLFURX2ITMZJ5UHRQ, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.070283672 (586h5m52.941366707s) [noise=true , address=45.91.116.11:4118 , piece_id=FM32Z47OXYSQXR3PVQOOX5764E4B4IHX2PATPNB3VG34WJZNTHQA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.072253904 (586h5m52.943336939s) [noise=true , address=45.91.116.10:4081 , piece_id=ZVDCI43IFZMDLG3VKWJSBBS57KA6OK5ZDPJU45NMV73AL3UWYIVA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.073051081 (586h5m52.944134126s) [noise=true , address=145.79.227.8:4329 , piece_id=GD6UN7ECXA4KLELVBPMZUHG3EGXL2GUI27DLA5DDC7B6QBNFZ7EA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.074116460 (586h5m52.945199496s) [noise=true , address=145.79.227.50:4373 , piece_id=FLWFANKIJXSRFE7LUZB5J74YCZEZ7QUHRGGPSNKF4VVYFUGYKMQA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.077440390 (586h5m52.948523435s) [noise=true , address=145.79.227.40:4050 , piece_id=2VFPC5NKF53RGPX47WG7MVTTD43PJECDJ42VJFNIED2P6GNVAEWA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.078955004 (586h5m52.950038039s) [noise=true , address=145.79.227.45:4230 , piece_id=2SCHR3NSIZYERLDLZ7QE3TNTYIVODZ3SUHGRVX5OK4OQV3IB2QBQ, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:54.763041389 (586h5m54.634124434s) [upload=513796]: Next returned nil segment Dec 30 01:34:54.763058281 (586h5m54.634141326s) [upload=513796]: Waiting for error group managing segments... Dec 30 01:34:55.046749328 (586h5m54.917832363s) [noise=true , address=145.79.227.11:4729 , piece_id=26TCLWT3G4OOYQI55KNUFCQT46T5BTT5R43TBD7M2R7ZPUWV7H5Q, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.050395240 (586h5m54.921478286s) [noise=true , address=145.79.227.45:4230 , piece_id=2SCHR3NSIZYERLDLZ7QE3TNTYIVODZ3SUHGRVX5OK4OQV3IB2QBQ, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.058869537 (586h5m54.929952572s) [noise=true , address=145.79.227.26:4653 , piece_id=23NKAZO4ND3WXKOJSR4SWHVTMUCWJ2FRKW2ZGLHKDCITRILHBXKA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.059713133 (586h5m54.930796178s) [noise=true , address=145.79.227.37:4567 , piece_id=OGKEXSXQP6APGE7H4QPOD2KASV26CYD73MV2NBK36FDOSIB4CEWQ, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.067453531 (586h5m54.938536566s) [noise=true , address=145.79.227.42:4122 , piece_id=4IL4HJLEI3GKNDZBEID5AUH57K47CJCAONTLQSRUYBY5WOLTIDAA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.071091760 (586h5m54.942174805s) [noise=true , address=145.79.227.38:4979 , piece_id=XS6Y45OFSXOXZ4SG2TG65EE6I6NSFLOKDPED3BZZJL4QXCLOZJOA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.073714904 (586h5m54.944797939s) [noise=true , address=145.79.227.28:4725 , piece_id=QI34SC6QSLLW6ZU3KQ2TUJSAAATNGWKDNS6V74LVAWDDKXDNLICA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.080796028 (586h5m54.951879063s) [noise=true , address=145.79.227.8:4329 , piece_id=GD6UN7ECXA4KLELVBPMZUHG3EGXL2GUI27DLA5DDC7B6QBNFZ7EA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.256029446 (586h5m55.127112481s) [noise=true , address=45.91.116.8:4045 , piece_id=E5TGZB4TARZ4KR2DKH74G7XHVD7VENPGYBU2I52UFJV4JYMZKIDQ, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.325496572 (586h5m55.196579607s) [noise=true , address=77.90.31.16:4736 , piece_id=3MMRF5O6NQAPS4DA4VDFS722IPZKG57PWFPZUX5MT3B7PBYK5SMQ, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.407302413 (586h5m55.278385478s) [noise=true , address=45.91.116.10:4081 , piece_id=ZVDCI43IFZMDLG3VKWJSBBS57KA6OK5ZDPJU45NMV73AL3UWYIVA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.608342330 (586h5m55.479425365s) [noise=true , address=45.91.116.13:4154 , piece_id=V3CSZOKKW7MNUJ4TJDAHEVSF75JSGOIOMSOLLFURX2ITMZJ5UHRQ, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:56.071907243 (586h5m55.942990308s) [noise=true , address=45.91.116.11:4118 , piece_id=FM32Z47OXYSQXR3PVQOOX5764E4B4IHX2PATPNB3VG34WJZNTHQA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:56.071936198 (586h5m55.943019233s) [seg_pos={0 0}, upload=513796]: Segment reached optimal threshold of 13 pieces. Dec 30 01:34:56.072162689 (586h5m55.943245724s) [noise=true , address=45.91.116.9:4262 , piece_id=ND52YV7EQ2O4VQRONQO7VJ5VYHITSPFZCB5LBBKFQWEPT5K3OYEA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:34650->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:34650->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34650->45.91.116.9:4262: use of closed network connection Dec 30 01:34:56.072171927 (586h5m55.943254972s) [noise=true , address=45.91.116.9:4262 , piece_id=ND52YV7EQ2O4VQRONQO7VJ5VYHITSPFZCB5LBBKFQWEPT5K3OYEA, seg_pos={0 0}, upload=513796]: Long tail cancellation due to optimal threshold Dec 30 01:34:56.072253001 (586h5m55.943336037s) [noise=true , address=145.79.227.50:4373 , piece_id=FLWFANKIJXSRFE7LUZB5J74YCZEZ7QUHRGGPSNKF4VVYFUGYKMQA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:44632->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.50:4373: use of closed network connection Dec 30 01:34:56.072274623 (586h5m55.943357698s) [noise=true , address=145.79.227.50:4373 , piece_id=FLWFANKIJXSRFE7LUZB5J74YCZEZ7QUHRGGPSNKF4VVYFUGYKMQA, seg_pos={0 0}, upload=513796]: Long tail cancellation due to optimal threshold Dec 30 01:34:56.072487628 (586h5m55.943570664s) [noise=true , address=145.79.227.40:4050 , piece_id=2VFPC5NKF53RGPX47WG7MVTTD43PJECDJ42VJFNIED2P6GNVAEWA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 01:34:56.072497417 (586h5m55.943580452s) [noise=true , address=145.79.227.40:4050 , piece_id=2VFPC5NKF53RGPX47WG7MVTTD43PJECDJ42VJFNIED2P6GNVAEWA, seg_pos={0 0}, upload=513796]: Long tail cancellation due to optimal threshold Dec 30 01:34:56.072519840 (586h5m55.943602875s) [seg_pos={0 0}, upload=513796]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:34:56.072610101 (586h5m55.943693146s) [upload=513796]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:34:56.072617535 (586h5m55.94370058s) [upload=513796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:34:56.367842725 (586h5m56.23892577s) [upload=513796]: Done uploading segments... Dec 30 01:34:56.367872270 (586h5m56.238955315s) [upload=513796]: Upload finished. err: <nil> Duration: 5.867454395s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 38.7701443s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:35:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 38.770158146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X.X...X....X.X.....X.................XX.....X.X......X.X.] duration: 6.03393716s trace: https://jaeger.dev.storj.io/trace/2a09f9b1206496cd FAIL (2025-12-30 01:35:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: V3CSZOKKW7MNUJ4TJDAHEVSF75JSGOIOMSOLLFURX2ITMZJ5UHRQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: FM32Z47OXYSQXR3PVQOOX5764E4B4IHX2PATPNB3VG34WJZNTHQA) context deadline exceeded ======LOGS====== 2.68503247s) [upload=513796]: Uploading segments... Dec 30 01:34:52.814048624 (586h5m52.685131679s) [upload=513796]: Got next segment. Inline: false Dec 30 01:34:52.814113458 (586h5m52.685196493s) [upload=513796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:34:52.814120321 (586h5m52.685203376s) [upload=513796]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:34:53.060766941 (586h5m52.931849976s) [seg_pos={0 0}, upload=513796]: Begin upload segment... Dec 30 01:34:53.060817417 (586h5m52.931900452s) [seg_pos={0 0}, upload=513796]: Done begin upload segment. Dec 30 01:34:53.060861981 (586h5m52.931945016s) [noise=true , address=45.91.116.8:4045 , piece_id=E5TGZB4TARZ4KR2DKH74G7XHVD7VENPGYBU2I52UFJV4JYMZKIDQ, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.060899894 (586h5m52.931982929s) [noise=true , address=145.79.227.11:4729 , piece_id=26TCLWT3G4OOYQI55KNUFCQT46T5BTT5R43TBD7M2R7ZPUWV7H5Q, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.061019171 (586h5m52.932102206s) [noise=true , address=145.79.227.26:4653 , piece_id=23NKAZO4ND3WXKOJSR4SWHVTMUCWJ2FRKW2ZGLHKDCITRILHBXKA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.061576032 (586h5m52.932659077s) [noise=true , address=145.79.227.42:4122 , piece_id=4IL4HJLEI3GKNDZBEID5AUH57K47CJCAONTLQSRUYBY5WOLTIDAA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.063425213 (586h5m52.934508248s) [noise=true , address=145.79.227.38:4979 , piece_id=XS6Y45OFSXOXZ4SG2TG65EE6I6NSFLOKDPED3BZZJL4QXCLOZJOA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.063972325 (586h5m52.93505536s) [noise=true , address=145.79.227.37:4567 , piece_id=OGKEXSXQP6APGE7H4QPOD2KASV26CYD73MV2NBK36FDOSIB4CEWQ, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.064962710 (586h5m52.936045745s) [noise=true , address=45.91.116.9:4262 , piece_id=ND52YV7EQ2O4VQRONQO7VJ5VYHITSPFZCB5LBBKFQWEPT5K3OYEA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.065706987 (586h5m52.936790032s) [noise=true , address=77.90.31.16:4736 , piece_id=3MMRF5O6NQAPS4DA4VDFS722IPZKG57PWFPZUX5MT3B7PBYK5SMQ, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.066365220 (586h5m52.937448266s) [noise=true , address=145.79.227.28:4725 , piece_id=QI34SC6QSLLW6ZU3KQ2TUJSAAATNGWKDNS6V74LVAWDDKXDNLICA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.068040060 (586h5m52.939123095s) [noise=true , address=45.91.116.13:4154 , piece_id=V3CSZOKKW7MNUJ4TJDAHEVSF75JSGOIOMSOLLFURX2ITMZJ5UHRQ, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.070283672 (586h5m52.941366707s) [noise=true , address=45.91.116.11:4118 , piece_id=FM32Z47OXYSQXR3PVQOOX5764E4B4IHX2PATPNB3VG34WJZNTHQA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.072253904 (586h5m52.943336939s) [noise=true , address=45.91.116.10:4081 , piece_id=ZVDCI43IFZMDLG3VKWJSBBS57KA6OK5ZDPJU45NMV73AL3UWYIVA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.073051081 (586h5m52.944134126s) [noise=true , address=145.79.227.8:4329 , piece_id=GD6UN7ECXA4KLELVBPMZUHG3EGXL2GUI27DLA5DDC7B6QBNFZ7EA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.074116460 (586h5m52.945199496s) [noise=true , address=145.79.227.50:4373 , piece_id=FLWFANKIJXSRFE7LUZB5J74YCZEZ7QUHRGGPSNKF4VVYFUGYKMQA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.077440390 (586h5m52.948523435s) [noise=true , address=145.79.227.40:4050 , piece_id=2VFPC5NKF53RGPX47WG7MVTTD43PJECDJ42VJFNIED2P6GNVAEWA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.078955004 (586h5m52.950038039s) [noise=true , address=145.79.227.45:4230 , piece_id=2SCHR3NSIZYERLDLZ7QE3TNTYIVODZ3SUHGRVX5OK4OQV3IB2QBQ, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:54.763041389 (586h5m54.634124434s) [upload=513796]: Next returned nil segment Dec 30 01:34:54.763058281 (586h5m54.634141326s) [upload=513796]: Waiting for error group managing segments... Dec 30 01:34:55.046749328 (586h5m54.917832363s) [noise=true , address=145.79.227.11:4729 , piece_id=26TCLWT3G4OOYQI55KNUFCQT46T5BTT5R43TBD7M2R7ZPUWV7H5Q, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.050395240 (586h5m54.921478286s) [noise=true , address=145.79.227.45:4230 , piece_id=2SCHR3NSIZYERLDLZ7QE3TNTYIVODZ3SUHGRVX5OK4OQV3IB2QBQ, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.058869537 (586h5m54.929952572s) [noise=true , address=145.79.227.26:4653 , piece_id=23NKAZO4ND3WXKOJSR4SWHVTMUCWJ2FRKW2ZGLHKDCITRILHBXKA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.059713133 (586h5m54.930796178s) [noise=true , address=145.79.227.37:4567 , piece_id=OGKEXSXQP6APGE7H4QPOD2KASV26CYD73MV2NBK36FDOSIB4CEWQ, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.067453531 (586h5m54.938536566s) [noise=true , address=145.79.227.42:4122 , piece_id=4IL4HJLEI3GKNDZBEID5AUH57K47CJCAONTLQSRUYBY5WOLTIDAA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.071091760 (586h5m54.942174805s) [noise=true , address=145.79.227.38:4979 , piece_id=XS6Y45OFSXOXZ4SG2TG65EE6I6NSFLOKDPED3BZZJL4QXCLOZJOA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.073714904 (586h5m54.944797939s) [noise=true , address=145.79.227.28:4725 , piece_id=QI34SC6QSLLW6ZU3KQ2TUJSAAATNGWKDNS6V74LVAWDDKXDNLICA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.080796028 (586h5m54.951879063s) [noise=true , address=145.79.227.8:4329 , piece_id=GD6UN7ECXA4KLELVBPMZUHG3EGXL2GUI27DLA5DDC7B6QBNFZ7EA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.256029446 (586h5m55.127112481s) [noise=true , address=45.91.116.8:4045 , piece_id=E5TGZB4TARZ4KR2DKH74G7XHVD7VENPGYBU2I52UFJV4JYMZKIDQ, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.325496572 (586h5m55.196579607s) [noise=true , address=77.90.31.16:4736 , piece_id=3MMRF5O6NQAPS4DA4VDFS722IPZKG57PWFPZUX5MT3B7PBYK5SMQ, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.407302413 (586h5m55.278385478s) [noise=true , address=45.91.116.10:4081 , piece_id=ZVDCI43IFZMDLG3VKWJSBBS57KA6OK5ZDPJU45NMV73AL3UWYIVA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.608342330 (586h5m55.479425365s) [noise=true , address=45.91.116.13:4154 , piece_id=V3CSZOKKW7MNUJ4TJDAHEVSF75JSGOIOMSOLLFURX2ITMZJ5UHRQ, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:56.071907243 (586h5m55.942990308s) [noise=true , address=45.91.116.11:4118 , piece_id=FM32Z47OXYSQXR3PVQOOX5764E4B4IHX2PATPNB3VG34WJZNTHQA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:56.071936198 (586h5m55.943019233s) [seg_pos={0 0}, upload=513796]: Segment reached optimal threshold of 13 pieces. Dec 30 01:34:56.072162689 (586h5m55.943245724s) [noise=true , address=45.91.116.9:4262 , piece_id=ND52YV7EQ2O4VQRONQO7VJ5VYHITSPFZCB5LBBKFQWEPT5K3OYEA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:34650->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:34650->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34650->45.91.116.9:4262: use of closed network connection Dec 30 01:34:56.072171927 (586h5m55.943254972s) [noise=true , address=45.91.116.9:4262 , piece_id=ND52YV7EQ2O4VQRONQO7VJ5VYHITSPFZCB5LBBKFQWEPT5K3OYEA, seg_pos={0 0}, upload=513796]: Long tail cancellation due to optimal threshold Dec 30 01:34:56.072253001 (586h5m55.943336037s) [noise=true , address=145.79.227.50:4373 , piece_id=FLWFANKIJXSRFE7LUZB5J74YCZEZ7QUHRGGPSNKF4VVYFUGYKMQA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:44632->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.50:4373: use of closed network connection Dec 30 01:34:56.072274623 (586h5m55.943357698s) [noise=true , address=145.79.227.50:4373 , piece_id=FLWFANKIJXSRFE7LUZB5J74YCZEZ7QUHRGGPSNKF4VVYFUGYKMQA, seg_pos={0 0}, upload=513796]: Long tail cancellation due to optimal threshold Dec 30 01:34:56.072487628 (586h5m55.943570664s) [noise=true , address=145.79.227.40:4050 , piece_id=2VFPC5NKF53RGPX47WG7MVTTD43PJECDJ42VJFNIED2P6GNVAEWA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 01:34:56.072497417 (586h5m55.943580452s) [noise=true , address=145.79.227.40:4050 , piece_id=2VFPC5NKF53RGPX47WG7MVTTD43PJECDJ42VJFNIED2P6GNVAEWA, seg_pos={0 0}, upload=513796]: Long tail cancellation due to optimal threshold Dec 30 01:34:56.072519840 (586h5m55.943602875s) [seg_pos={0 0}, upload=513796]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:34:56.072610101 (586h5m55.943693146s) [upload=513796]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:34:56.072617535 (586h5m55.94370058s) [upload=513796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:34:56.367842725 (586h5m56.23892577s) [upload=513796]: Done uploading segments... Dec 30 01:34:56.367872270 (586h5m56.238955315s) [upload=513796]: Upload finished. err: <nil> Duration: 6.03395345s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X.X...X....X.X.....X.................XX.....X.X......X.X] duration: 30.00358871s trace: https://jaeger.dev.storj.io/trace/2a09f9b1206496cd FAIL (2025-12-30 01:35:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: V3CSZOKKW7MNUJ4TJDAHEVSF75JSGOIOMSOLLFURX2ITMZJ5UHRQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: FM32Z47OXYSQXR3PVQOOX5764E4B4IHX2PATPNB3VG34WJZNTHQA) context deadline exceeded ======LOGS====== 2.68503247s) [upload=513796]: Uploading segments... Dec 30 01:34:52.814048624 (586h5m52.685131679s) [upload=513796]: Got next segment. Inline: false Dec 30 01:34:52.814113458 (586h5m52.685196493s) [upload=513796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:34:52.814120321 (586h5m52.685203376s) [upload=513796]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:34:53.060766941 (586h5m52.931849976s) [seg_pos={0 0}, upload=513796]: Begin upload segment... Dec 30 01:34:53.060817417 (586h5m52.931900452s) [seg_pos={0 0}, upload=513796]: Done begin upload segment. Dec 30 01:34:53.060861981 (586h5m52.931945016s) [noise=true , address=45.91.116.8:4045 , piece_id=E5TGZB4TARZ4KR2DKH74G7XHVD7VENPGYBU2I52UFJV4JYMZKIDQ, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.060899894 (586h5m52.931982929s) [noise=true , address=145.79.227.11:4729 , piece_id=26TCLWT3G4OOYQI55KNUFCQT46T5BTT5R43TBD7M2R7ZPUWV7H5Q, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.061019171 (586h5m52.932102206s) [noise=true , address=145.79.227.26:4653 , piece_id=23NKAZO4ND3WXKOJSR4SWHVTMUCWJ2FRKW2ZGLHKDCITRILHBXKA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.061576032 (586h5m52.932659077s) [noise=true , address=145.79.227.42:4122 , piece_id=4IL4HJLEI3GKNDZBEID5AUH57K47CJCAONTLQSRUYBY5WOLTIDAA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.063425213 (586h5m52.934508248s) [noise=true , address=145.79.227.38:4979 , piece_id=XS6Y45OFSXOXZ4SG2TG65EE6I6NSFLOKDPED3BZZJL4QXCLOZJOA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.063972325 (586h5m52.93505536s) [noise=true , address=145.79.227.37:4567 , piece_id=OGKEXSXQP6APGE7H4QPOD2KASV26CYD73MV2NBK36FDOSIB4CEWQ, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.064962710 (586h5m52.936045745s) [noise=true , address=45.91.116.9:4262 , piece_id=ND52YV7EQ2O4VQRONQO7VJ5VYHITSPFZCB5LBBKFQWEPT5K3OYEA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.065706987 (586h5m52.936790032s) [noise=true , address=77.90.31.16:4736 , piece_id=3MMRF5O6NQAPS4DA4VDFS722IPZKG57PWFPZUX5MT3B7PBYK5SMQ, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.066365220 (586h5m52.937448266s) [noise=true , address=145.79.227.28:4725 , piece_id=QI34SC6QSLLW6ZU3KQ2TUJSAAATNGWKDNS6V74LVAWDDKXDNLICA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.068040060 (586h5m52.939123095s) [noise=true , address=45.91.116.13:4154 , piece_id=V3CSZOKKW7MNUJ4TJDAHEVSF75JSGOIOMSOLLFURX2ITMZJ5UHRQ, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.070283672 (586h5m52.941366707s) [noise=true , address=45.91.116.11:4118 , piece_id=FM32Z47OXYSQXR3PVQOOX5764E4B4IHX2PATPNB3VG34WJZNTHQA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.072253904 (586h5m52.943336939s) [noise=true , address=45.91.116.10:4081 , piece_id=ZVDCI43IFZMDLG3VKWJSBBS57KA6OK5ZDPJU45NMV73AL3UWYIVA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.073051081 (586h5m52.944134126s) [noise=true , address=145.79.227.8:4329 , piece_id=GD6UN7ECXA4KLELVBPMZUHG3EGXL2GUI27DLA5DDC7B6QBNFZ7EA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.074116460 (586h5m52.945199496s) [noise=true , address=145.79.227.50:4373 , piece_id=FLWFANKIJXSRFE7LUZB5J74YCZEZ7QUHRGGPSNKF4VVYFUGYKMQA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.077440390 (586h5m52.948523435s) [noise=true , address=145.79.227.40:4050 , piece_id=2VFPC5NKF53RGPX47WG7MVTTD43PJECDJ42VJFNIED2P6GNVAEWA, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:53.078955004 (586h5m52.950038039s) [noise=true , address=145.79.227.45:4230 , piece_id=2SCHR3NSIZYERLDLZ7QE3TNTYIVODZ3SUHGRVX5OK4OQV3IB2QBQ, seg_pos={0 0}, upload=513796]: Uploading piece... Dec 30 01:34:54.763041389 (586h5m54.634124434s) [upload=513796]: Next returned nil segment Dec 30 01:34:54.763058281 (586h5m54.634141326s) [upload=513796]: Waiting for error group managing segments... Dec 30 01:34:55.046749328 (586h5m54.917832363s) [noise=true , address=145.79.227.11:4729 , piece_id=26TCLWT3G4OOYQI55KNUFCQT46T5BTT5R43TBD7M2R7ZPUWV7H5Q, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.050395240 (586h5m54.921478286s) [noise=true , address=145.79.227.45:4230 , piece_id=2SCHR3NSIZYERLDLZ7QE3TNTYIVODZ3SUHGRVX5OK4OQV3IB2QBQ, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.058869537 (586h5m54.929952572s) [noise=true , address=145.79.227.26:4653 , piece_id=23NKAZO4ND3WXKOJSR4SWHVTMUCWJ2FRKW2ZGLHKDCITRILHBXKA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.059713133 (586h5m54.930796178s) [noise=true , address=145.79.227.37:4567 , piece_id=OGKEXSXQP6APGE7H4QPOD2KASV26CYD73MV2NBK36FDOSIB4CEWQ, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.067453531 (586h5m54.938536566s) [noise=true , address=145.79.227.42:4122 , piece_id=4IL4HJLEI3GKNDZBEID5AUH57K47CJCAONTLQSRUYBY5WOLTIDAA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.071091760 (586h5m54.942174805s) [noise=true , address=145.79.227.38:4979 , piece_id=XS6Y45OFSXOXZ4SG2TG65EE6I6NSFLOKDPED3BZZJL4QXCLOZJOA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.073714904 (586h5m54.944797939s) [noise=true , address=145.79.227.28:4725 , piece_id=QI34SC6QSLLW6ZU3KQ2TUJSAAATNGWKDNS6V74LVAWDDKXDNLICA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.080796028 (586h5m54.951879063s) [noise=true , address=145.79.227.8:4329 , piece_id=GD6UN7ECXA4KLELVBPMZUHG3EGXL2GUI27DLA5DDC7B6QBNFZ7EA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.256029446 (586h5m55.127112481s) [noise=true , address=45.91.116.8:4045 , piece_id=E5TGZB4TARZ4KR2DKH74G7XHVD7VENPGYBU2I52UFJV4JYMZKIDQ, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.325496572 (586h5m55.196579607s) [noise=true , address=77.90.31.16:4736 , piece_id=3MMRF5O6NQAPS4DA4VDFS722IPZKG57PWFPZUX5MT3B7PBYK5SMQ, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.407302413 (586h5m55.278385478s) [noise=true , address=45.91.116.10:4081 , piece_id=ZVDCI43IFZMDLG3VKWJSBBS57KA6OK5ZDPJU45NMV73AL3UWYIVA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:55.608342330 (586h5m55.479425365s) [noise=true , address=45.91.116.13:4154 , piece_id=V3CSZOKKW7MNUJ4TJDAHEVSF75JSGOIOMSOLLFURX2ITMZJ5UHRQ, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:56.071907243 (586h5m55.942990308s) [noise=true , address=45.91.116.11:4118 , piece_id=FM32Z47OXYSQXR3PVQOOX5764E4B4IHX2PATPNB3VG34WJZNTHQA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: <nil> Dec 30 01:34:56.071936198 (586h5m55.943019233s) [seg_pos={0 0}, upload=513796]: Segment reached optimal threshold of 13 pieces. Dec 30 01:34:56.072162689 (586h5m55.943245724s) [noise=true , address=45.91.116.9:4262 , piece_id=ND52YV7EQ2O4VQRONQO7VJ5VYHITSPFZCB5LBBKFQWEPT5K3OYEA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:34650->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:34650->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34650->45.91.116.9:4262: use of closed network connection Dec 30 01:34:56.072171927 (586h5m55.943254972s) [noise=true , address=45.91.116.9:4262 , piece_id=ND52YV7EQ2O4VQRONQO7VJ5VYHITSPFZCB5LBBKFQWEPT5K3OYEA, seg_pos={0 0}, upload=513796]: Long tail cancellation due to optimal threshold Dec 30 01:34:56.072253001 (586h5m55.943336037s) [noise=true , address=145.79.227.50:4373 , piece_id=FLWFANKIJXSRFE7LUZB5J74YCZEZ7QUHRGGPSNKF4VVYFUGYKMQA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:44632->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:44632->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44632->145.79.227.50:4373: use of closed network connection Dec 30 01:34:56.072274623 (586h5m55.943357698s) [noise=true , address=145.79.227.50:4373 , piece_id=FLWFANKIJXSRFE7LUZB5J74YCZEZ7QUHRGGPSNKF4VVYFUGYKMQA, seg_pos={0 0}, upload=513796]: Long tail cancellation due to optimal threshold Dec 30 01:34:56.072487628 (586h5m55.943570664s) [noise=true , address=145.79.227.40:4050 , piece_id=2VFPC5NKF53RGPX47WG7MVTTD43PJECDJ42VJFNIED2P6GNVAEWA, seg_pos={0 0}, upload=513796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 01:34:56.072497417 (586h5m55.943580452s) [noise=true , address=145.79.227.40:4050 , piece_id=2VFPC5NKF53RGPX47WG7MVTTD43PJECDJ42VJFNIED2P6GNVAEWA, seg_pos={0 0}, upload=513796]: Long tail cancellation due to optimal threshold Dec 30 01:34:56.072519840 (586h5m55.943602875s) [seg_pos={0 0}, upload=513796]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:34:56.072610101 (586h5m55.943693146s) [upload=513796]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:34:56.072617535 (586h5m55.94370058s) [upload=513796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:34:56.367842725 (586h5m56.23892577s) [upload=513796]: Done uploading segments... Dec 30 01:34:56.367872270 (586h5m56.238955315s) [upload=513796]: Upload finished. err: <nil> Duration: 30.003616383s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 38.594710561s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:35:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 38.594738174s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 15.435851295s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:34:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 15.43588003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....X.X...X....X.X.....X.................XX.....X.X......X.] duration: 7.260025619s trace: https://jaeger.dev.storj.io/trace/314a28864b138124 FAIL (2025-12-30 01:34:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 01:33:33.870782663 (586h4m33.741865708s) [upload=513778]: Starting upload Dec 30 01:33:33.870865761 (586h4m33.741948806s) [upload=513778]: Done starting upload Dec 30 01:33:33.870913813 (586h4m33.741996858s) [upload=513778]: Uploading segments... Dec 30 01:33:33.871111980 (586h4m33.742195025s) [upload=513778]: Got next segment. Inline: false Dec 30 01:33:33.871154361 (586h4m33.742237406s) [upload=513778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:33:33.871160021 (586h4m33.742243056s) [upload=513778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:33:34.119853267 (586h4m33.990936322s) [seg_pos={0 0}, upload=513778]: Begin upload segment... Dec 30 01:33:34.119908643 (586h4m33.990991678s) [seg_pos={0 0}, upload=513778]: Done begin upload segment. Dec 30 01:33:34.119944701 (586h4m33.991027747s) [noise=true , address=77.90.31.15:4697 , piece_id=BPP34Y3UHJBVURL6MDRHGHIRD7PRM5LBB4GVWTZTPF5UJYKGNOIA, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.120012591 (586h4m33.991095636s) [noise=true , address=145.79.227.36:4519 , piece_id=ORJORK6XV7JJDD7OBAVOZDQY4SFGMBARAKSPVOUBKT6TMVKODVOA, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.120637922 (586h4m33.991720957s) [noise=true , address=145.79.227.50:4374 , piece_id=ZS6QML42LAYPU53VOZH2C22XZ7F7ZJDO3QWXZQDWAVBQ5FYRZ4MA, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.120647389 (586h4m33.991730434s) [noise=true , address=145.79.227.12:4767 , piece_id=6LNAKJC3AIPITXQJ3MTSLYU2P464ODHSRDIJTTMGEFJTN5YLMB7A, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.121007154 (586h4m33.992090189s) [noise=true , address=145.79.227.37:4566 , piece_id=7PZYPUZQE237CXEL4TDBVKQCND7PAAFFMUB4SW2JUV6JDVMCD5XA, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.121096835 (586h4m33.99217988s) [noise=true , address=145.79.227.33:4870 , piece_id=3EZX26S7PYC5Q6MTYGRSJCTT45ILIYRWNDL6YNSKONKEIKYFIRKA, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.121388029 (586h4m33.992471064s) [noise=true , address=145.79.227.28:4725 , piece_id=3EGFFKDMZVH6VRJOP3XSBTINIDHQXUF5H4EZ4J5ZINEC4KXK6E5A, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.121537093 (586h4m33.992620138s) [noise=true , address=145.79.227.39:4013 , piece_id=XOKMR45OD5Y6JIMIVBZYAEZAO7OPBDAW5NFC7PRO4KORNIP2WCJQ, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.121756051 (586h4m33.992839086s) [noise=true , address=145.79.227.19:4078 , piece_id=LRDFFDLI4UIOLDDDKHH3QKMCNCV6ZDN2QMOYF4GAQRXT5UOYIGJQ, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.122101779 (586h4m33.993184814s) [noise=true , address=145.79.227.16:4910 , piece_id=U3AKY5YLG7XB2UIDA3QBX4DGNPP7TIG6AK6RXDUQ7ADXUWRW2ELQ, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.122269918 (586h4m33.993352953s) [noise=true , address=145.79.227.6:4258 , piece_id=ALLPCN52KL5OUUDIRW3SKSQYGF4EA2HED2MPJO2LPYSXQV6GP2YQ, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.122534442 (586h4m33.993617487s) [noise=true , address=145.79.227.3:4150 , piece_id=BTWUF7PXE2JYOVLPOVDLG53YZJCKP266RXIPLYTHG4QUSLDPL43Q, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.122677755 (586h4m33.99376079s) [noise=true , address=145.79.227.17:4971 , piece_id=PZY4BLLUQ3RFUKX7S6UKPQ42BMNK4ISCBXMMW2QC4EBGZQU5LWLQ, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.123008065 (586h4m33.9940911s) [noise=true , address=145.79.227.14:4837 , piece_id=3SREG5J23RJ4B45I5ELL54BIMLHTJTKQQXLJVBB5MMDJYSVBI5XA, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.123060384 (586h4m33.994143419s) [noise=true , address=45.91.116.7:4011 , piece_id=CYXZN7E53UNM7R6SJY5CGASZPB4L2PZYKK2D6OUO5GWHPHQPID4A, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.123438023 (586h4m33.994521058s) [noise=true , address=45.91.116.14:4191 , piece_id=6BHFVITLU3NW327UT7RVMQNRIV74AX2GZO3MS6UWS3QKQXQBWWFQ, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:35.792589912 (586h4m35.663672947s) [upload=513778]: Next returned nil segment Dec 30 01:33:35.792602886 (586h4m35.663685932s) [upload=513778]: Waiting for error group managing segments... Dec 30 01:33:36.094678501 (586h4m35.965761556s) [noise=true , address=145.79.227.3:4150 , piece_id=BTWUF7PXE2JYOVLPOVDLG53YZJCKP266RXIPLYTHG4QUSLDPL43Q, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.094748234 (586h4m35.965831279s) [noise=true , address=145.79.227.19:4078 , piece_id=LRDFFDLI4UIOLDDDKHH3QKMCNCV6ZDN2QMOYF4GAQRXT5UOYIGJQ, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.107195254 (586h4m35.9782783s) [noise=true , address=145.79.227.16:4910 , piece_id=U3AKY5YLG7XB2UIDA3QBX4DGNPP7TIG6AK6RXDUQ7ADXUWRW2ELQ, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.108075360 (586h4m35.979158395s) [noise=true , address=145.79.227.14:4837 , piece_id=3SREG5J23RJ4B45I5ELL54BIMLHTJTKQQXLJVBB5MMDJYSVBI5XA, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.122604756 (586h4m35.993687801s) [noise=true , address=145.79.227.17:4971 , piece_id=PZY4BLLUQ3RFUKX7S6UKPQ42BMNK4ISCBXMMW2QC4EBGZQU5LWLQ, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.125214085 (586h4m35.99629713s) [noise=true , address=145.79.227.12:4767 , piece_id=6LNAKJC3AIPITXQJ3MTSLYU2P464ODHSRDIJTTMGEFJTN5YLMB7A, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.125436027 (586h4m35.996519062s) [noise=true , address=145.79.227.28:4725 , piece_id=3EGFFKDMZVH6VRJOP3XSBTINIDHQXUF5H4EZ4J5ZINEC4KXK6E5A, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.126256860 (586h4m35.997339896s) [noise=true , address=145.79.227.39:4013 , piece_id=XOKMR45OD5Y6JIMIVBZYAEZAO7OPBDAW5NFC7PRO4KORNIP2WCJQ, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.134094595 (586h4m36.00517764s) [noise=true , address=145.79.227.36:4519 , piece_id=ORJORK6XV7JJDD7OBAVOZDQY4SFGMBARAKSPVOUBKT6TMVKODVOA, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.138582069 (586h4m36.009665114s) [noise=true , address=145.79.227.6:4258 , piece_id=ALLPCN52KL5OUUDIRW3SKSQYGF4EA2HED2MPJO2LPYSXQV6GP2YQ, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.140817015 (586h4m36.01190005s) [noise=true , address=145.79.227.37:4566 , piece_id=7PZYPUZQE237CXEL4TDBVKQCND7PAAFFMUB4SW2JUV6JDVMCD5XA, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.436106427 (586h4m36.307189472s) [noise=true , address=45.91.116.7:4011 , piece_id=CYXZN7E53UNM7R6SJY5CGASZPB4L2PZYKK2D6OUO5GWHPHQPID4A, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:37.059753738 (586h4m36.930836773s) [noise=true , address=77.90.31.15:4697 , piece_id=BPP34Y3UHJBVURL6MDRHGHIRD7PRM5LBB4GVWTZTPF5UJYKGNOIA, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:37.059779718 (586h4m36.930862753s) [seg_pos={0 0}, upload=513778]: Segment reached optimal threshold of 13 pieces. Dec 30 01:33:37.060245084 (586h4m36.931328119s) [noise=true , address=45.91.116.14:4191 , piece_id=6BHFVITLU3NW327UT7RVMQNRIV74AX2GZO3MS6UWS3QKQXQBWWFQ, seg_pos={0 0}, upload=513778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 30 01:33:37.060254391 (586h4m36.931337426s) [noise=true , address=45.91.116.14:4191 , piece_id=6BHFVITLU3NW327UT7RVMQNRIV74AX2GZO3MS6UWS3QKQXQBWWFQ, seg_pos={0 0}, upload=513778]: Long tail cancellation due to optimal threshold Dec 30 01:33:37.062370311 (586h4m36.933453376s) [noise=true , address=145.79.227.50:4374 , piece_id=ZS6QML42LAYPU53VOZH2C22XZ7F7ZJDO3QWXZQDWAVBQ5FYRZ4MA, seg_pos={0 0}, upload=513778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: write tcp 188.166.225.170:54938->145.79.227.50:4374: use of closed network connection; write tcp 188.166.225.170:54938->145.79.227.50:4374: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54938->145.79.227.50:4374: use of closed network connection Dec 30 01:33:37.062386582 (586h4m36.933469617s) [noise=true , address=145.79.227.50:4374 , piece_id=ZS6QML42LAYPU53VOZH2C22XZ7F7ZJDO3QWXZQDWAVBQ5FYRZ4MA, seg_pos={0 0}, upload=513778]: Long tail cancellation due to optimal threshold Dec 30 01:33:37.062602663 (586h4m36.933685718s) [noise=true , address=145.79.227.33:4870 , piece_id=3EZX26S7PYC5Q6MTYGRSJCTT45ILIYRWNDL6YNSKONKEIKYFIRKA, seg_pos={0 0}, upload=513778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 01:33:37.062610989 (586h4m36.933694044s) [noise=true , address=145.79.227.33:4870 , piece_id=3EZX26S7PYC5Q6MTYGRSJCTT45ILIYRWNDL6YNSKONKEIKYFIRKA, seg_pos={0 0}, upload=513778]: Long tail cancellation due to optimal threshold Dec 30 01:33:37.062630316 (586h4m36.933713361s) [seg_pos={0 0}, upload=513778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:33:37.062735607 (586h4m36.933818642s) [upload=513778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:33:37.062741428 (586h4m36.933824473s) [upload=513778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:33:37.358179553 (586h4m37.229262598s) [upload=513778]: Done uploading segments... Dec 30 01:33:37.358208748 (586h4m37.229291783s) [upload=513778]: Upload finished. err: <nil> Duration: 7.260043433s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 30.210758329s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:34:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 30.210782466s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......X.X...X....X.X.....X.................XX.....X.X......X] duration: 30.003670312s trace: https://jaeger.dev.storj.io/trace/314a28864b138124 FAIL (2025-12-30 01:34:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 01:33:33.870782663 (586h4m33.741865708s) [upload=513778]: Starting upload Dec 30 01:33:33.870865761 (586h4m33.741948806s) [upload=513778]: Done starting upload Dec 30 01:33:33.870913813 (586h4m33.741996858s) [upload=513778]: Uploading segments... Dec 30 01:33:33.871111980 (586h4m33.742195025s) [upload=513778]: Got next segment. Inline: false Dec 30 01:33:33.871154361 (586h4m33.742237406s) [upload=513778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:33:33.871160021 (586h4m33.742243056s) [upload=513778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:33:34.119853267 (586h4m33.990936322s) [seg_pos={0 0}, upload=513778]: Begin upload segment... Dec 30 01:33:34.119908643 (586h4m33.990991678s) [seg_pos={0 0}, upload=513778]: Done begin upload segment. Dec 30 01:33:34.119944701 (586h4m33.991027747s) [noise=true , address=77.90.31.15:4697 , piece_id=BPP34Y3UHJBVURL6MDRHGHIRD7PRM5LBB4GVWTZTPF5UJYKGNOIA, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.120012591 (586h4m33.991095636s) [noise=true , address=145.79.227.36:4519 , piece_id=ORJORK6XV7JJDD7OBAVOZDQY4SFGMBARAKSPVOUBKT6TMVKODVOA, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.120637922 (586h4m33.991720957s) [noise=true , address=145.79.227.50:4374 , piece_id=ZS6QML42LAYPU53VOZH2C22XZ7F7ZJDO3QWXZQDWAVBQ5FYRZ4MA, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.120647389 (586h4m33.991730434s) [noise=true , address=145.79.227.12:4767 , piece_id=6LNAKJC3AIPITXQJ3MTSLYU2P464ODHSRDIJTTMGEFJTN5YLMB7A, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.121007154 (586h4m33.992090189s) [noise=true , address=145.79.227.37:4566 , piece_id=7PZYPUZQE237CXEL4TDBVKQCND7PAAFFMUB4SW2JUV6JDVMCD5XA, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.121096835 (586h4m33.99217988s) [noise=true , address=145.79.227.33:4870 , piece_id=3EZX26S7PYC5Q6MTYGRSJCTT45ILIYRWNDL6YNSKONKEIKYFIRKA, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.121388029 (586h4m33.992471064s) [noise=true , address=145.79.227.28:4725 , piece_id=3EGFFKDMZVH6VRJOP3XSBTINIDHQXUF5H4EZ4J5ZINEC4KXK6E5A, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.121537093 (586h4m33.992620138s) [noise=true , address=145.79.227.39:4013 , piece_id=XOKMR45OD5Y6JIMIVBZYAEZAO7OPBDAW5NFC7PRO4KORNIP2WCJQ, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.121756051 (586h4m33.992839086s) [noise=true , address=145.79.227.19:4078 , piece_id=LRDFFDLI4UIOLDDDKHH3QKMCNCV6ZDN2QMOYF4GAQRXT5UOYIGJQ, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.122101779 (586h4m33.993184814s) [noise=true , address=145.79.227.16:4910 , piece_id=U3AKY5YLG7XB2UIDA3QBX4DGNPP7TIG6AK6RXDUQ7ADXUWRW2ELQ, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.122269918 (586h4m33.993352953s) [noise=true , address=145.79.227.6:4258 , piece_id=ALLPCN52KL5OUUDIRW3SKSQYGF4EA2HED2MPJO2LPYSXQV6GP2YQ, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.122534442 (586h4m33.993617487s) [noise=true , address=145.79.227.3:4150 , piece_id=BTWUF7PXE2JYOVLPOVDLG53YZJCKP266RXIPLYTHG4QUSLDPL43Q, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.122677755 (586h4m33.99376079s) [noise=true , address=145.79.227.17:4971 , piece_id=PZY4BLLUQ3RFUKX7S6UKPQ42BMNK4ISCBXMMW2QC4EBGZQU5LWLQ, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.123008065 (586h4m33.9940911s) [noise=true , address=145.79.227.14:4837 , piece_id=3SREG5J23RJ4B45I5ELL54BIMLHTJTKQQXLJVBB5MMDJYSVBI5XA, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.123060384 (586h4m33.994143419s) [noise=true , address=45.91.116.7:4011 , piece_id=CYXZN7E53UNM7R6SJY5CGASZPB4L2PZYKK2D6OUO5GWHPHQPID4A, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:34.123438023 (586h4m33.994521058s) [noise=true , address=45.91.116.14:4191 , piece_id=6BHFVITLU3NW327UT7RVMQNRIV74AX2GZO3MS6UWS3QKQXQBWWFQ, seg_pos={0 0}, upload=513778]: Uploading piece... Dec 30 01:33:35.792589912 (586h4m35.663672947s) [upload=513778]: Next returned nil segment Dec 30 01:33:35.792602886 (586h4m35.663685932s) [upload=513778]: Waiting for error group managing segments... Dec 30 01:33:36.094678501 (586h4m35.965761556s) [noise=true , address=145.79.227.3:4150 , piece_id=BTWUF7PXE2JYOVLPOVDLG53YZJCKP266RXIPLYTHG4QUSLDPL43Q, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.094748234 (586h4m35.965831279s) [noise=true , address=145.79.227.19:4078 , piece_id=LRDFFDLI4UIOLDDDKHH3QKMCNCV6ZDN2QMOYF4GAQRXT5UOYIGJQ, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.107195254 (586h4m35.9782783s) [noise=true , address=145.79.227.16:4910 , piece_id=U3AKY5YLG7XB2UIDA3QBX4DGNPP7TIG6AK6RXDUQ7ADXUWRW2ELQ, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.108075360 (586h4m35.979158395s) [noise=true , address=145.79.227.14:4837 , piece_id=3SREG5J23RJ4B45I5ELL54BIMLHTJTKQQXLJVBB5MMDJYSVBI5XA, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.122604756 (586h4m35.993687801s) [noise=true , address=145.79.227.17:4971 , piece_id=PZY4BLLUQ3RFUKX7S6UKPQ42BMNK4ISCBXMMW2QC4EBGZQU5LWLQ, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.125214085 (586h4m35.99629713s) [noise=true , address=145.79.227.12:4767 , piece_id=6LNAKJC3AIPITXQJ3MTSLYU2P464ODHSRDIJTTMGEFJTN5YLMB7A, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.125436027 (586h4m35.996519062s) [noise=true , address=145.79.227.28:4725 , piece_id=3EGFFKDMZVH6VRJOP3XSBTINIDHQXUF5H4EZ4J5ZINEC4KXK6E5A, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.126256860 (586h4m35.997339896s) [noise=true , address=145.79.227.39:4013 , piece_id=XOKMR45OD5Y6JIMIVBZYAEZAO7OPBDAW5NFC7PRO4KORNIP2WCJQ, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.134094595 (586h4m36.00517764s) [noise=true , address=145.79.227.36:4519 , piece_id=ORJORK6XV7JJDD7OBAVOZDQY4SFGMBARAKSPVOUBKT6TMVKODVOA, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.138582069 (586h4m36.009665114s) [noise=true , address=145.79.227.6:4258 , piece_id=ALLPCN52KL5OUUDIRW3SKSQYGF4EA2HED2MPJO2LPYSXQV6GP2YQ, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.140817015 (586h4m36.01190005s) [noise=true , address=145.79.227.37:4566 , piece_id=7PZYPUZQE237CXEL4TDBVKQCND7PAAFFMUB4SW2JUV6JDVMCD5XA, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:36.436106427 (586h4m36.307189472s) [noise=true , address=45.91.116.7:4011 , piece_id=CYXZN7E53UNM7R6SJY5CGASZPB4L2PZYKK2D6OUO5GWHPHQPID4A, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:37.059753738 (586h4m36.930836773s) [noise=true , address=77.90.31.15:4697 , piece_id=BPP34Y3UHJBVURL6MDRHGHIRD7PRM5LBB4GVWTZTPF5UJYKGNOIA, seg_pos={0 0}, upload=513778]: Done uploading piece. err: <nil> Dec 30 01:33:37.059779718 (586h4m36.930862753s) [seg_pos={0 0}, upload=513778]: Segment reached optimal threshold of 13 pieces. Dec 30 01:33:37.060245084 (586h4m36.931328119s) [noise=true , address=45.91.116.14:4191 , piece_id=6BHFVITLU3NW327UT7RVMQNRIV74AX2GZO3MS6UWS3QKQXQBWWFQ, seg_pos={0 0}, upload=513778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 30 01:33:37.060254391 (586h4m36.931337426s) [noise=true , address=45.91.116.14:4191 , piece_id=6BHFVITLU3NW327UT7RVMQNRIV74AX2GZO3MS6UWS3QKQXQBWWFQ, seg_pos={0 0}, upload=513778]: Long tail cancellation due to optimal threshold Dec 30 01:33:37.062370311 (586h4m36.933453376s) [noise=true , address=145.79.227.50:4374 , piece_id=ZS6QML42LAYPU53VOZH2C22XZ7F7ZJDO3QWXZQDWAVBQ5FYRZ4MA, seg_pos={0 0}, upload=513778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: write tcp 188.166.225.170:54938->145.79.227.50:4374: use of closed network connection; write tcp 188.166.225.170:54938->145.79.227.50:4374: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54938->145.79.227.50:4374: use of closed network connection Dec 30 01:33:37.062386582 (586h4m36.933469617s) [noise=true , address=145.79.227.50:4374 , piece_id=ZS6QML42LAYPU53VOZH2C22XZ7F7ZJDO3QWXZQDWAVBQ5FYRZ4MA, seg_pos={0 0}, upload=513778]: Long tail cancellation due to optimal threshold Dec 30 01:33:37.062602663 (586h4m36.933685718s) [noise=true , address=145.79.227.33:4870 , piece_id=3EZX26S7PYC5Q6MTYGRSJCTT45ILIYRWNDL6YNSKONKEIKYFIRKA, seg_pos={0 0}, upload=513778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 01:33:37.062610989 (586h4m36.933694044s) [noise=true , address=145.79.227.33:4870 , piece_id=3EZX26S7PYC5Q6MTYGRSJCTT45ILIYRWNDL6YNSKONKEIKYFIRKA, seg_pos={0 0}, upload=513778]: Long tail cancellation due to optimal threshold Dec 30 01:33:37.062630316 (586h4m36.933713361s) [seg_pos={0 0}, upload=513778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:33:37.062735607 (586h4m36.933818642s) [upload=513778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:33:37.062741428 (586h4m36.933824473s) [upload=513778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:33:37.358179553 (586h4m37.229262598s) [upload=513778]: Done uploading segments... Dec 30 01:33:37.358208748 (586h4m37.229291783s) [upload=513778]: Upload finished. err: <nil> Duration: 30.003697504s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 15.358097982s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:33:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 15.358121257s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 15.09791039s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:32:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 15.097936359s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X......X.X...X....X.X.....X.................XX.....X.X......] duration: 7.5330398s trace: https://jaeger.dev.storj.io/trace/22d6ffcc1c4b6e7f FAIL (2025-12-30 01:32:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27:43.593396426 (585h58m43.464479461s) [upload=513690]: Starting upload Dec 30 01:27:43.593455890 (585h58m43.464538925s) [upload=513690]: Done starting upload Dec 30 01:27:43.593640080 (585h58m43.464723115s) [upload=513690]: Uploading segments... Dec 30 01:27:43.593684705 (585h58m43.46476775s) [upload=513690]: Got next segment. Inline: false Dec 30 01:27:43.593720193 (585h58m43.464803228s) [upload=513690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:27:43.593724761 (585h58m43.464807796s) [upload=513690]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:27:43.841048388 (585h58m43.712131433s) [seg_pos={0 0}, upload=513690]: Begin upload segment... Dec 30 01:27:43.841105667 (585h58m43.712188712s) [seg_pos={0 0}, upload=513690]: Done begin upload segment. Dec 30 01:27:43.841135033 (585h58m43.712218068s) [noise=true , address=45.91.116.13:4154 , piece_id=6XSDYKFCMI3JLTGYTTWCIS3WHXBQYVUQD5IAUGBV3RLHBVIAXWZA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.846794124 (585h58m43.717877169s) [noise=true , address=145.79.227.34:4907 , piece_id=BWTXN7ILYAM3YBF3PKLSUXPBZS62C3T5JITT5RV2UF27RS4ANJXQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.847687495 (585h58m43.71877054s) [noise=true , address=145.79.227.36:4519 , piece_id=GWTCGD2AICM6HVVE3U4LB6W3GTEO765R45KTTL3V3Z7VMZNDP27Q, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.848019897 (585h58m43.719102932s) [noise=true , address=145.79.227.37:4567 , piece_id=ALHGJOJQR2QQ2QWC5EN347MBWAQGPHL3FEIH6ZER4Q6NX6FAW34A, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.848362470 (585h58m43.719445505s) [noise=true , address=45.91.116.12:4228 , piece_id=H2ILHVVJQ6LAD67OYQZJFSWT4I6A7PZRKEG7P4BJABNRW5NP7JDA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.848703859 (585h58m43.719786904s) [noise=true , address=45.91.116.7:4010 , piece_id=BY2LCGCQF4BFEO5QC43MP4EBOUIHPBEYPBP6R5EEN3PR5HHZXYRQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.849050659 (585h58m43.720133694s) [noise=true , address=145.79.227.26:4654 , piece_id=2JY3GTBPFN2T2IJD3WIBXS76RMB3VXMXMFZJXIBWM6V3XO773XMQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.849388282 (585h58m43.720471317s) [noise=true , address=145.79.227.38:4979 , piece_id=JZAGRA6TX4BOGAL5J2K5NLV6RZ2TQZXJGSPGBEUHQYV6UYEQH4DQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.849727838 (585h58m43.720810883s) [noise=true , address=77.90.31.11:4555 , piece_id=P4JTRDJURCVZAFOKX73LONMF3FLVBEMP6EN7NHAATPJ6KJ3IBDQA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850002681 (585h58m43.721085736s) [noise=true , address=145.79.227.11:4732 , piece_id=KHMB5D4233GN7TFWG4G7XSLG3NYGU4CGLEO2UMN5RMAG3VFTQT4A, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850191261 (585h58m43.721274306s) [noise=true , address=77.90.31.13:4627 , piece_id=6AUPOIIVG5REWLT3DJLLFYSHJLI46JST4R4FMULZRIIWRHPOFB2Q, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850634494 (585h58m43.721717539s) [noise=true , address=145.79.227.5:4223 , piece_id=GQDKDXGFGI5WMZ3N764ZDFBZL5DJPSMNNXS4T4T57ANMIOTYUMRQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850788557 (585h58m43.721871592s) [noise=true , address=145.79.227.45:4230 , piece_id=X7NLAU6OYA2OC52J5KGFXFNDLGC4O4GTJRBX2RBFDJQENW6EST3A, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.851327273 (585h58m43.722410308s) [noise=true , address=145.79.227.47:4266 , piece_id=5QMHAY2T3EDDEWAKLRYLN5ERGKVLIL5BUEDDGKNIB5TUICAW7AWA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.851897969 (585h58m43.722981004s) [noise=true , address=145.79.227.14:4837 , piece_id=UT7J3YUHHB4RGIRB5EP2PJHPRGAJTQ3ITHQ43PPK5OXTCJKTUVOA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.852473204 (585h58m43.723556249s) [noise=true , address=145.79.227.52:4447 , piece_id=R5VK67OTLXAPTNN5HH2D2555SL55AAMSSWFBHK5SSKAAXCCK3AJA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:45.490164438 (585h58m45.361247483s) [upload=513690]: Next returned nil segment Dec 30 01:27:45.490198152 (585h58m45.361281197s) [upload=513690]: Waiting for error group managing segments... Dec 30 01:27:45.830727878 (585h58m45.701810933s) [noise=true , address=145.79.227.14:4837 , piece_id=UT7J3YUHHB4RGIRB5EP2PJHPRGAJTQ3ITHQ43PPK5OXTCJKTUVOA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.833647284 (585h58m45.704730319s) [noise=true , address=145.79.227.45:4230 , piece_id=X7NLAU6OYA2OC52J5KGFXFNDLGC4O4GTJRBX2RBFDJQENW6EST3A, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.852769451 (585h58m45.723852486s) [noise=true , address=145.79.227.5:4223 , piece_id=GQDKDXGFGI5WMZ3N764ZDFBZL5DJPSMNNXS4T4T57ANMIOTYUMRQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.852980513 (585h58m45.724063548s) [noise=true , address=145.79.227.38:4979 , piece_id=JZAGRA6TX4BOGAL5J2K5NLV6RZ2TQZXJGSPGBEUHQYV6UYEQH4DQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.856242251 (585h58m45.727325317s) [noise=true , address=145.79.227.47:4266 , piece_id=5QMHAY2T3EDDEWAKLRYLN5ERGKVLIL5BUEDDGKNIB5TUICAW7AWA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.858762338 (585h58m45.729845373s) [noise=true , address=145.79.227.52:4447 , piece_id=R5VK67OTLXAPTNN5HH2D2555SL55AAMSSWFBHK5SSKAAXCCK3AJA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.859500203 (585h58m45.730583238s) [noise=true , address=145.79.227.36:4519 , piece_id=GWTCGD2AICM6HVVE3U4LB6W3GTEO765R45KTTL3V3Z7VMZNDP27Q, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.859874234 (585h58m45.730957269s) [noise=true , address=145.79.227.26:4654 , piece_id=2JY3GTBPFN2T2IJD3WIBXS76RMB3VXMXMFZJXIBWM6V3XO773XMQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.860342475 (585h58m45.73142551s) [noise=true , address=145.79.227.11:4732 , piece_id=KHMB5D4233GN7TFWG4G7XSLG3NYGU4CGLEO2UMN5RMAG3VFTQT4A, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.126796667 (585h58m45.997879702s) [noise=true , address=77.90.31.11:4555 , piece_id=P4JTRDJURCVZAFOKX73LONMF3FLVBEMP6EN7NHAATPJ6KJ3IBDQA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.180715827 (585h58m46.051798862s) [noise=true , address=45.91.116.7:4010 , piece_id=BY2LCGCQF4BFEO5QC43MP4EBOUIHPBEYPBP6R5EEN3PR5HHZXYRQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.668961231 (585h58m46.540044266s) [noise=true , address=77.90.31.13:4627 , piece_id=6AUPOIIVG5REWLT3DJLLFYSHJLI46JST4R4FMULZRIIWRHPOFB2Q, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.933356712 (585h58m46.804439747s) [noise=true , address=45.91.116.13:4154 , piece_id=6XSDYKFCMI3JLTGYTTWCIS3WHXBQYVUQD5IAUGBV3RLHBVIAXWZA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.933373945 (585h58m46.80445698s) [seg_pos={0 0}, upload=513690]: Segment reached optimal threshold of 13 pieces. Dec 30 01:27:46.933540653 (585h58m46.804623688s) [noise=true , address=45.91.116.12:4228 , piece_id=H2ILHVVJQ6LAD67OYQZJFSWT4I6A7PZRKEG7P4BJABNRW5NP7JDA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 30 01:27:46.933547756 (585h58m46.804630791s) [noise=true , address=45.91.116.12:4228 , piece_id=H2ILHVVJQ6LAD67OYQZJFSWT4I6A7PZRKEG7P4BJABNRW5NP7JDA, seg_pos={0 0}, upload=513690]: Long tail cancellation due to optimal threshold Dec 30 01:27:46.933629041 (585h58m46.804712086s) [noise=true , address=145.79.227.37:4567 , piece_id=ALHGJOJQR2QQ2QWC5EN347MBWAQGPHL3FEIH6ZER4Q6NX6FAW34A, seg_pos={0 0}, upload=513690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:52334->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:52334->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52334->145.79.227.37:4567: use of closed network connection Dec 30 01:27:46.933636274 (585h58m46.804719309s) [noise=true , address=145.79.227.37:4567 , piece_id=ALHGJOJQR2QQ2QWC5EN347MBWAQGPHL3FEIH6ZER4Q6NX6FAW34A, seg_pos={0 0}, upload=513690]: Long tail cancellation due to optimal threshold Dec 30 01:27:46.933768126 (585h58m46.804851161s) [noise=true , address=145.79.227.34:4907 , piece_id=BWTXN7ILYAM3YBF3PKLSUXPBZS62C3T5JITT5RV2UF27RS4ANJXQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 01:27:46.933774417 (585h58m46.804857462s) [noise=true , address=145.79.227.34:4907 , piece_id=BWTXN7ILYAM3YBF3PKLSUXPBZS62C3T5JITT5RV2UF27RS4ANJXQ, seg_pos={0 0}, upload=513690]: Long tail cancellation due to optimal threshold Dec 30 01:27:46.933788274 (585h58m46.804871309s) [seg_pos={0 0}, upload=513690]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:27:46.933885048 (585h58m46.804968113s) [upload=513690]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:27:46.933889797 (585h58m46.804972832s) [upload=513690]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:27:47.227746916 (585h58m47.098829951s) [upload=513690]: Done uploading segments... Dec 30 01:27:47.227765671 (585h58m47.098848706s) [upload=513690]: Upload finished. err: <nil> Duration: 7.533089206s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 35.51265582s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:32:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 35.512684104s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X......X.X...X....X.X.....X.................XX.....X.X.....] duration: 6.162690456s trace: https://jaeger.dev.storj.io/trace/22d6ffcc1c4b6e7f FAIL (2025-12-30 01:32:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27:43.593396426 (585h58m43.464479461s) [upload=513690]: Starting upload Dec 30 01:27:43.593455890 (585h58m43.464538925s) [upload=513690]: Done starting upload Dec 30 01:27:43.593640080 (585h58m43.464723115s) [upload=513690]: Uploading segments... Dec 30 01:27:43.593684705 (585h58m43.46476775s) [upload=513690]: Got next segment. Inline: false Dec 30 01:27:43.593720193 (585h58m43.464803228s) [upload=513690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:27:43.593724761 (585h58m43.464807796s) [upload=513690]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:27:43.841048388 (585h58m43.712131433s) [seg_pos={0 0}, upload=513690]: Begin upload segment... Dec 30 01:27:43.841105667 (585h58m43.712188712s) [seg_pos={0 0}, upload=513690]: Done begin upload segment. Dec 30 01:27:43.841135033 (585h58m43.712218068s) [noise=true , address=45.91.116.13:4154 , piece_id=6XSDYKFCMI3JLTGYTTWCIS3WHXBQYVUQD5IAUGBV3RLHBVIAXWZA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.846794124 (585h58m43.717877169s) [noise=true , address=145.79.227.34:4907 , piece_id=BWTXN7ILYAM3YBF3PKLSUXPBZS62C3T5JITT5RV2UF27RS4ANJXQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.847687495 (585h58m43.71877054s) [noise=true , address=145.79.227.36:4519 , piece_id=GWTCGD2AICM6HVVE3U4LB6W3GTEO765R45KTTL3V3Z7VMZNDP27Q, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.848019897 (585h58m43.719102932s) [noise=true , address=145.79.227.37:4567 , piece_id=ALHGJOJQR2QQ2QWC5EN347MBWAQGPHL3FEIH6ZER4Q6NX6FAW34A, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.848362470 (585h58m43.719445505s) [noise=true , address=45.91.116.12:4228 , piece_id=H2ILHVVJQ6LAD67OYQZJFSWT4I6A7PZRKEG7P4BJABNRW5NP7JDA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.848703859 (585h58m43.719786904s) [noise=true , address=45.91.116.7:4010 , piece_id=BY2LCGCQF4BFEO5QC43MP4EBOUIHPBEYPBP6R5EEN3PR5HHZXYRQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.849050659 (585h58m43.720133694s) [noise=true , address=145.79.227.26:4654 , piece_id=2JY3GTBPFN2T2IJD3WIBXS76RMB3VXMXMFZJXIBWM6V3XO773XMQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.849388282 (585h58m43.720471317s) [noise=true , address=145.79.227.38:4979 , piece_id=JZAGRA6TX4BOGAL5J2K5NLV6RZ2TQZXJGSPGBEUHQYV6UYEQH4DQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.849727838 (585h58m43.720810883s) [noise=true , address=77.90.31.11:4555 , piece_id=P4JTRDJURCVZAFOKX73LONMF3FLVBEMP6EN7NHAATPJ6KJ3IBDQA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850002681 (585h58m43.721085736s) [noise=true , address=145.79.227.11:4732 , piece_id=KHMB5D4233GN7TFWG4G7XSLG3NYGU4CGLEO2UMN5RMAG3VFTQT4A, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850191261 (585h58m43.721274306s) [noise=true , address=77.90.31.13:4627 , piece_id=6AUPOIIVG5REWLT3DJLLFYSHJLI46JST4R4FMULZRIIWRHPOFB2Q, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850634494 (585h58m43.721717539s) [noise=true , address=145.79.227.5:4223 , piece_id=GQDKDXGFGI5WMZ3N764ZDFBZL5DJPSMNNXS4T4T57ANMIOTYUMRQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850788557 (585h58m43.721871592s) [noise=true , address=145.79.227.45:4230 , piece_id=X7NLAU6OYA2OC52J5KGFXFNDLGC4O4GTJRBX2RBFDJQENW6EST3A, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.851327273 (585h58m43.722410308s) [noise=true , address=145.79.227.47:4266 , piece_id=5QMHAY2T3EDDEWAKLRYLN5ERGKVLIL5BUEDDGKNIB5TUICAW7AWA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.851897969 (585h58m43.722981004s) [noise=true , address=145.79.227.14:4837 , piece_id=UT7J3YUHHB4RGIRB5EP2PJHPRGAJTQ3ITHQ43PPK5OXTCJKTUVOA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.852473204 (585h58m43.723556249s) [noise=true , address=145.79.227.52:4447 , piece_id=R5VK67OTLXAPTNN5HH2D2555SL55AAMSSWFBHK5SSKAAXCCK3AJA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:45.490164438 (585h58m45.361247483s) [upload=513690]: Next returned nil segment Dec 30 01:27:45.490198152 (585h58m45.361281197s) [upload=513690]: Waiting for error group managing segments... Dec 30 01:27:45.830727878 (585h58m45.701810933s) [noise=true , address=145.79.227.14:4837 , piece_id=UT7J3YUHHB4RGIRB5EP2PJHPRGAJTQ3ITHQ43PPK5OXTCJKTUVOA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.833647284 (585h58m45.704730319s) [noise=true , address=145.79.227.45:4230 , piece_id=X7NLAU6OYA2OC52J5KGFXFNDLGC4O4GTJRBX2RBFDJQENW6EST3A, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.852769451 (585h58m45.723852486s) [noise=true , address=145.79.227.5:4223 , piece_id=GQDKDXGFGI5WMZ3N764ZDFBZL5DJPSMNNXS4T4T57ANMIOTYUMRQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.852980513 (585h58m45.724063548s) [noise=true , address=145.79.227.38:4979 , piece_id=JZAGRA6TX4BOGAL5J2K5NLV6RZ2TQZXJGSPGBEUHQYV6UYEQH4DQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.856242251 (585h58m45.727325317s) [noise=true , address=145.79.227.47:4266 , piece_id=5QMHAY2T3EDDEWAKLRYLN5ERGKVLIL5BUEDDGKNIB5TUICAW7AWA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.858762338 (585h58m45.729845373s) [noise=true , address=145.79.227.52:4447 , piece_id=R5VK67OTLXAPTNN5HH2D2555SL55AAMSSWFBHK5SSKAAXCCK3AJA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.859500203 (585h58m45.730583238s) [noise=true , address=145.79.227.36:4519 , piece_id=GWTCGD2AICM6HVVE3U4LB6W3GTEO765R45KTTL3V3Z7VMZNDP27Q, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.859874234 (585h58m45.730957269s) [noise=true , address=145.79.227.26:4654 , piece_id=2JY3GTBPFN2T2IJD3WIBXS76RMB3VXMXMFZJXIBWM6V3XO773XMQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.860342475 (585h58m45.73142551s) [noise=true , address=145.79.227.11:4732 , piece_id=KHMB5D4233GN7TFWG4G7XSLG3NYGU4CGLEO2UMN5RMAG3VFTQT4A, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.126796667 (585h58m45.997879702s) [noise=true , address=77.90.31.11:4555 , piece_id=P4JTRDJURCVZAFOKX73LONMF3FLVBEMP6EN7NHAATPJ6KJ3IBDQA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.180715827 (585h58m46.051798862s) [noise=true , address=45.91.116.7:4010 , piece_id=BY2LCGCQF4BFEO5QC43MP4EBOUIHPBEYPBP6R5EEN3PR5HHZXYRQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.668961231 (585h58m46.540044266s) [noise=true , address=77.90.31.13:4627 , piece_id=6AUPOIIVG5REWLT3DJLLFYSHJLI46JST4R4FMULZRIIWRHPOFB2Q, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.933356712 (585h58m46.804439747s) [noise=true , address=45.91.116.13:4154 , piece_id=6XSDYKFCMI3JLTGYTTWCIS3WHXBQYVUQD5IAUGBV3RLHBVIAXWZA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.933373945 (585h58m46.80445698s) [seg_pos={0 0}, upload=513690]: Segment reached optimal threshold of 13 pieces. Dec 30 01:27:46.933540653 (585h58m46.804623688s) [noise=true , address=45.91.116.12:4228 , piece_id=H2ILHVVJQ6LAD67OYQZJFSWT4I6A7PZRKEG7P4BJABNRW5NP7JDA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 30 01:27:46.933547756 (585h58m46.804630791s) [noise=true , address=45.91.116.12:4228 , piece_id=H2ILHVVJQ6LAD67OYQZJFSWT4I6A7PZRKEG7P4BJABNRW5NP7JDA, seg_pos={0 0}, upload=513690]: Long tail cancellation due to optimal threshold Dec 30 01:27:46.933629041 (585h58m46.804712086s) [noise=true , address=145.79.227.37:4567 , piece_id=ALHGJOJQR2QQ2QWC5EN347MBWAQGPHL3FEIH6ZER4Q6NX6FAW34A, seg_pos={0 0}, upload=513690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:52334->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:52334->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52334->145.79.227.37:4567: use of closed network connection Dec 30 01:27:46.933636274 (585h58m46.804719309s) [noise=true , address=145.79.227.37:4567 , piece_id=ALHGJOJQR2QQ2QWC5EN347MBWAQGPHL3FEIH6ZER4Q6NX6FAW34A, seg_pos={0 0}, upload=513690]: Long tail cancellation due to optimal threshold Dec 30 01:27:46.933768126 (585h58m46.804851161s) [noise=true , address=145.79.227.34:4907 , piece_id=BWTXN7ILYAM3YBF3PKLSUXPBZS62C3T5JITT5RV2UF27RS4ANJXQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 01:27:46.933774417 (585h58m46.804857462s) [noise=true , address=145.79.227.34:4907 , piece_id=BWTXN7ILYAM3YBF3PKLSUXPBZS62C3T5JITT5RV2UF27RS4ANJXQ, seg_pos={0 0}, upload=513690]: Long tail cancellation due to optimal threshold Dec 30 01:27:46.933788274 (585h58m46.804871309s) [seg_pos={0 0}, upload=513690]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:27:46.933885048 (585h58m46.804968113s) [upload=513690]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:27:46.933889797 (585h58m46.804972832s) [upload=513690]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:27:47.227746916 (585h58m47.098829951s) [upload=513690]: Done uploading segments... Dec 30 01:27:47.227765671 (585h58m47.098848706s) [upload=513690]: Upload finished. err: <nil> Duration: 6.162738738s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 34.681482819s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:31:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 34.681506444s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X......X.X...X....X.X.....X.................XX.....X.X....] duration: 5.726964438s trace: https://jaeger.dev.storj.io/trace/22d6ffcc1c4b6e7f FAIL (2025-12-30 01:31:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27:43.593396426 (585h58m43.464479461s) [upload=513690]: Starting upload Dec 30 01:27:43.593455890 (585h58m43.464538925s) [upload=513690]: Done starting upload Dec 30 01:27:43.593640080 (585h58m43.464723115s) [upload=513690]: Uploading segments... Dec 30 01:27:43.593684705 (585h58m43.46476775s) [upload=513690]: Got next segment. Inline: false Dec 30 01:27:43.593720193 (585h58m43.464803228s) [upload=513690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:27:43.593724761 (585h58m43.464807796s) [upload=513690]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:27:43.841048388 (585h58m43.712131433s) [seg_pos={0 0}, upload=513690]: Begin upload segment... Dec 30 01:27:43.841105667 (585h58m43.712188712s) [seg_pos={0 0}, upload=513690]: Done begin upload segment. Dec 30 01:27:43.841135033 (585h58m43.712218068s) [noise=true , address=45.91.116.13:4154 , piece_id=6XSDYKFCMI3JLTGYTTWCIS3WHXBQYVUQD5IAUGBV3RLHBVIAXWZA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.846794124 (585h58m43.717877169s) [noise=true , address=145.79.227.34:4907 , piece_id=BWTXN7ILYAM3YBF3PKLSUXPBZS62C3T5JITT5RV2UF27RS4ANJXQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.847687495 (585h58m43.71877054s) [noise=true , address=145.79.227.36:4519 , piece_id=GWTCGD2AICM6HVVE3U4LB6W3GTEO765R45KTTL3V3Z7VMZNDP27Q, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.848019897 (585h58m43.719102932s) [noise=true , address=145.79.227.37:4567 , piece_id=ALHGJOJQR2QQ2QWC5EN347MBWAQGPHL3FEIH6ZER4Q6NX6FAW34A, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.848362470 (585h58m43.719445505s) [noise=true , address=45.91.116.12:4228 , piece_id=H2ILHVVJQ6LAD67OYQZJFSWT4I6A7PZRKEG7P4BJABNRW5NP7JDA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.848703859 (585h58m43.719786904s) [noise=true , address=45.91.116.7:4010 , piece_id=BY2LCGCQF4BFEO5QC43MP4EBOUIHPBEYPBP6R5EEN3PR5HHZXYRQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.849050659 (585h58m43.720133694s) [noise=true , address=145.79.227.26:4654 , piece_id=2JY3GTBPFN2T2IJD3WIBXS76RMB3VXMXMFZJXIBWM6V3XO773XMQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.849388282 (585h58m43.720471317s) [noise=true , address=145.79.227.38:4979 , piece_id=JZAGRA6TX4BOGAL5J2K5NLV6RZ2TQZXJGSPGBEUHQYV6UYEQH4DQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.849727838 (585h58m43.720810883s) [noise=true , address=77.90.31.11:4555 , piece_id=P4JTRDJURCVZAFOKX73LONMF3FLVBEMP6EN7NHAATPJ6KJ3IBDQA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850002681 (585h58m43.721085736s) [noise=true , address=145.79.227.11:4732 , piece_id=KHMB5D4233GN7TFWG4G7XSLG3NYGU4CGLEO2UMN5RMAG3VFTQT4A, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850191261 (585h58m43.721274306s) [noise=true , address=77.90.31.13:4627 , piece_id=6AUPOIIVG5REWLT3DJLLFYSHJLI46JST4R4FMULZRIIWRHPOFB2Q, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850634494 (585h58m43.721717539s) [noise=true , address=145.79.227.5:4223 , piece_id=GQDKDXGFGI5WMZ3N764ZDFBZL5DJPSMNNXS4T4T57ANMIOTYUMRQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850788557 (585h58m43.721871592s) [noise=true , address=145.79.227.45:4230 , piece_id=X7NLAU6OYA2OC52J5KGFXFNDLGC4O4GTJRBX2RBFDJQENW6EST3A, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.851327273 (585h58m43.722410308s) [noise=true , address=145.79.227.47:4266 , piece_id=5QMHAY2T3EDDEWAKLRYLN5ERGKVLIL5BUEDDGKNIB5TUICAW7AWA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.851897969 (585h58m43.722981004s) [noise=true , address=145.79.227.14:4837 , piece_id=UT7J3YUHHB4RGIRB5EP2PJHPRGAJTQ3ITHQ43PPK5OXTCJKTUVOA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.852473204 (585h58m43.723556249s) [noise=true , address=145.79.227.52:4447 , piece_id=R5VK67OTLXAPTNN5HH2D2555SL55AAMSSWFBHK5SSKAAXCCK3AJA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:45.490164438 (585h58m45.361247483s) [upload=513690]: Next returned nil segment Dec 30 01:27:45.490198152 (585h58m45.361281197s) [upload=513690]: Waiting for error group managing segments... Dec 30 01:27:45.830727878 (585h58m45.701810933s) [noise=true , address=145.79.227.14:4837 , piece_id=UT7J3YUHHB4RGIRB5EP2PJHPRGAJTQ3ITHQ43PPK5OXTCJKTUVOA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.833647284 (585h58m45.704730319s) [noise=true , address=145.79.227.45:4230 , piece_id=X7NLAU6OYA2OC52J5KGFXFNDLGC4O4GTJRBX2RBFDJQENW6EST3A, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.852769451 (585h58m45.723852486s) [noise=true , address=145.79.227.5:4223 , piece_id=GQDKDXGFGI5WMZ3N764ZDFBZL5DJPSMNNXS4T4T57ANMIOTYUMRQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.852980513 (585h58m45.724063548s) [noise=true , address=145.79.227.38:4979 , piece_id=JZAGRA6TX4BOGAL5J2K5NLV6RZ2TQZXJGSPGBEUHQYV6UYEQH4DQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.856242251 (585h58m45.727325317s) [noise=true , address=145.79.227.47:4266 , piece_id=5QMHAY2T3EDDEWAKLRYLN5ERGKVLIL5BUEDDGKNIB5TUICAW7AWA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.858762338 (585h58m45.729845373s) [noise=true , address=145.79.227.52:4447 , piece_id=R5VK67OTLXAPTNN5HH2D2555SL55AAMSSWFBHK5SSKAAXCCK3AJA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.859500203 (585h58m45.730583238s) [noise=true , address=145.79.227.36:4519 , piece_id=GWTCGD2AICM6HVVE3U4LB6W3GTEO765R45KTTL3V3Z7VMZNDP27Q, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.859874234 (585h58m45.730957269s) [noise=true , address=145.79.227.26:4654 , piece_id=2JY3GTBPFN2T2IJD3WIBXS76RMB3VXMXMFZJXIBWM6V3XO773XMQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.860342475 (585h58m45.73142551s) [noise=true , address=145.79.227.11:4732 , piece_id=KHMB5D4233GN7TFWG4G7XSLG3NYGU4CGLEO2UMN5RMAG3VFTQT4A, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.126796667 (585h58m45.997879702s) [noise=true , address=77.90.31.11:4555 , piece_id=P4JTRDJURCVZAFOKX73LONMF3FLVBEMP6EN7NHAATPJ6KJ3IBDQA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.180715827 (585h58m46.051798862s) [noise=true , address=45.91.116.7:4010 , piece_id=BY2LCGCQF4BFEO5QC43MP4EBOUIHPBEYPBP6R5EEN3PR5HHZXYRQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.668961231 (585h58m46.540044266s) [noise=true , address=77.90.31.13:4627 , piece_id=6AUPOIIVG5REWLT3DJLLFYSHJLI46JST4R4FMULZRIIWRHPOFB2Q, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.933356712 (585h58m46.804439747s) [noise=true , address=45.91.116.13:4154 , piece_id=6XSDYKFCMI3JLTGYTTWCIS3WHXBQYVUQD5IAUGBV3RLHBVIAXWZA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.933373945 (585h58m46.80445698s) [seg_pos={0 0}, upload=513690]: Segment reached optimal threshold of 13 pieces. Dec 30 01:27:46.933540653 (585h58m46.804623688s) [noise=true , address=45.91.116.12:4228 , piece_id=H2ILHVVJQ6LAD67OYQZJFSWT4I6A7PZRKEG7P4BJABNRW5NP7JDA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 30 01:27:46.933547756 (585h58m46.804630791s) [noise=true , address=45.91.116.12:4228 , piece_id=H2ILHVVJQ6LAD67OYQZJFSWT4I6A7PZRKEG7P4BJABNRW5NP7JDA, seg_pos={0 0}, upload=513690]: Long tail cancellation due to optimal threshold Dec 30 01:27:46.933629041 (585h58m46.804712086s) [noise=true , address=145.79.227.37:4567 , piece_id=ALHGJOJQR2QQ2QWC5EN347MBWAQGPHL3FEIH6ZER4Q6NX6FAW34A, seg_pos={0 0}, upload=513690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:52334->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:52334->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52334->145.79.227.37:4567: use of closed network connection Dec 30 01:27:46.933636274 (585h58m46.804719309s) [noise=true , address=145.79.227.37:4567 , piece_id=ALHGJOJQR2QQ2QWC5EN347MBWAQGPHL3FEIH6ZER4Q6NX6FAW34A, seg_pos={0 0}, upload=513690]: Long tail cancellation due to optimal threshold Dec 30 01:27:46.933768126 (585h58m46.804851161s) [noise=true , address=145.79.227.34:4907 , piece_id=BWTXN7ILYAM3YBF3PKLSUXPBZS62C3T5JITT5RV2UF27RS4ANJXQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 01:27:46.933774417 (585h58m46.804857462s) [noise=true , address=145.79.227.34:4907 , piece_id=BWTXN7ILYAM3YBF3PKLSUXPBZS62C3T5JITT5RV2UF27RS4ANJXQ, seg_pos={0 0}, upload=513690]: Long tail cancellation due to optimal threshold Dec 30 01:27:46.933788274 (585h58m46.804871309s) [seg_pos={0 0}, upload=513690]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:27:46.933885048 (585h58m46.804968113s) [upload=513690]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:27:46.933889797 (585h58m46.804972832s) [upload=513690]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:27:47.227746916 (585h58m47.098829951s) [upload=513690]: Done uploading segments... Dec 30 01:27:47.227765671 (585h58m47.098848706s) [upload=513690]: Upload finished. err: <nil> Duration: 5.726990255s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 25.385438276s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:30:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 25.385479987s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X......X.X...X....X.X.....X.................XX.....X.X...] duration: 6.06348728s trace: https://jaeger.dev.storj.io/trace/22d6ffcc1c4b6e7f FAIL (2025-12-30 01:30:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27:43.593396426 (585h58m43.464479461s) [upload=513690]: Starting upload Dec 30 01:27:43.593455890 (585h58m43.464538925s) [upload=513690]: Done starting upload Dec 30 01:27:43.593640080 (585h58m43.464723115s) [upload=513690]: Uploading segments... Dec 30 01:27:43.593684705 (585h58m43.46476775s) [upload=513690]: Got next segment. Inline: false Dec 30 01:27:43.593720193 (585h58m43.464803228s) [upload=513690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:27:43.593724761 (585h58m43.464807796s) [upload=513690]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:27:43.841048388 (585h58m43.712131433s) [seg_pos={0 0}, upload=513690]: Begin upload segment... Dec 30 01:27:43.841105667 (585h58m43.712188712s) [seg_pos={0 0}, upload=513690]: Done begin upload segment. Dec 30 01:27:43.841135033 (585h58m43.712218068s) [noise=true , address=45.91.116.13:4154 , piece_id=6XSDYKFCMI3JLTGYTTWCIS3WHXBQYVUQD5IAUGBV3RLHBVIAXWZA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.846794124 (585h58m43.717877169s) [noise=true , address=145.79.227.34:4907 , piece_id=BWTXN7ILYAM3YBF3PKLSUXPBZS62C3T5JITT5RV2UF27RS4ANJXQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.847687495 (585h58m43.71877054s) [noise=true , address=145.79.227.36:4519 , piece_id=GWTCGD2AICM6HVVE3U4LB6W3GTEO765R45KTTL3V3Z7VMZNDP27Q, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.848019897 (585h58m43.719102932s) [noise=true , address=145.79.227.37:4567 , piece_id=ALHGJOJQR2QQ2QWC5EN347MBWAQGPHL3FEIH6ZER4Q6NX6FAW34A, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.848362470 (585h58m43.719445505s) [noise=true , address=45.91.116.12:4228 , piece_id=H2ILHVVJQ6LAD67OYQZJFSWT4I6A7PZRKEG7P4BJABNRW5NP7JDA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.848703859 (585h58m43.719786904s) [noise=true , address=45.91.116.7:4010 , piece_id=BY2LCGCQF4BFEO5QC43MP4EBOUIHPBEYPBP6R5EEN3PR5HHZXYRQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.849050659 (585h58m43.720133694s) [noise=true , address=145.79.227.26:4654 , piece_id=2JY3GTBPFN2T2IJD3WIBXS76RMB3VXMXMFZJXIBWM6V3XO773XMQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.849388282 (585h58m43.720471317s) [noise=true , address=145.79.227.38:4979 , piece_id=JZAGRA6TX4BOGAL5J2K5NLV6RZ2TQZXJGSPGBEUHQYV6UYEQH4DQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.849727838 (585h58m43.720810883s) [noise=true , address=77.90.31.11:4555 , piece_id=P4JTRDJURCVZAFOKX73LONMF3FLVBEMP6EN7NHAATPJ6KJ3IBDQA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850002681 (585h58m43.721085736s) [noise=true , address=145.79.227.11:4732 , piece_id=KHMB5D4233GN7TFWG4G7XSLG3NYGU4CGLEO2UMN5RMAG3VFTQT4A, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850191261 (585h58m43.721274306s) [noise=true , address=77.90.31.13:4627 , piece_id=6AUPOIIVG5REWLT3DJLLFYSHJLI46JST4R4FMULZRIIWRHPOFB2Q, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850634494 (585h58m43.721717539s) [noise=true , address=145.79.227.5:4223 , piece_id=GQDKDXGFGI5WMZ3N764ZDFBZL5DJPSMNNXS4T4T57ANMIOTYUMRQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850788557 (585h58m43.721871592s) [noise=true , address=145.79.227.45:4230 , piece_id=X7NLAU6OYA2OC52J5KGFXFNDLGC4O4GTJRBX2RBFDJQENW6EST3A, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.851327273 (585h58m43.722410308s) [noise=true , address=145.79.227.47:4266 , piece_id=5QMHAY2T3EDDEWAKLRYLN5ERGKVLIL5BUEDDGKNIB5TUICAW7AWA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.851897969 (585h58m43.722981004s) [noise=true , address=145.79.227.14:4837 , piece_id=UT7J3YUHHB4RGIRB5EP2PJHPRGAJTQ3ITHQ43PPK5OXTCJKTUVOA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.852473204 (585h58m43.723556249s) [noise=true , address=145.79.227.52:4447 , piece_id=R5VK67OTLXAPTNN5HH2D2555SL55AAMSSWFBHK5SSKAAXCCK3AJA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:45.490164438 (585h58m45.361247483s) [upload=513690]: Next returned nil segment Dec 30 01:27:45.490198152 (585h58m45.361281197s) [upload=513690]: Waiting for error group managing segments... Dec 30 01:27:45.830727878 (585h58m45.701810933s) [noise=true , address=145.79.227.14:4837 , piece_id=UT7J3YUHHB4RGIRB5EP2PJHPRGAJTQ3ITHQ43PPK5OXTCJKTUVOA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.833647284 (585h58m45.704730319s) [noise=true , address=145.79.227.45:4230 , piece_id=X7NLAU6OYA2OC52J5KGFXFNDLGC4O4GTJRBX2RBFDJQENW6EST3A, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.852769451 (585h58m45.723852486s) [noise=true , address=145.79.227.5:4223 , piece_id=GQDKDXGFGI5WMZ3N764ZDFBZL5DJPSMNNXS4T4T57ANMIOTYUMRQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.852980513 (585h58m45.724063548s) [noise=true , address=145.79.227.38:4979 , piece_id=JZAGRA6TX4BOGAL5J2K5NLV6RZ2TQZXJGSPGBEUHQYV6UYEQH4DQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.856242251 (585h58m45.727325317s) [noise=true , address=145.79.227.47:4266 , piece_id=5QMHAY2T3EDDEWAKLRYLN5ERGKVLIL5BUEDDGKNIB5TUICAW7AWA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.858762338 (585h58m45.729845373s) [noise=true , address=145.79.227.52:4447 , piece_id=R5VK67OTLXAPTNN5HH2D2555SL55AAMSSWFBHK5SSKAAXCCK3AJA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.859500203 (585h58m45.730583238s) [noise=true , address=145.79.227.36:4519 , piece_id=GWTCGD2AICM6HVVE3U4LB6W3GTEO765R45KTTL3V3Z7VMZNDP27Q, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.859874234 (585h58m45.730957269s) [noise=true , address=145.79.227.26:4654 , piece_id=2JY3GTBPFN2T2IJD3WIBXS76RMB3VXMXMFZJXIBWM6V3XO773XMQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.860342475 (585h58m45.73142551s) [noise=true , address=145.79.227.11:4732 , piece_id=KHMB5D4233GN7TFWG4G7XSLG3NYGU4CGLEO2UMN5RMAG3VFTQT4A, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.126796667 (585h58m45.997879702s) [noise=true , address=77.90.31.11:4555 , piece_id=P4JTRDJURCVZAFOKX73LONMF3FLVBEMP6EN7NHAATPJ6KJ3IBDQA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.180715827 (585h58m46.051798862s) [noise=true , address=45.91.116.7:4010 , piece_id=BY2LCGCQF4BFEO5QC43MP4EBOUIHPBEYPBP6R5EEN3PR5HHZXYRQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.668961231 (585h58m46.540044266s) [noise=true , address=77.90.31.13:4627 , piece_id=6AUPOIIVG5REWLT3DJLLFYSHJLI46JST4R4FMULZRIIWRHPOFB2Q, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.933356712 (585h58m46.804439747s) [noise=true , address=45.91.116.13:4154 , piece_id=6XSDYKFCMI3JLTGYTTWCIS3WHXBQYVUQD5IAUGBV3RLHBVIAXWZA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.933373945 (585h58m46.80445698s) [seg_pos={0 0}, upload=513690]: Segment reached optimal threshold of 13 pieces. Dec 30 01:27:46.933540653 (585h58m46.804623688s) [noise=true , address=45.91.116.12:4228 , piece_id=H2ILHVVJQ6LAD67OYQZJFSWT4I6A7PZRKEG7P4BJABNRW5NP7JDA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 30 01:27:46.933547756 (585h58m46.804630791s) [noise=true , address=45.91.116.12:4228 , piece_id=H2ILHVVJQ6LAD67OYQZJFSWT4I6A7PZRKEG7P4BJABNRW5NP7JDA, seg_pos={0 0}, upload=513690]: Long tail cancellation due to optimal threshold Dec 30 01:27:46.933629041 (585h58m46.804712086s) [noise=true , address=145.79.227.37:4567 , piece_id=ALHGJOJQR2QQ2QWC5EN347MBWAQGPHL3FEIH6ZER4Q6NX6FAW34A, seg_pos={0 0}, upload=513690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:52334->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:52334->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52334->145.79.227.37:4567: use of closed network connection Dec 30 01:27:46.933636274 (585h58m46.804719309s) [noise=true , address=145.79.227.37:4567 , piece_id=ALHGJOJQR2QQ2QWC5EN347MBWAQGPHL3FEIH6ZER4Q6NX6FAW34A, seg_pos={0 0}, upload=513690]: Long tail cancellation due to optimal threshold Dec 30 01:27:46.933768126 (585h58m46.804851161s) [noise=true , address=145.79.227.34:4907 , piece_id=BWTXN7ILYAM3YBF3PKLSUXPBZS62C3T5JITT5RV2UF27RS4ANJXQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 01:27:46.933774417 (585h58m46.804857462s) [noise=true , address=145.79.227.34:4907 , piece_id=BWTXN7ILYAM3YBF3PKLSUXPBZS62C3T5JITT5RV2UF27RS4ANJXQ, seg_pos={0 0}, upload=513690]: Long tail cancellation due to optimal threshold Dec 30 01:27:46.933788274 (585h58m46.804871309s) [seg_pos={0 0}, upload=513690]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:27:46.933885048 (585h58m46.804968113s) [upload=513690]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:27:46.933889797 (585h58m46.804972832s) [upload=513690]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:27:47.227746916 (585h58m47.098829951s) [upload=513690]: Done uploading segments... Dec 30 01:27:47.227765671 (585h58m47.098848706s) [upload=513690]: Upload finished. err: <nil> Duration: 6.063517066s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 52.551156097s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:30:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 52.551180574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X......X.X...X....X.X.....X.................XX.....X.X..] duration: 5.991355853s trace: https://jaeger.dev.storj.io/trace/22d6ffcc1c4b6e7f FAIL (2025-12-30 01:29:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27:43.593396426 (585h58m43.464479461s) [upload=513690]: Starting upload Dec 30 01:27:43.593455890 (585h58m43.464538925s) [upload=513690]: Done starting upload Dec 30 01:27:43.593640080 (585h58m43.464723115s) [upload=513690]: Uploading segments... Dec 30 01:27:43.593684705 (585h58m43.46476775s) [upload=513690]: Got next segment. Inline: false Dec 30 01:27:43.593720193 (585h58m43.464803228s) [upload=513690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:27:43.593724761 (585h58m43.464807796s) [upload=513690]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:27:43.841048388 (585h58m43.712131433s) [seg_pos={0 0}, upload=513690]: Begin upload segment... Dec 30 01:27:43.841105667 (585h58m43.712188712s) [seg_pos={0 0}, upload=513690]: Done begin upload segment. Dec 30 01:27:43.841135033 (585h58m43.712218068s) [noise=true , address=45.91.116.13:4154 , piece_id=6XSDYKFCMI3JLTGYTTWCIS3WHXBQYVUQD5IAUGBV3RLHBVIAXWZA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.846794124 (585h58m43.717877169s) [noise=true , address=145.79.227.34:4907 , piece_id=BWTXN7ILYAM3YBF3PKLSUXPBZS62C3T5JITT5RV2UF27RS4ANJXQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.847687495 (585h58m43.71877054s) [noise=true , address=145.79.227.36:4519 , piece_id=GWTCGD2AICM6HVVE3U4LB6W3GTEO765R45KTTL3V3Z7VMZNDP27Q, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.848019897 (585h58m43.719102932s) [noise=true , address=145.79.227.37:4567 , piece_id=ALHGJOJQR2QQ2QWC5EN347MBWAQGPHL3FEIH6ZER4Q6NX6FAW34A, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.848362470 (585h58m43.719445505s) [noise=true , address=45.91.116.12:4228 , piece_id=H2ILHVVJQ6LAD67OYQZJFSWT4I6A7PZRKEG7P4BJABNRW5NP7JDA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.848703859 (585h58m43.719786904s) [noise=true , address=45.91.116.7:4010 , piece_id=BY2LCGCQF4BFEO5QC43MP4EBOUIHPBEYPBP6R5EEN3PR5HHZXYRQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.849050659 (585h58m43.720133694s) [noise=true , address=145.79.227.26:4654 , piece_id=2JY3GTBPFN2T2IJD3WIBXS76RMB3VXMXMFZJXIBWM6V3XO773XMQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.849388282 (585h58m43.720471317s) [noise=true , address=145.79.227.38:4979 , piece_id=JZAGRA6TX4BOGAL5J2K5NLV6RZ2TQZXJGSPGBEUHQYV6UYEQH4DQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.849727838 (585h58m43.720810883s) [noise=true , address=77.90.31.11:4555 , piece_id=P4JTRDJURCVZAFOKX73LONMF3FLVBEMP6EN7NHAATPJ6KJ3IBDQA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850002681 (585h58m43.721085736s) [noise=true , address=145.79.227.11:4732 , piece_id=KHMB5D4233GN7TFWG4G7XSLG3NYGU4CGLEO2UMN5RMAG3VFTQT4A, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850191261 (585h58m43.721274306s) [noise=true , address=77.90.31.13:4627 , piece_id=6AUPOIIVG5REWLT3DJLLFYSHJLI46JST4R4FMULZRIIWRHPOFB2Q, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850634494 (585h58m43.721717539s) [noise=true , address=145.79.227.5:4223 , piece_id=GQDKDXGFGI5WMZ3N764ZDFBZL5DJPSMNNXS4T4T57ANMIOTYUMRQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850788557 (585h58m43.721871592s) [noise=true , address=145.79.227.45:4230 , piece_id=X7NLAU6OYA2OC52J5KGFXFNDLGC4O4GTJRBX2RBFDJQENW6EST3A, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.851327273 (585h58m43.722410308s) [noise=true , address=145.79.227.47:4266 , piece_id=5QMHAY2T3EDDEWAKLRYLN5ERGKVLIL5BUEDDGKNIB5TUICAW7AWA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.851897969 (585h58m43.722981004s) [noise=true , address=145.79.227.14:4837 , piece_id=UT7J3YUHHB4RGIRB5EP2PJHPRGAJTQ3ITHQ43PPK5OXTCJKTUVOA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.852473204 (585h58m43.723556249s) [noise=true , address=145.79.227.52:4447 , piece_id=R5VK67OTLXAPTNN5HH2D2555SL55AAMSSWFBHK5SSKAAXCCK3AJA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:45.490164438 (585h58m45.361247483s) [upload=513690]: Next returned nil segment Dec 30 01:27:45.490198152 (585h58m45.361281197s) [upload=513690]: Waiting for error group managing segments... Dec 30 01:27:45.830727878 (585h58m45.701810933s) [noise=true , address=145.79.227.14:4837 , piece_id=UT7J3YUHHB4RGIRB5EP2PJHPRGAJTQ3ITHQ43PPK5OXTCJKTUVOA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.833647284 (585h58m45.704730319s) [noise=true , address=145.79.227.45:4230 , piece_id=X7NLAU6OYA2OC52J5KGFXFNDLGC4O4GTJRBX2RBFDJQENW6EST3A, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.852769451 (585h58m45.723852486s) [noise=true , address=145.79.227.5:4223 , piece_id=GQDKDXGFGI5WMZ3N764ZDFBZL5DJPSMNNXS4T4T57ANMIOTYUMRQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.852980513 (585h58m45.724063548s) [noise=true , address=145.79.227.38:4979 , piece_id=JZAGRA6TX4BOGAL5J2K5NLV6RZ2TQZXJGSPGBEUHQYV6UYEQH4DQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.856242251 (585h58m45.727325317s) [noise=true , address=145.79.227.47:4266 , piece_id=5QMHAY2T3EDDEWAKLRYLN5ERGKVLIL5BUEDDGKNIB5TUICAW7AWA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.858762338 (585h58m45.729845373s) [noise=true , address=145.79.227.52:4447 , piece_id=R5VK67OTLXAPTNN5HH2D2555SL55AAMSSWFBHK5SSKAAXCCK3AJA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.859500203 (585h58m45.730583238s) [noise=true , address=145.79.227.36:4519 , piece_id=GWTCGD2AICM6HVVE3U4LB6W3GTEO765R45KTTL3V3Z7VMZNDP27Q, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.859874234 (585h58m45.730957269s) [noise=true , address=145.79.227.26:4654 , piece_id=2JY3GTBPFN2T2IJD3WIBXS76RMB3VXMXMFZJXIBWM6V3XO773XMQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.860342475 (585h58m45.73142551s) [noise=true , address=145.79.227.11:4732 , piece_id=KHMB5D4233GN7TFWG4G7XSLG3NYGU4CGLEO2UMN5RMAG3VFTQT4A, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.126796667 (585h58m45.997879702s) [noise=true , address=77.90.31.11:4555 , piece_id=P4JTRDJURCVZAFOKX73LONMF3FLVBEMP6EN7NHAATPJ6KJ3IBDQA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.180715827 (585h58m46.051798862s) [noise=true , address=45.91.116.7:4010 , piece_id=BY2LCGCQF4BFEO5QC43MP4EBOUIHPBEYPBP6R5EEN3PR5HHZXYRQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.668961231 (585h58m46.540044266s) [noise=true , address=77.90.31.13:4627 , piece_id=6AUPOIIVG5REWLT3DJLLFYSHJLI46JST4R4FMULZRIIWRHPOFB2Q, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.933356712 (585h58m46.804439747s) [noise=true , address=45.91.116.13:4154 , piece_id=6XSDYKFCMI3JLTGYTTWCIS3WHXBQYVUQD5IAUGBV3RLHBVIAXWZA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.933373945 (585h58m46.80445698s) [seg_pos={0 0}, upload=513690]: Segment reached optimal threshold of 13 pieces. Dec 30 01:27:46.933540653 (585h58m46.804623688s) [noise=true , address=45.91.116.12:4228 , piece_id=H2ILHVVJQ6LAD67OYQZJFSWT4I6A7PZRKEG7P4BJABNRW5NP7JDA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 30 01:27:46.933547756 (585h58m46.804630791s) [noise=true , address=45.91.116.12:4228 , piece_id=H2ILHVVJQ6LAD67OYQZJFSWT4I6A7PZRKEG7P4BJABNRW5NP7JDA, seg_pos={0 0}, upload=513690]: Long tail cancellation due to optimal threshold Dec 30 01:27:46.933629041 (585h58m46.804712086s) [noise=true , address=145.79.227.37:4567 , piece_id=ALHGJOJQR2QQ2QWC5EN347MBWAQGPHL3FEIH6ZER4Q6NX6FAW34A, seg_pos={0 0}, upload=513690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:52334->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:52334->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52334->145.79.227.37:4567: use of closed network connection Dec 30 01:27:46.933636274 (585h58m46.804719309s) [noise=true , address=145.79.227.37:4567 , piece_id=ALHGJOJQR2QQ2QWC5EN347MBWAQGPHL3FEIH6ZER4Q6NX6FAW34A, seg_pos={0 0}, upload=513690]: Long tail cancellation due to optimal threshold Dec 30 01:27:46.933768126 (585h58m46.804851161s) [noise=true , address=145.79.227.34:4907 , piece_id=BWTXN7ILYAM3YBF3PKLSUXPBZS62C3T5JITT5RV2UF27RS4ANJXQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 01:27:46.933774417 (585h58m46.804857462s) [noise=true , address=145.79.227.34:4907 , piece_id=BWTXN7ILYAM3YBF3PKLSUXPBZS62C3T5JITT5RV2UF27RS4ANJXQ, seg_pos={0 0}, upload=513690]: Long tail cancellation due to optimal threshold Dec 30 01:27:46.933788274 (585h58m46.804871309s) [seg_pos={0 0}, upload=513690]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:27:46.933885048 (585h58m46.804968113s) [upload=513690]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:27:46.933889797 (585h58m46.804972832s) [upload=513690]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:27:47.227746916 (585h58m47.098829951s) [upload=513690]: Done uploading segments... Dec 30 01:27:47.227765671 (585h58m47.098848706s) [upload=513690]: Upload finished. err: <nil> Duration: 5.99139645s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....X......X.X...X....X.X.....X.................XX.....X.X.] duration: 5.80754591s trace: https://jaeger.dev.storj.io/trace/22d6ffcc1c4b6e7f FAIL (2025-12-30 01:28:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27:43.593396426 (585h58m43.464479461s) [upload=513690]: Starting upload Dec 30 01:27:43.593455890 (585h58m43.464538925s) [upload=513690]: Done starting upload Dec 30 01:27:43.593640080 (585h58m43.464723115s) [upload=513690]: Uploading segments... Dec 30 01:27:43.593684705 (585h58m43.46476775s) [upload=513690]: Got next segment. Inline: false Dec 30 01:27:43.593720193 (585h58m43.464803228s) [upload=513690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:27:43.593724761 (585h58m43.464807796s) [upload=513690]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:27:43.841048388 (585h58m43.712131433s) [seg_pos={0 0}, upload=513690]: Begin upload segment... Dec 30 01:27:43.841105667 (585h58m43.712188712s) [seg_pos={0 0}, upload=513690]: Done begin upload segment. Dec 30 01:27:43.841135033 (585h58m43.712218068s) [noise=true , address=45.91.116.13:4154 , piece_id=6XSDYKFCMI3JLTGYTTWCIS3WHXBQYVUQD5IAUGBV3RLHBVIAXWZA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.846794124 (585h58m43.717877169s) [noise=true , address=145.79.227.34:4907 , piece_id=BWTXN7ILYAM3YBF3PKLSUXPBZS62C3T5JITT5RV2UF27RS4ANJXQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.847687495 (585h58m43.71877054s) [noise=true , address=145.79.227.36:4519 , piece_id=GWTCGD2AICM6HVVE3U4LB6W3GTEO765R45KTTL3V3Z7VMZNDP27Q, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.848019897 (585h58m43.719102932s) [noise=true , address=145.79.227.37:4567 , piece_id=ALHGJOJQR2QQ2QWC5EN347MBWAQGPHL3FEIH6ZER4Q6NX6FAW34A, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.848362470 (585h58m43.719445505s) [noise=true , address=45.91.116.12:4228 , piece_id=H2ILHVVJQ6LAD67OYQZJFSWT4I6A7PZRKEG7P4BJABNRW5NP7JDA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.848703859 (585h58m43.719786904s) [noise=true , address=45.91.116.7:4010 , piece_id=BY2LCGCQF4BFEO5QC43MP4EBOUIHPBEYPBP6R5EEN3PR5HHZXYRQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.849050659 (585h58m43.720133694s) [noise=true , address=145.79.227.26:4654 , piece_id=2JY3GTBPFN2T2IJD3WIBXS76RMB3VXMXMFZJXIBWM6V3XO773XMQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.849388282 (585h58m43.720471317s) [noise=true , address=145.79.227.38:4979 , piece_id=JZAGRA6TX4BOGAL5J2K5NLV6RZ2TQZXJGSPGBEUHQYV6UYEQH4DQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.849727838 (585h58m43.720810883s) [noise=true , address=77.90.31.11:4555 , piece_id=P4JTRDJURCVZAFOKX73LONMF3FLVBEMP6EN7NHAATPJ6KJ3IBDQA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850002681 (585h58m43.721085736s) [noise=true , address=145.79.227.11:4732 , piece_id=KHMB5D4233GN7TFWG4G7XSLG3NYGU4CGLEO2UMN5RMAG3VFTQT4A, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850191261 (585h58m43.721274306s) [noise=true , address=77.90.31.13:4627 , piece_id=6AUPOIIVG5REWLT3DJLLFYSHJLI46JST4R4FMULZRIIWRHPOFB2Q, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850634494 (585h58m43.721717539s) [noise=true , address=145.79.227.5:4223 , piece_id=GQDKDXGFGI5WMZ3N764ZDFBZL5DJPSMNNXS4T4T57ANMIOTYUMRQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850788557 (585h58m43.721871592s) [noise=true , address=145.79.227.45:4230 , piece_id=X7NLAU6OYA2OC52J5KGFXFNDLGC4O4GTJRBX2RBFDJQENW6EST3A, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.851327273 (585h58m43.722410308s) [noise=true , address=145.79.227.47:4266 , piece_id=5QMHAY2T3EDDEWAKLRYLN5ERGKVLIL5BUEDDGKNIB5TUICAW7AWA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.851897969 (585h58m43.722981004s) [noise=true , address=145.79.227.14:4837 , piece_id=UT7J3YUHHB4RGIRB5EP2PJHPRGAJTQ3ITHQ43PPK5OXTCJKTUVOA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.852473204 (585h58m43.723556249s) [noise=true , address=145.79.227.52:4447 , piece_id=R5VK67OTLXAPTNN5HH2D2555SL55AAMSSWFBHK5SSKAAXCCK3AJA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:45.490164438 (585h58m45.361247483s) [upload=513690]: Next returned nil segment Dec 30 01:27:45.490198152 (585h58m45.361281197s) [upload=513690]: Waiting for error group managing segments... Dec 30 01:27:45.830727878 (585h58m45.701810933s) [noise=true , address=145.79.227.14:4837 , piece_id=UT7J3YUHHB4RGIRB5EP2PJHPRGAJTQ3ITHQ43PPK5OXTCJKTUVOA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.833647284 (585h58m45.704730319s) [noise=true , address=145.79.227.45:4230 , piece_id=X7NLAU6OYA2OC52J5KGFXFNDLGC4O4GTJRBX2RBFDJQENW6EST3A, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.852769451 (585h58m45.723852486s) [noise=true , address=145.79.227.5:4223 , piece_id=GQDKDXGFGI5WMZ3N764ZDFBZL5DJPSMNNXS4T4T57ANMIOTYUMRQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.852980513 (585h58m45.724063548s) [noise=true , address=145.79.227.38:4979 , piece_id=JZAGRA6TX4BOGAL5J2K5NLV6RZ2TQZXJGSPGBEUHQYV6UYEQH4DQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.856242251 (585h58m45.727325317s) [noise=true , address=145.79.227.47:4266 , piece_id=5QMHAY2T3EDDEWAKLRYLN5ERGKVLIL5BUEDDGKNIB5TUICAW7AWA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.858762338 (585h58m45.729845373s) [noise=true , address=145.79.227.52:4447 , piece_id=R5VK67OTLXAPTNN5HH2D2555SL55AAMSSWFBHK5SSKAAXCCK3AJA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.859500203 (585h58m45.730583238s) [noise=true , address=145.79.227.36:4519 , piece_id=GWTCGD2AICM6HVVE3U4LB6W3GTEO765R45KTTL3V3Z7VMZNDP27Q, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.859874234 (585h58m45.730957269s) [noise=true , address=145.79.227.26:4654 , piece_id=2JY3GTBPFN2T2IJD3WIBXS76RMB3VXMXMFZJXIBWM6V3XO773XMQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.860342475 (585h58m45.73142551s) [noise=true , address=145.79.227.11:4732 , piece_id=KHMB5D4233GN7TFWG4G7XSLG3NYGU4CGLEO2UMN5RMAG3VFTQT4A, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.126796667 (585h58m45.997879702s) [noise=true , address=77.90.31.11:4555 , piece_id=P4JTRDJURCVZAFOKX73LONMF3FLVBEMP6EN7NHAATPJ6KJ3IBDQA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.180715827 (585h58m46.051798862s) [noise=true , address=45.91.116.7:4010 , piece_id=BY2LCGCQF4BFEO5QC43MP4EBOUIHPBEYPBP6R5EEN3PR5HHZXYRQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.668961231 (585h58m46.540044266s) [noise=true , address=77.90.31.13:4627 , piece_id=6AUPOIIVG5REWLT3DJLLFYSHJLI46JST4R4FMULZRIIWRHPOFB2Q, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.933356712 (585h58m46.804439747s) [noise=true , address=45.91.116.13:4154 , piece_id=6XSDYKFCMI3JLTGYTTWCIS3WHXBQYVUQD5IAUGBV3RLHBVIAXWZA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.933373945 (585h58m46.80445698s) [seg_pos={0 0}, upload=513690]: Segment reached optimal threshold of 13 pieces. Dec 30 01:27:46.933540653 (585h58m46.804623688s) [noise=true , address=45.91.116.12:4228 , piece_id=H2ILHVVJQ6LAD67OYQZJFSWT4I6A7PZRKEG7P4BJABNRW5NP7JDA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 30 01:27:46.933547756 (585h58m46.804630791s) [noise=true , address=45.91.116.12:4228 , piece_id=H2ILHVVJQ6LAD67OYQZJFSWT4I6A7PZRKEG7P4BJABNRW5NP7JDA, seg_pos={0 0}, upload=513690]: Long tail cancellation due to optimal threshold Dec 30 01:27:46.933629041 (585h58m46.804712086s) [noise=true , address=145.79.227.37:4567 , piece_id=ALHGJOJQR2QQ2QWC5EN347MBWAQGPHL3FEIH6ZER4Q6NX6FAW34A, seg_pos={0 0}, upload=513690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:52334->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:52334->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52334->145.79.227.37:4567: use of closed network connection Dec 30 01:27:46.933636274 (585h58m46.804719309s) [noise=true , address=145.79.227.37:4567 , piece_id=ALHGJOJQR2QQ2QWC5EN347MBWAQGPHL3FEIH6ZER4Q6NX6FAW34A, seg_pos={0 0}, upload=513690]: Long tail cancellation due to optimal threshold Dec 30 01:27:46.933768126 (585h58m46.804851161s) [noise=true , address=145.79.227.34:4907 , piece_id=BWTXN7ILYAM3YBF3PKLSUXPBZS62C3T5JITT5RV2UF27RS4ANJXQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 01:27:46.933774417 (585h58m46.804857462s) [noise=true , address=145.79.227.34:4907 , piece_id=BWTXN7ILYAM3YBF3PKLSUXPBZS62C3T5JITT5RV2UF27RS4ANJXQ, seg_pos={0 0}, upload=513690]: Long tail cancellation due to optimal threshold Dec 30 01:27:46.933788274 (585h58m46.804871309s) [seg_pos={0 0}, upload=513690]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:27:46.933885048 (585h58m46.804968113s) [upload=513690]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:27:46.933889797 (585h58m46.804972832s) [upload=513690]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:27:47.227746916 (585h58m47.098829951s) [upload=513690]: Done uploading segments... Dec 30 01:27:47.227765671 (585h58m47.098848706s) [upload=513690]: Upload finished. err: <nil> Duration: 5.807574083s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 31.096767457s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:28:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 31.096797384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 43.713555129s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:28:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 43.713576451s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......X......X.X...X....X.X.....X.................XX.....X.X] duration: 30.004355311s trace: https://jaeger.dev.storj.io/trace/22d6ffcc1c4b6e7f FAIL (2025-12-30 01:28:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27:43.593396426 (585h58m43.464479461s) [upload=513690]: Starting upload Dec 30 01:27:43.593455890 (585h58m43.464538925s) [upload=513690]: Done starting upload Dec 30 01:27:43.593640080 (585h58m43.464723115s) [upload=513690]: Uploading segments... Dec 30 01:27:43.593684705 (585h58m43.46476775s) [upload=513690]: Got next segment. Inline: false Dec 30 01:27:43.593720193 (585h58m43.464803228s) [upload=513690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:27:43.593724761 (585h58m43.464807796s) [upload=513690]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:27:43.841048388 (585h58m43.712131433s) [seg_pos={0 0}, upload=513690]: Begin upload segment... Dec 30 01:27:43.841105667 (585h58m43.712188712s) [seg_pos={0 0}, upload=513690]: Done begin upload segment. Dec 30 01:27:43.841135033 (585h58m43.712218068s) [noise=true , address=45.91.116.13:4154 , piece_id=6XSDYKFCMI3JLTGYTTWCIS3WHXBQYVUQD5IAUGBV3RLHBVIAXWZA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.846794124 (585h58m43.717877169s) [noise=true , address=145.79.227.34:4907 , piece_id=BWTXN7ILYAM3YBF3PKLSUXPBZS62C3T5JITT5RV2UF27RS4ANJXQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.847687495 (585h58m43.71877054s) [noise=true , address=145.79.227.36:4519 , piece_id=GWTCGD2AICM6HVVE3U4LB6W3GTEO765R45KTTL3V3Z7VMZNDP27Q, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.848019897 (585h58m43.719102932s) [noise=true , address=145.79.227.37:4567 , piece_id=ALHGJOJQR2QQ2QWC5EN347MBWAQGPHL3FEIH6ZER4Q6NX6FAW34A, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.848362470 (585h58m43.719445505s) [noise=true , address=45.91.116.12:4228 , piece_id=H2ILHVVJQ6LAD67OYQZJFSWT4I6A7PZRKEG7P4BJABNRW5NP7JDA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.848703859 (585h58m43.719786904s) [noise=true , address=45.91.116.7:4010 , piece_id=BY2LCGCQF4BFEO5QC43MP4EBOUIHPBEYPBP6R5EEN3PR5HHZXYRQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.849050659 (585h58m43.720133694s) [noise=true , address=145.79.227.26:4654 , piece_id=2JY3GTBPFN2T2IJD3WIBXS76RMB3VXMXMFZJXIBWM6V3XO773XMQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.849388282 (585h58m43.720471317s) [noise=true , address=145.79.227.38:4979 , piece_id=JZAGRA6TX4BOGAL5J2K5NLV6RZ2TQZXJGSPGBEUHQYV6UYEQH4DQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.849727838 (585h58m43.720810883s) [noise=true , address=77.90.31.11:4555 , piece_id=P4JTRDJURCVZAFOKX73LONMF3FLVBEMP6EN7NHAATPJ6KJ3IBDQA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850002681 (585h58m43.721085736s) [noise=true , address=145.79.227.11:4732 , piece_id=KHMB5D4233GN7TFWG4G7XSLG3NYGU4CGLEO2UMN5RMAG3VFTQT4A, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850191261 (585h58m43.721274306s) [noise=true , address=77.90.31.13:4627 , piece_id=6AUPOIIVG5REWLT3DJLLFYSHJLI46JST4R4FMULZRIIWRHPOFB2Q, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850634494 (585h58m43.721717539s) [noise=true , address=145.79.227.5:4223 , piece_id=GQDKDXGFGI5WMZ3N764ZDFBZL5DJPSMNNXS4T4T57ANMIOTYUMRQ, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.850788557 (585h58m43.721871592s) [noise=true , address=145.79.227.45:4230 , piece_id=X7NLAU6OYA2OC52J5KGFXFNDLGC4O4GTJRBX2RBFDJQENW6EST3A, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.851327273 (585h58m43.722410308s) [noise=true , address=145.79.227.47:4266 , piece_id=5QMHAY2T3EDDEWAKLRYLN5ERGKVLIL5BUEDDGKNIB5TUICAW7AWA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.851897969 (585h58m43.722981004s) [noise=true , address=145.79.227.14:4837 , piece_id=UT7J3YUHHB4RGIRB5EP2PJHPRGAJTQ3ITHQ43PPK5OXTCJKTUVOA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:43.852473204 (585h58m43.723556249s) [noise=true , address=145.79.227.52:4447 , piece_id=R5VK67OTLXAPTNN5HH2D2555SL55AAMSSWFBHK5SSKAAXCCK3AJA, seg_pos={0 0}, upload=513690]: Uploading piece... Dec 30 01:27:45.490164438 (585h58m45.361247483s) [upload=513690]: Next returned nil segment Dec 30 01:27:45.490198152 (585h58m45.361281197s) [upload=513690]: Waiting for error group managing segments... Dec 30 01:27:45.830727878 (585h58m45.701810933s) [noise=true , address=145.79.227.14:4837 , piece_id=UT7J3YUHHB4RGIRB5EP2PJHPRGAJTQ3ITHQ43PPK5OXTCJKTUVOA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.833647284 (585h58m45.704730319s) [noise=true , address=145.79.227.45:4230 , piece_id=X7NLAU6OYA2OC52J5KGFXFNDLGC4O4GTJRBX2RBFDJQENW6EST3A, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.852769451 (585h58m45.723852486s) [noise=true , address=145.79.227.5:4223 , piece_id=GQDKDXGFGI5WMZ3N764ZDFBZL5DJPSMNNXS4T4T57ANMIOTYUMRQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.852980513 (585h58m45.724063548s) [noise=true , address=145.79.227.38:4979 , piece_id=JZAGRA6TX4BOGAL5J2K5NLV6RZ2TQZXJGSPGBEUHQYV6UYEQH4DQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.856242251 (585h58m45.727325317s) [noise=true , address=145.79.227.47:4266 , piece_id=5QMHAY2T3EDDEWAKLRYLN5ERGKVLIL5BUEDDGKNIB5TUICAW7AWA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.858762338 (585h58m45.729845373s) [noise=true , address=145.79.227.52:4447 , piece_id=R5VK67OTLXAPTNN5HH2D2555SL55AAMSSWFBHK5SSKAAXCCK3AJA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.859500203 (585h58m45.730583238s) [noise=true , address=145.79.227.36:4519 , piece_id=GWTCGD2AICM6HVVE3U4LB6W3GTEO765R45KTTL3V3Z7VMZNDP27Q, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.859874234 (585h58m45.730957269s) [noise=true , address=145.79.227.26:4654 , piece_id=2JY3GTBPFN2T2IJD3WIBXS76RMB3VXMXMFZJXIBWM6V3XO773XMQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:45.860342475 (585h58m45.73142551s) [noise=true , address=145.79.227.11:4732 , piece_id=KHMB5D4233GN7TFWG4G7XSLG3NYGU4CGLEO2UMN5RMAG3VFTQT4A, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.126796667 (585h58m45.997879702s) [noise=true , address=77.90.31.11:4555 , piece_id=P4JTRDJURCVZAFOKX73LONMF3FLVBEMP6EN7NHAATPJ6KJ3IBDQA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.180715827 (585h58m46.051798862s) [noise=true , address=45.91.116.7:4010 , piece_id=BY2LCGCQF4BFEO5QC43MP4EBOUIHPBEYPBP6R5EEN3PR5HHZXYRQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.668961231 (585h58m46.540044266s) [noise=true , address=77.90.31.13:4627 , piece_id=6AUPOIIVG5REWLT3DJLLFYSHJLI46JST4R4FMULZRIIWRHPOFB2Q, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.933356712 (585h58m46.804439747s) [noise=true , address=45.91.116.13:4154 , piece_id=6XSDYKFCMI3JLTGYTTWCIS3WHXBQYVUQD5IAUGBV3RLHBVIAXWZA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: <nil> Dec 30 01:27:46.933373945 (585h58m46.80445698s) [seg_pos={0 0}, upload=513690]: Segment reached optimal threshold of 13 pieces. Dec 30 01:27:46.933540653 (585h58m46.804623688s) [noise=true , address=45.91.116.12:4228 , piece_id=H2ILHVVJQ6LAD67OYQZJFSWT4I6A7PZRKEG7P4BJABNRW5NP7JDA, seg_pos={0 0}, upload=513690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 30 01:27:46.933547756 (585h58m46.804630791s) [noise=true , address=45.91.116.12:4228 , piece_id=H2ILHVVJQ6LAD67OYQZJFSWT4I6A7PZRKEG7P4BJABNRW5NP7JDA, seg_pos={0 0}, upload=513690]: Long tail cancellation due to optimal threshold Dec 30 01:27:46.933629041 (585h58m46.804712086s) [noise=true , address=145.79.227.37:4567 , piece_id=ALHGJOJQR2QQ2QWC5EN347MBWAQGPHL3FEIH6ZER4Q6NX6FAW34A, seg_pos={0 0}, upload=513690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:52334->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:52334->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52334->145.79.227.37:4567: use of closed network connection Dec 30 01:27:46.933636274 (585h58m46.804719309s) [noise=true , address=145.79.227.37:4567 , piece_id=ALHGJOJQR2QQ2QWC5EN347MBWAQGPHL3FEIH6ZER4Q6NX6FAW34A, seg_pos={0 0}, upload=513690]: Long tail cancellation due to optimal threshold Dec 30 01:27:46.933768126 (585h58m46.804851161s) [noise=true , address=145.79.227.34:4907 , piece_id=BWTXN7ILYAM3YBF3PKLSUXPBZS62C3T5JITT5RV2UF27RS4ANJXQ, seg_pos={0 0}, upload=513690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 01:27:46.933774417 (585h58m46.804857462s) [noise=true , address=145.79.227.34:4907 , piece_id=BWTXN7ILYAM3YBF3PKLSUXPBZS62C3T5JITT5RV2UF27RS4ANJXQ, seg_pos={0 0}, upload=513690]: Long tail cancellation due to optimal threshold Dec 30 01:27:46.933788274 (585h58m46.804871309s) [seg_pos={0 0}, upload=513690]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:27:46.933885048 (585h58m46.804968113s) [upload=513690]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:27:46.933889797 (585h58m46.804972832s) [upload=513690]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:27:47.227746916 (585h58m47.098829951s) [upload=513690]: Done uploading segments... Dec 30 01:27:47.227765671 (585h58m47.098848706s) [upload=513690]: Upload finished. err: <nil> Duration: 30.004378727s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......X......X.X...X....X.X.....X.................XX.....X.] duration: 6.011664348s trace: https://jaeger.dev.storj.io/trace/c397a876440ddad FAIL (2025-12-30 01:27:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: TNR6UIPYMDWWHMYW5H5XL6BISZ3MVQLYOC3CQ4BYWCQH5CK2LACQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 5DQCCUSC2LGILPIAF5AU4XL6AMIGT6GZGSFQTOISMGQXDGNA7QVQ) context deadline exceeded ======LOGS====== Dec 30 01:26:20.003606125 (585h57m19.87468917s) [upload=513669]: Starting upload Dec 30 01:26:20.003731103 (585h57m19.874814148s) [upload=513669]: Done starting upload Dec 30 01:26:20.003979346 (585h57m19.875062391s) [upload=513669]: Uploading segments... Dec 30 01:26:20.004025794 (585h57m19.875108829s) [upload=513669]: Got next segment. Inline: false Dec 30 01:26:20.004064858 (585h57m19.875147893s) [upload=513669]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:26:20.004069747 (585h57m19.875152792s) [upload=513669]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:26:20.252636560 (585h57m20.123719615s) [seg_pos={0 0}, upload=513669]: Begin upload segment... Dec 30 01:26:20.252713246 (585h57m20.123796291s) [seg_pos={0 0}, upload=513669]: Done begin upload segment. Dec 30 01:26:20.252739556 (585h57m20.123822601s) [noise=true , address=145.79.227.11:4730 , piece_id=HAFQQSXMMRCFKVIRD3RTHXUQGJ4BVDBWEBSRZHZNOLJLFBZAYZUQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.253864839 (585h57m20.124947874s) [noise=true , address=145.79.227.47:4267 , piece_id=B62KREWTBELVUDLTSIKRZZKYZ646V5F3GVKFHZVUEVWX7WXXQWOA, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.254391901 (585h57m20.125474936s) [noise=true , address=145.79.227.5:4223 , piece_id=ZHJIK4L3VLILAHEPW44POAJRQDSSB2JHK5IYFLRK2AWT4VBJ2SAQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.254847368 (585h57m20.125930413s) [noise=true , address=145.79.227.49:4338 , piece_id=I5GJ4DIMG77FWUHAJKNAUUOLUM3VJCXREN526WAFAIBCIJZ5MHKQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.255253371 (585h57m20.126336416s) [noise=true , address=145.79.227.10:4401 , piece_id=FTVQKHFJ4KZZKH74E3QBOAN5V4BICHGHMDO5A7VQPJFX5X7SCVEQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.255734778 (585h57m20.126817813s) [noise=true , address=145.79.227.16:4912 , piece_id=VWLJCBE5UK6DGLE4QMIHBM6ER7GJ4TTFCJSFCXLQR4VN5OHFDLQA, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.257152405 (585h57m20.12823544s) [noise=true , address=145.79.227.36:4518 , piece_id=VF5W3AYMAYPDLMGCNS4EUNRRR7WZUHXA43D3PX77GAHVQDEBKMLQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.258747412 (585h57m20.129830447s) [noise=true , address=145.79.227.42:4123 , piece_id=EZJ6WS7SB232CKFI627CPW3MQF5J4QESBQR2BCYVSIXNGFOCYS2Q, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.260291070 (585h57m20.131374115s) [noise=true , address=45.91.116.6:4975 , piece_id=FLPG4XB4YR6NHDRTKU6TI6ZUEBQBOPD2C3RZLDSBPG6FEAKGN3VA, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.262997652 (585h57m20.134080687s) [noise=true , address=45.91.116.7:4009 , piece_id=M62L3MI2LOHPFX2LNTU7OMN75B2BMD4OAHFN55G2QMVFKDEDQ6TQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.265624722 (585h57m20.136707767s) [noise=true , address=145.79.227.48:4303 , piece_id=JV6233MJNTCGJGNT2CWTEWIIBQOWLGBU6NZUHLAFYQ7JJEC3PTZQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.267431511 (585h57m20.138514546s) [noise=true , address=145.79.227.29:4943 , piece_id=5GZUO6CDZU7HGBWTBKJQYJAH7KPFPYN5RVUYO47QZCK57KONEJZQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.268614433 (585h57m20.139697468s) [noise=true , address=145.79.227.18:4006 , piece_id=GN5BXVK4RN3XATFDDYHSNADTTMXOXQQSZYFZYMTWZODURQJIJUYQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.268958969 (585h57m20.140042004s) [noise=true , address=77.90.31.17:4770 , piece_id=TNR6UIPYMDWWHMYW5H5XL6BISZ3MVQLYOC3CQ4BYWCQH5CK2LACQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.272310848 (585h57m20.143393883s) [noise=true , address=45.91.116.8:4048 , piece_id=5DQCCUSC2LGILPIAF5AU4XL6AMIGT6GZGSFQTOISMGQXDGNA7QVQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.273267069 (585h57m20.144350104s) [noise=true , address=145.79.227.44:4195 , piece_id=R6JI73ZBCXE3BHFR73IVX6HSHDJMYSUMXTDGXQJY5JN6JVSFMIXQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:21.914731767 (585h57m21.785814802s) [upload=513669]: Next returned nil segment Dec 30 01:26:21.914747928 (585h57m21.785830963s) [upload=513669]: Waiting for error group managing segments... Dec 30 01:26:22.246548408 (585h57m22.117631453s) [noise=true , address=145.79.227.42:4123 , piece_id=EZJ6WS7SB232CKFI627CPW3MQF5J4QESBQR2BCYVSIXNGFOCYS2Q, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.252488464 (585h57m22.123571509s) [noise=true , address=145.79.227.48:4303 , piece_id=JV6233MJNTCGJGNT2CWTEWIIBQOWLGBU6NZUHLAFYQ7JJEC3PTZQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.256843233 (585h57m22.127926278s) [noise=true , address=145.79.227.18:4006 , piece_id=GN5BXVK4RN3XATFDDYHSNADTTMXOXQQSZYFZYMTWZODURQJIJUYQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.261027117 (585h57m22.132110152s) [noise=true , address=145.79.227.10:4401 , piece_id=FTVQKHFJ4KZZKH74E3QBOAN5V4BICHGHMDO5A7VQPJFX5X7SCVEQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.262331990 (585h57m22.133415035s) [noise=true , address=145.79.227.47:4267 , piece_id=B62KREWTBELVUDLTSIKRZZKYZ646V5F3GVKFHZVUEVWX7WXXQWOA, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.268404669 (585h57m22.139487704s) [noise=true , address=145.79.227.11:4730 , piece_id=HAFQQSXMMRCFKVIRD3RTHXUQGJ4BVDBWEBSRZHZNOLJLFBZAYZUQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.273582805 (585h57m22.14466584s) [noise=true , address=145.79.227.36:4518 , piece_id=VF5W3AYMAYPDLMGCNS4EUNRRR7WZUHXA43D3PX77GAHVQDEBKMLQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.281079524 (585h57m22.152162579s) [noise=true , address=145.79.227.16:4912 , piece_id=VWLJCBE5UK6DGLE4QMIHBM6ER7GJ4TTFCJSFCXLQR4VN5OHFDLQA, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.283875487 (585h57m22.154958532s) [noise=true , address=145.79.227.5:4223 , piece_id=ZHJIK4L3VLILAHEPW44POAJRQDSSB2JHK5IYFLRK2AWT4VBJ2SAQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.451238134 (585h57m22.32232118s) [noise=true , address=77.90.31.17:4770 , piece_id=TNR6UIPYMDWWHMYW5H5XL6BISZ3MVQLYOC3CQ4BYWCQH5CK2LACQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.494286744 (585h57m22.365369789s) [noise=true , address=45.91.116.6:4975 , piece_id=FLPG4XB4YR6NHDRTKU6TI6ZUEBQBOPD2C3RZLDSBPG6FEAKGN3VA, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.521028605 (585h57m22.39211164s) [noise=true , address=45.91.116.8:4048 , piece_id=5DQCCUSC2LGILPIAF5AU4XL6AMIGT6GZGSFQTOISMGQXDGNA7QVQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.621980587 (585h57m22.493063622s) [noise=true , address=45.91.116.7:4009 , piece_id=M62L3MI2LOHPFX2LNTU7OMN75B2BMD4OAHFN55G2QMVFKDEDQ6TQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.621998781 (585h57m22.493081826s) [seg_pos={0 0}, upload=513669]: Segment reached optimal threshold of 13 pieces. Dec 30 01:26:22.622255239 (585h57m22.493338274s) [noise=true , address=145.79.227.29:4943 , piece_id=5GZUO6CDZU7HGBWTBKJQYJAH7KPFPYN5RVUYO47QZCK57KONEJZQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 30 01:26:22.622264447 (585h57m22.493347482s) [noise=true , address=145.79.227.29:4943 , piece_id=5GZUO6CDZU7HGBWTBKJQYJAH7KPFPYN5RVUYO47QZCK57KONEJZQ, seg_pos={0 0}, upload=513669]: Long tail cancellation due to optimal threshold Dec 30 01:26:22.622497100 (585h57m22.493580135s) [noise=true , address=145.79.227.49:4338 , piece_id=I5GJ4DIMG77FWUHAJKNAUUOLUM3VJCXREN526WAFAIBCIJZ5MHKQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 30 01:26:22.622503321 (585h57m22.493586356s) [noise=true , address=145.79.227.49:4338 , piece_id=I5GJ4DIMG77FWUHAJKNAUUOLUM3VJCXREN526WAFAIBCIJZ5MHKQ, seg_pos={0 0}, upload=513669]: Long tail cancellation due to optimal threshold Dec 30 01:26:22.622604023 (585h57m22.493687058s) [noise=true , address=145.79.227.44:4195 , piece_id=R6JI73ZBCXE3BHFR73IVX6HSHDJMYSUMXTDGXQJY5JN6JVSFMIXQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 01:26:22.622608962 (585h57m22.493692018s) [noise=true , address=145.79.227.44:4195 , piece_id=R6JI73ZBCXE3BHFR73IVX6HSHDJMYSUMXTDGXQJY5JN6JVSFMIXQ, seg_pos={0 0}, upload=513669]: Long tail cancellation due to optimal threshold Dec 30 01:26:22.622627268 (585h57m22.493710303s) [seg_pos={0 0}, upload=513669]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:26:22.622694857 (585h57m22.493777892s) [upload=513669]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:26:22.622700718 (585h57m22.493783753s) [upload=513669]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:26:22.914487837 (585h57m22.785570892s) [upload=513669]: Done uploading segments... Dec 30 01:26:22.914509539 (585h57m22.785592574s) [upload=513669]: Upload finished. err: <nil> Duration: 6.011692753s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 15.395737279s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:27:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 15.395755805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........X......X.X...X....X.X.....X.................XX.....X] duration: 30.002332592s trace: https://jaeger.dev.storj.io/trace/c397a876440ddad FAIL (2025-12-30 01:26:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: TNR6UIPYMDWWHMYW5H5XL6BISZ3MVQLYOC3CQ4BYWCQH5CK2LACQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 5DQCCUSC2LGILPIAF5AU4XL6AMIGT6GZGSFQTOISMGQXDGNA7QVQ) context deadline exceeded ======LOGS====== Dec 30 01:26:20.003606125 (585h57m19.87468917s) [upload=513669]: Starting upload Dec 30 01:26:20.003731103 (585h57m19.874814148s) [upload=513669]: Done starting upload Dec 30 01:26:20.003979346 (585h57m19.875062391s) [upload=513669]: Uploading segments... Dec 30 01:26:20.004025794 (585h57m19.875108829s) [upload=513669]: Got next segment. Inline: false Dec 30 01:26:20.004064858 (585h57m19.875147893s) [upload=513669]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:26:20.004069747 (585h57m19.875152792s) [upload=513669]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:26:20.252636560 (585h57m20.123719615s) [seg_pos={0 0}, upload=513669]: Begin upload segment... Dec 30 01:26:20.252713246 (585h57m20.123796291s) [seg_pos={0 0}, upload=513669]: Done begin upload segment. Dec 30 01:26:20.252739556 (585h57m20.123822601s) [noise=true , address=145.79.227.11:4730 , piece_id=HAFQQSXMMRCFKVIRD3RTHXUQGJ4BVDBWEBSRZHZNOLJLFBZAYZUQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.253864839 (585h57m20.124947874s) [noise=true , address=145.79.227.47:4267 , piece_id=B62KREWTBELVUDLTSIKRZZKYZ646V5F3GVKFHZVUEVWX7WXXQWOA, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.254391901 (585h57m20.125474936s) [noise=true , address=145.79.227.5:4223 , piece_id=ZHJIK4L3VLILAHEPW44POAJRQDSSB2JHK5IYFLRK2AWT4VBJ2SAQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.254847368 (585h57m20.125930413s) [noise=true , address=145.79.227.49:4338 , piece_id=I5GJ4DIMG77FWUHAJKNAUUOLUM3VJCXREN526WAFAIBCIJZ5MHKQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.255253371 (585h57m20.126336416s) [noise=true , address=145.79.227.10:4401 , piece_id=FTVQKHFJ4KZZKH74E3QBOAN5V4BICHGHMDO5A7VQPJFX5X7SCVEQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.255734778 (585h57m20.126817813s) [noise=true , address=145.79.227.16:4912 , piece_id=VWLJCBE5UK6DGLE4QMIHBM6ER7GJ4TTFCJSFCXLQR4VN5OHFDLQA, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.257152405 (585h57m20.12823544s) [noise=true , address=145.79.227.36:4518 , piece_id=VF5W3AYMAYPDLMGCNS4EUNRRR7WZUHXA43D3PX77GAHVQDEBKMLQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.258747412 (585h57m20.129830447s) [noise=true , address=145.79.227.42:4123 , piece_id=EZJ6WS7SB232CKFI627CPW3MQF5J4QESBQR2BCYVSIXNGFOCYS2Q, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.260291070 (585h57m20.131374115s) [noise=true , address=45.91.116.6:4975 , piece_id=FLPG4XB4YR6NHDRTKU6TI6ZUEBQBOPD2C3RZLDSBPG6FEAKGN3VA, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.262997652 (585h57m20.134080687s) [noise=true , address=45.91.116.7:4009 , piece_id=M62L3MI2LOHPFX2LNTU7OMN75B2BMD4OAHFN55G2QMVFKDEDQ6TQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.265624722 (585h57m20.136707767s) [noise=true , address=145.79.227.48:4303 , piece_id=JV6233MJNTCGJGNT2CWTEWIIBQOWLGBU6NZUHLAFYQ7JJEC3PTZQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.267431511 (585h57m20.138514546s) [noise=true , address=145.79.227.29:4943 , piece_id=5GZUO6CDZU7HGBWTBKJQYJAH7KPFPYN5RVUYO47QZCK57KONEJZQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.268614433 (585h57m20.139697468s) [noise=true , address=145.79.227.18:4006 , piece_id=GN5BXVK4RN3XATFDDYHSNADTTMXOXQQSZYFZYMTWZODURQJIJUYQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.268958969 (585h57m20.140042004s) [noise=true , address=77.90.31.17:4770 , piece_id=TNR6UIPYMDWWHMYW5H5XL6BISZ3MVQLYOC3CQ4BYWCQH5CK2LACQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.272310848 (585h57m20.143393883s) [noise=true , address=45.91.116.8:4048 , piece_id=5DQCCUSC2LGILPIAF5AU4XL6AMIGT6GZGSFQTOISMGQXDGNA7QVQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:20.273267069 (585h57m20.144350104s) [noise=true , address=145.79.227.44:4195 , piece_id=R6JI73ZBCXE3BHFR73IVX6HSHDJMYSUMXTDGXQJY5JN6JVSFMIXQ, seg_pos={0 0}, upload=513669]: Uploading piece... Dec 30 01:26:21.914731767 (585h57m21.785814802s) [upload=513669]: Next returned nil segment Dec 30 01:26:21.914747928 (585h57m21.785830963s) [upload=513669]: Waiting for error group managing segments... Dec 30 01:26:22.246548408 (585h57m22.117631453s) [noise=true , address=145.79.227.42:4123 , piece_id=EZJ6WS7SB232CKFI627CPW3MQF5J4QESBQR2BCYVSIXNGFOCYS2Q, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.252488464 (585h57m22.123571509s) [noise=true , address=145.79.227.48:4303 , piece_id=JV6233MJNTCGJGNT2CWTEWIIBQOWLGBU6NZUHLAFYQ7JJEC3PTZQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.256843233 (585h57m22.127926278s) [noise=true , address=145.79.227.18:4006 , piece_id=GN5BXVK4RN3XATFDDYHSNADTTMXOXQQSZYFZYMTWZODURQJIJUYQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.261027117 (585h57m22.132110152s) [noise=true , address=145.79.227.10:4401 , piece_id=FTVQKHFJ4KZZKH74E3QBOAN5V4BICHGHMDO5A7VQPJFX5X7SCVEQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.262331990 (585h57m22.133415035s) [noise=true , address=145.79.227.47:4267 , piece_id=B62KREWTBELVUDLTSIKRZZKYZ646V5F3GVKFHZVUEVWX7WXXQWOA, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.268404669 (585h57m22.139487704s) [noise=true , address=145.79.227.11:4730 , piece_id=HAFQQSXMMRCFKVIRD3RTHXUQGJ4BVDBWEBSRZHZNOLJLFBZAYZUQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.273582805 (585h57m22.14466584s) [noise=true , address=145.79.227.36:4518 , piece_id=VF5W3AYMAYPDLMGCNS4EUNRRR7WZUHXA43D3PX77GAHVQDEBKMLQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.281079524 (585h57m22.152162579s) [noise=true , address=145.79.227.16:4912 , piece_id=VWLJCBE5UK6DGLE4QMIHBM6ER7GJ4TTFCJSFCXLQR4VN5OHFDLQA, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.283875487 (585h57m22.154958532s) [noise=true , address=145.79.227.5:4223 , piece_id=ZHJIK4L3VLILAHEPW44POAJRQDSSB2JHK5IYFLRK2AWT4VBJ2SAQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.451238134 (585h57m22.32232118s) [noise=true , address=77.90.31.17:4770 , piece_id=TNR6UIPYMDWWHMYW5H5XL6BISZ3MVQLYOC3CQ4BYWCQH5CK2LACQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.494286744 (585h57m22.365369789s) [noise=true , address=45.91.116.6:4975 , piece_id=FLPG4XB4YR6NHDRTKU6TI6ZUEBQBOPD2C3RZLDSBPG6FEAKGN3VA, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.521028605 (585h57m22.39211164s) [noise=true , address=45.91.116.8:4048 , piece_id=5DQCCUSC2LGILPIAF5AU4XL6AMIGT6GZGSFQTOISMGQXDGNA7QVQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.621980587 (585h57m22.493063622s) [noise=true , address=45.91.116.7:4009 , piece_id=M62L3MI2LOHPFX2LNTU7OMN75B2BMD4OAHFN55G2QMVFKDEDQ6TQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: <nil> Dec 30 01:26:22.621998781 (585h57m22.493081826s) [seg_pos={0 0}, upload=513669]: Segment reached optimal threshold of 13 pieces. Dec 30 01:26:22.622255239 (585h57m22.493338274s) [noise=true , address=145.79.227.29:4943 , piece_id=5GZUO6CDZU7HGBWTBKJQYJAH7KPFPYN5RVUYO47QZCK57KONEJZQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 30 01:26:22.622264447 (585h57m22.493347482s) [noise=true , address=145.79.227.29:4943 , piece_id=5GZUO6CDZU7HGBWTBKJQYJAH7KPFPYN5RVUYO47QZCK57KONEJZQ, seg_pos={0 0}, upload=513669]: Long tail cancellation due to optimal threshold Dec 30 01:26:22.622497100 (585h57m22.493580135s) [noise=true , address=145.79.227.49:4338 , piece_id=I5GJ4DIMG77FWUHAJKNAUUOLUM3VJCXREN526WAFAIBCIJZ5MHKQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 30 01:26:22.622503321 (585h57m22.493586356s) [noise=true , address=145.79.227.49:4338 , piece_id=I5GJ4DIMG77FWUHAJKNAUUOLUM3VJCXREN526WAFAIBCIJZ5MHKQ, seg_pos={0 0}, upload=513669]: Long tail cancellation due to optimal threshold Dec 30 01:26:22.622604023 (585h57m22.493687058s) [noise=true , address=145.79.227.44:4195 , piece_id=R6JI73ZBCXE3BHFR73IVX6HSHDJMYSUMXTDGXQJY5JN6JVSFMIXQ, seg_pos={0 0}, upload=513669]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 01:26:22.622608962 (585h57m22.493692018s) [noise=true , address=145.79.227.44:4195 , piece_id=R6JI73ZBCXE3BHFR73IVX6HSHDJMYSUMXTDGXQJY5JN6JVSFMIXQ, seg_pos={0 0}, upload=513669]: Long tail cancellation due to optimal threshold Dec 30 01:26:22.622627268 (585h57m22.493710303s) [seg_pos={0 0}, upload=513669]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:26:22.622694857 (585h57m22.493777892s) [upload=513669]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:26:22.622700718 (585h57m22.493783753s) [upload=513669]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:26:22.914487837 (585h57m22.785570892s) [upload=513669]: Done uploading segments... Dec 30 01:26:22.914509539 (585h57m22.785592574s) [upload=513669]: Upload finished. err: <nil> Duration: 30.002352891s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 15.35995493s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:26:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 15.359985527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........X......X.X...X....X.X.....X.................XX.....] duration: 6.464199391s trace: https://jaeger.dev.storj.io/trace/3af02f72f9549d7 FAIL (2025-12-30 01:26:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: N2FAP6WJI644OBBSDWXYWXZF7YLUWTLKOKIEHG6ANN4ZJDKNY7JQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: OLQCP377TKM6YUJYQ2EZWU33AZ6NNYYKVGQMIO53EHWZHODXMVQQ) context deadline exceeded ======LOGS====== Dec 30 01:23:14.158861942 (585h54m14.029944987s) [upload=513620]: Starting upload Dec 30 01:23:14.158944258 (585h54m14.030027314s) [upload=513620]: Done starting upload Dec 30 01:23:14.159068506 (585h54m14.030151541s) [upload=513620]: Uploading segments... Dec 30 01:23:14.159156653 (585h54m14.030239688s) [upload=513620]: Got next segment. Inline: false Dec 30 01:23:14.159190828 (585h54m14.030273863s) [upload=513620]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:23:14.159196008 (585h54m14.030279053s) [upload=513620]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:23:14.405670689 (585h54m14.276753724s) [seg_pos={0 0}, upload=513620]: Begin upload segment... Dec 30 01:23:14.405717478 (585h54m14.276800513s) [seg_pos={0 0}, upload=513620]: Done begin upload segment. Dec 30 01:23:14.405741774 (585h54m14.276824809s) [noise=true , address=45.91.116.4:4939 , piece_id=6ZOKYXBYAVZ6GOEBGJIXKGRBA2MUTCM6XF7UMZYD3CHW2OFGRNFQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.405881039 (585h54m14.276964074s) [noise=true , address=145.79.227.7:4295 , piece_id=XEUHRX3UMQ3MLLRNKKSBXZJ2KCS5QRLYFLSDV53MR2JCUNDVYCXA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.406599146 (585h54m14.277682181s) [noise=true , address=145.79.227.52:4445 , piece_id=VZYM4WDHWOVC2TRZFEDMWJXA4HRCKALIRTPLBLJXKNA6SCGWDFBA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.412496261 (585h54m14.283579296s) [noise=true , address=77.90.31.10:4520 , piece_id=LSCPY3E6IB6Y22KX4D24HQ756IVBQCZCZH6NTIOTU23UDQF5GDGA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.412589529 (585h54m14.283672574s) [noise=true , address=77.90.31.16:4735 , piece_id=CJC6PCJ2SR37UH2RCC2QW7566JRTGCY2BWQGFRDJOYCJW3ITSGWQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.414307297 (585h54m14.285390332s) [noise=true , address=145.79.227.17:4971 , piece_id=D2LB4XJUW2T43R33NWNKAF7GSVO5263YH52MIXCODTSFENWSYTIA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.414401076 (585h54m14.285484111s) [noise=true , address=145.79.227.10:4402 , piece_id=IUQ37VKBNTJFHXWSM5JQFZ66ITV5RLODUYH7YFXAHCAMQU53Z3BQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.415385491 (585h54m14.286468526s) [noise=true , address=145.79.227.8:4329 , piece_id=ONUB2LL5SNPIE542HGDODHGW4K6UDZM5YJCPO6KQ3VTZAOAXJX3Q, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.415893828 (585h54m14.286976863s) [noise=true , address=145.79.227.20:4439 , piece_id=RKZO5TT33LMIBOOZIB3IULHHLH3ERVQL3I6JAILFZ2O4X2MIBLDA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.416403217 (585h54m14.287486252s) [noise=true , address=145.79.227.36:4518 , piece_id=ZKDV72FHKAQCSM6HLXAOL6THAIT4UFWH5C2KY7WT76OVDPWAHXLQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.417390857 (585h54m14.288473893s) [noise=true , address=45.91.116.12:4226 , piece_id=ZPVJGN2AREUUJZEW3AT5HOXT73I6FAFVEO43L45EWKVZKGCC6MLA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.417419492 (585h54m14.288502537s) [noise=true , address=145.79.227.15:4874 , piece_id=RLMXGSIKFAUAAUL5HORQ2COJ5736N2IAKYHPMVXRKR77RDQ6QPFA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.418498746 (585h54m14.289581781s) [noise=true , address=45.91.116.6:4974 , piece_id=N2FAP6WJI644OBBSDWXYWXZF7YLUWTLKOKIEHG6ANN4ZJDKNY7JQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.418799899 (585h54m14.289882944s) [noise=true , address=45.91.116.7:4010 , piece_id=C2RGGVV3ETAESIEAUTSGGEXZETXB3BXQFRKMS4TWLSW7IYECY4FQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.420225372 (585h54m14.291308408s) [noise=true , address=45.91.116.13:4155 , piece_id=OLQCP377TKM6YUJYQ2EZWU33AZ6NNYYKVGQMIO53EHWZHODXMVQQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.420748388 (585h54m14.291831423s) [noise=true , address=145.79.227.46:4602 , piece_id=MP4MSU2UMX7P3HRZIRPLI6NCUCGJ475LMXEW3PEWYZ74UH3TNNVQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:16.113634421 (585h54m15.984717466s) [upload=513620]: Next returned nil segment Dec 30 01:23:16.113664088 (585h54m15.984747123s) [upload=513620]: Waiting for error group managing segments... Dec 30 01:23:16.399775522 (585h54m16.270858558s) [noise=true , address=145.79.227.52:4445 , piece_id=VZYM4WDHWOVC2TRZFEDMWJXA4HRCKALIRTPLBLJXKNA6SCGWDFBA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.405286742 (585h54m16.276369778s) [noise=true , address=145.79.227.20:4439 , piece_id=RKZO5TT33LMIBOOZIB3IULHHLH3ERVQL3I6JAILFZ2O4X2MIBLDA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.405369731 (585h54m16.276452766s) [noise=true , address=145.79.227.36:4518 , piece_id=ZKDV72FHKAQCSM6HLXAOL6THAIT4UFWH5C2KY7WT76OVDPWAHXLQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.407602491 (585h54m16.278685536s) [noise=true , address=145.79.227.17:4971 , piece_id=D2LB4XJUW2T43R33NWNKAF7GSVO5263YH52MIXCODTSFENWSYTIA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.412068081 (585h54m16.283151126s) [noise=true , address=145.79.227.8:4329 , piece_id=ONUB2LL5SNPIE542HGDODHGW4K6UDZM5YJCPO6KQ3VTZAOAXJX3Q, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.412197457 (585h54m16.283280492s) [noise=true , address=145.79.227.15:4874 , piece_id=RLMXGSIKFAUAAUL5HORQ2COJ5736N2IAKYHPMVXRKR77RDQ6QPFA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.420180532 (585h54m16.291263567s) [noise=true , address=145.79.227.10:4402 , piece_id=IUQ37VKBNTJFHXWSM5JQFZ66ITV5RLODUYH7YFXAHCAMQU53Z3BQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.426707214 (585h54m16.29779026s) [noise=true , address=145.79.227.46:4602 , piece_id=MP4MSU2UMX7P3HRZIRPLI6NCUCGJ475LMXEW3PEWYZ74UH3TNNVQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.586338838 (585h54m16.457421883s) [noise=true , address=77.90.31.16:4735 , piece_id=CJC6PCJ2SR37UH2RCC2QW7566JRTGCY2BWQGFRDJOYCJW3ITSGWQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.662930924 (585h54m16.534013969s) [noise=true , address=45.91.116.6:4974 , piece_id=N2FAP6WJI644OBBSDWXYWXZF7YLUWTLKOKIEHG6ANN4ZJDKNY7JQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.680218899 (585h54m16.551301934s) [noise=true , address=45.91.116.12:4226 , piece_id=ZPVJGN2AREUUJZEW3AT5HOXT73I6FAFVEO43L45EWKVZKGCC6MLA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.695093331 (585h54m16.566176367s) [noise=true , address=45.91.116.4:4939 , piece_id=6ZOKYXBYAVZ6GOEBGJIXKGRBA2MUTCM6XF7UMZYD3CHW2OFGRNFQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.717354203 (585h54m16.588437248s) [noise=true , address=45.91.116.13:4155 , piece_id=OLQCP377TKM6YUJYQ2EZWU33AZ6NNYYKVGQMIO53EHWZHODXMVQQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.717377557 (585h54m16.588460602s) [seg_pos={0 0}, upload=513620]: Segment reached optimal threshold of 13 pieces. Dec 30 01:23:16.717574872 (585h54m16.588657907s) [noise=true , address=45.91.116.7:4010 , piece_id=C2RGGVV3ETAESIEAUTSGGEXZETXB3BXQFRKMS4TWLSW7IYECY4FQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 30 01:23:16.717581765 (585h54m16.58866481s) [noise=true , address=45.91.116.7:4010 , piece_id=C2RGGVV3ETAESIEAUTSGGEXZETXB3BXQFRKMS4TWLSW7IYECY4FQ, seg_pos={0 0}, upload=513620]: Long tail cancellation due to optimal threshold Dec 30 01:23:16.717711633 (585h54m16.588794668s) [noise=true , address=145.79.227.7:4295 , piece_id=XEUHRX3UMQ3MLLRNKKSBXZJ2KCS5QRLYFLSDV53MR2JCUNDVYCXA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Dec 30 01:23:16.717720750 (585h54m16.588803785s) [noise=true , address=145.79.227.7:4295 , piece_id=XEUHRX3UMQ3MLLRNKKSBXZJ2KCS5QRLYFLSDV53MR2JCUNDVYCXA, seg_pos={0 0}, upload=513620]: Long tail cancellation due to optimal threshold Dec 30 01:23:16.717976106 (585h54m16.589059151s) [noise=true , address=77.90.31.10:4520 , piece_id=LSCPY3E6IB6Y22KX4D24HQ756IVBQCZCZH6NTIOTU23UDQF5GDGA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 30 01:23:16.717992227 (585h54m16.589075272s) [noise=true , address=77.90.31.10:4520 , piece_id=LSCPY3E6IB6Y22KX4D24HQ756IVBQCZCZH6NTIOTU23UDQF5GDGA, seg_pos={0 0}, upload=513620]: Long tail cancellation due to optimal threshold Dec 30 01:23:16.718026112 (585h54m16.589109147s) [seg_pos={0 0}, upload=513620]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:23:16.718109991 (585h54m16.589193036s) [upload=513620]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:23:16.718117305 (585h54m16.58920034s) [upload=513620]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:23:17.016443429 (585h54m16.887526474s) [upload=513620]: Done uploading segments... Dec 30 01:23:17.016467595 (585h54m16.88755064s) [upload=513620]: Upload finished. err: <nil> Duration: 6.46422983s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.........X......X.X...X....X.X.....X.................XX....] duration: 5.901394257s trace: https://jaeger.dev.storj.io/trace/3af02f72f9549d7 FAIL (2025-12-30 01:25:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: N2FAP6WJI644OBBSDWXYWXZF7YLUWTLKOKIEHG6ANN4ZJDKNY7JQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: OLQCP377TKM6YUJYQ2EZWU33AZ6NNYYKVGQMIO53EHWZHODXMVQQ) context deadline exceeded ======LOGS====== Dec 30 01:23:14.158861942 (585h54m14.029944987s) [upload=513620]: Starting upload Dec 30 01:23:14.158944258 (585h54m14.030027314s) [upload=513620]: Done starting upload Dec 30 01:23:14.159068506 (585h54m14.030151541s) [upload=513620]: Uploading segments... Dec 30 01:23:14.159156653 (585h54m14.030239688s) [upload=513620]: Got next segment. Inline: false Dec 30 01:23:14.159190828 (585h54m14.030273863s) [upload=513620]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:23:14.159196008 (585h54m14.030279053s) [upload=513620]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:23:14.405670689 (585h54m14.276753724s) [seg_pos={0 0}, upload=513620]: Begin upload segment... Dec 30 01:23:14.405717478 (585h54m14.276800513s) [seg_pos={0 0}, upload=513620]: Done begin upload segment. Dec 30 01:23:14.405741774 (585h54m14.276824809s) [noise=true , address=45.91.116.4:4939 , piece_id=6ZOKYXBYAVZ6GOEBGJIXKGRBA2MUTCM6XF7UMZYD3CHW2OFGRNFQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.405881039 (585h54m14.276964074s) [noise=true , address=145.79.227.7:4295 , piece_id=XEUHRX3UMQ3MLLRNKKSBXZJ2KCS5QRLYFLSDV53MR2JCUNDVYCXA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.406599146 (585h54m14.277682181s) [noise=true , address=145.79.227.52:4445 , piece_id=VZYM4WDHWOVC2TRZFEDMWJXA4HRCKALIRTPLBLJXKNA6SCGWDFBA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.412496261 (585h54m14.283579296s) [noise=true , address=77.90.31.10:4520 , piece_id=LSCPY3E6IB6Y22KX4D24HQ756IVBQCZCZH6NTIOTU23UDQF5GDGA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.412589529 (585h54m14.283672574s) [noise=true , address=77.90.31.16:4735 , piece_id=CJC6PCJ2SR37UH2RCC2QW7566JRTGCY2BWQGFRDJOYCJW3ITSGWQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.414307297 (585h54m14.285390332s) [noise=true , address=145.79.227.17:4971 , piece_id=D2LB4XJUW2T43R33NWNKAF7GSVO5263YH52MIXCODTSFENWSYTIA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.414401076 (585h54m14.285484111s) [noise=true , address=145.79.227.10:4402 , piece_id=IUQ37VKBNTJFHXWSM5JQFZ66ITV5RLODUYH7YFXAHCAMQU53Z3BQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.415385491 (585h54m14.286468526s) [noise=true , address=145.79.227.8:4329 , piece_id=ONUB2LL5SNPIE542HGDODHGW4K6UDZM5YJCPO6KQ3VTZAOAXJX3Q, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.415893828 (585h54m14.286976863s) [noise=true , address=145.79.227.20:4439 , piece_id=RKZO5TT33LMIBOOZIB3IULHHLH3ERVQL3I6JAILFZ2O4X2MIBLDA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.416403217 (585h54m14.287486252s) [noise=true , address=145.79.227.36:4518 , piece_id=ZKDV72FHKAQCSM6HLXAOL6THAIT4UFWH5C2KY7WT76OVDPWAHXLQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.417390857 (585h54m14.288473893s) [noise=true , address=45.91.116.12:4226 , piece_id=ZPVJGN2AREUUJZEW3AT5HOXT73I6FAFVEO43L45EWKVZKGCC6MLA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.417419492 (585h54m14.288502537s) [noise=true , address=145.79.227.15:4874 , piece_id=RLMXGSIKFAUAAUL5HORQ2COJ5736N2IAKYHPMVXRKR77RDQ6QPFA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.418498746 (585h54m14.289581781s) [noise=true , address=45.91.116.6:4974 , piece_id=N2FAP6WJI644OBBSDWXYWXZF7YLUWTLKOKIEHG6ANN4ZJDKNY7JQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.418799899 (585h54m14.289882944s) [noise=true , address=45.91.116.7:4010 , piece_id=C2RGGVV3ETAESIEAUTSGGEXZETXB3BXQFRKMS4TWLSW7IYECY4FQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.420225372 (585h54m14.291308408s) [noise=true , address=45.91.116.13:4155 , piece_id=OLQCP377TKM6YUJYQ2EZWU33AZ6NNYYKVGQMIO53EHWZHODXMVQQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.420748388 (585h54m14.291831423s) [noise=true , address=145.79.227.46:4602 , piece_id=MP4MSU2UMX7P3HRZIRPLI6NCUCGJ475LMXEW3PEWYZ74UH3TNNVQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:16.113634421 (585h54m15.984717466s) [upload=513620]: Next returned nil segment Dec 30 01:23:16.113664088 (585h54m15.984747123s) [upload=513620]: Waiting for error group managing segments... Dec 30 01:23:16.399775522 (585h54m16.270858558s) [noise=true , address=145.79.227.52:4445 , piece_id=VZYM4WDHWOVC2TRZFEDMWJXA4HRCKALIRTPLBLJXKNA6SCGWDFBA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.405286742 (585h54m16.276369778s) [noise=true , address=145.79.227.20:4439 , piece_id=RKZO5TT33LMIBOOZIB3IULHHLH3ERVQL3I6JAILFZ2O4X2MIBLDA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.405369731 (585h54m16.276452766s) [noise=true , address=145.79.227.36:4518 , piece_id=ZKDV72FHKAQCSM6HLXAOL6THAIT4UFWH5C2KY7WT76OVDPWAHXLQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.407602491 (585h54m16.278685536s) [noise=true , address=145.79.227.17:4971 , piece_id=D2LB4XJUW2T43R33NWNKAF7GSVO5263YH52MIXCODTSFENWSYTIA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.412068081 (585h54m16.283151126s) [noise=true , address=145.79.227.8:4329 , piece_id=ONUB2LL5SNPIE542HGDODHGW4K6UDZM5YJCPO6KQ3VTZAOAXJX3Q, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.412197457 (585h54m16.283280492s) [noise=true , address=145.79.227.15:4874 , piece_id=RLMXGSIKFAUAAUL5HORQ2COJ5736N2IAKYHPMVXRKR77RDQ6QPFA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.420180532 (585h54m16.291263567s) [noise=true , address=145.79.227.10:4402 , piece_id=IUQ37VKBNTJFHXWSM5JQFZ66ITV5RLODUYH7YFXAHCAMQU53Z3BQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.426707214 (585h54m16.29779026s) [noise=true , address=145.79.227.46:4602 , piece_id=MP4MSU2UMX7P3HRZIRPLI6NCUCGJ475LMXEW3PEWYZ74UH3TNNVQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.586338838 (585h54m16.457421883s) [noise=true , address=77.90.31.16:4735 , piece_id=CJC6PCJ2SR37UH2RCC2QW7566JRTGCY2BWQGFRDJOYCJW3ITSGWQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.662930924 (585h54m16.534013969s) [noise=true , address=45.91.116.6:4974 , piece_id=N2FAP6WJI644OBBSDWXYWXZF7YLUWTLKOKIEHG6ANN4ZJDKNY7JQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.680218899 (585h54m16.551301934s) [noise=true , address=45.91.116.12:4226 , piece_id=ZPVJGN2AREUUJZEW3AT5HOXT73I6FAFVEO43L45EWKVZKGCC6MLA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.695093331 (585h54m16.566176367s) [noise=true , address=45.91.116.4:4939 , piece_id=6ZOKYXBYAVZ6GOEBGJIXKGRBA2MUTCM6XF7UMZYD3CHW2OFGRNFQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.717354203 (585h54m16.588437248s) [noise=true , address=45.91.116.13:4155 , piece_id=OLQCP377TKM6YUJYQ2EZWU33AZ6NNYYKVGQMIO53EHWZHODXMVQQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.717377557 (585h54m16.588460602s) [seg_pos={0 0}, upload=513620]: Segment reached optimal threshold of 13 pieces. Dec 30 01:23:16.717574872 (585h54m16.588657907s) [noise=true , address=45.91.116.7:4010 , piece_id=C2RGGVV3ETAESIEAUTSGGEXZETXB3BXQFRKMS4TWLSW7IYECY4FQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 30 01:23:16.717581765 (585h54m16.58866481s) [noise=true , address=45.91.116.7:4010 , piece_id=C2RGGVV3ETAESIEAUTSGGEXZETXB3BXQFRKMS4TWLSW7IYECY4FQ, seg_pos={0 0}, upload=513620]: Long tail cancellation due to optimal threshold Dec 30 01:23:16.717711633 (585h54m16.588794668s) [noise=true , address=145.79.227.7:4295 , piece_id=XEUHRX3UMQ3MLLRNKKSBXZJ2KCS5QRLYFLSDV53MR2JCUNDVYCXA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Dec 30 01:23:16.717720750 (585h54m16.588803785s) [noise=true , address=145.79.227.7:4295 , piece_id=XEUHRX3UMQ3MLLRNKKSBXZJ2KCS5QRLYFLSDV53MR2JCUNDVYCXA, seg_pos={0 0}, upload=513620]: Long tail cancellation due to optimal threshold Dec 30 01:23:16.717976106 (585h54m16.589059151s) [noise=true , address=77.90.31.10:4520 , piece_id=LSCPY3E6IB6Y22KX4D24HQ756IVBQCZCZH6NTIOTU23UDQF5GDGA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 30 01:23:16.717992227 (585h54m16.589075272s) [noise=true , address=77.90.31.10:4520 , piece_id=LSCPY3E6IB6Y22KX4D24HQ756IVBQCZCZH6NTIOTU23UDQF5GDGA, seg_pos={0 0}, upload=513620]: Long tail cancellation due to optimal threshold Dec 30 01:23:16.718026112 (585h54m16.589109147s) [seg_pos={0 0}, upload=513620]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:23:16.718109991 (585h54m16.589193036s) [upload=513620]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:23:16.718117305 (585h54m16.58920034s) [upload=513620]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:23:17.016443429 (585h54m16.887526474s) [upload=513620]: Done uploading segments... Dec 30 01:23:17.016467595 (585h54m16.88755064s) [upload=513620]: Upload finished. err: <nil> Duration: 5.901419685s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X.........X......X.X...X....X.X.....X.................XX...] duration: 22.983712772s trace: https://jaeger.dev.storj.io/trace/3af02f72f9549d7 FAIL (2025-12-30 01:25:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: N2FAP6WJI644OBBSDWXYWXZF7YLUWTLKOKIEHG6ANN4ZJDKNY7JQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: OLQCP377TKM6YUJYQ2EZWU33AZ6NNYYKVGQMIO53EHWZHODXMVQQ) context deadline exceeded ======LOGS====== Dec 30 01:23:14.158861942 (585h54m14.029944987s) [upload=513620]: Starting upload Dec 30 01:23:14.158944258 (585h54m14.030027314s) [upload=513620]: Done starting upload Dec 30 01:23:14.159068506 (585h54m14.030151541s) [upload=513620]: Uploading segments... Dec 30 01:23:14.159156653 (585h54m14.030239688s) [upload=513620]: Got next segment. Inline: false Dec 30 01:23:14.159190828 (585h54m14.030273863s) [upload=513620]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:23:14.159196008 (585h54m14.030279053s) [upload=513620]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:23:14.405670689 (585h54m14.276753724s) [seg_pos={0 0}, upload=513620]: Begin upload segment... Dec 30 01:23:14.405717478 (585h54m14.276800513s) [seg_pos={0 0}, upload=513620]: Done begin upload segment. Dec 30 01:23:14.405741774 (585h54m14.276824809s) [noise=true , address=45.91.116.4:4939 , piece_id=6ZOKYXBYAVZ6GOEBGJIXKGRBA2MUTCM6XF7UMZYD3CHW2OFGRNFQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.405881039 (585h54m14.276964074s) [noise=true , address=145.79.227.7:4295 , piece_id=XEUHRX3UMQ3MLLRNKKSBXZJ2KCS5QRLYFLSDV53MR2JCUNDVYCXA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.406599146 (585h54m14.277682181s) [noise=true , address=145.79.227.52:4445 , piece_id=VZYM4WDHWOVC2TRZFEDMWJXA4HRCKALIRTPLBLJXKNA6SCGWDFBA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.412496261 (585h54m14.283579296s) [noise=true , address=77.90.31.10:4520 , piece_id=LSCPY3E6IB6Y22KX4D24HQ756IVBQCZCZH6NTIOTU23UDQF5GDGA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.412589529 (585h54m14.283672574s) [noise=true , address=77.90.31.16:4735 , piece_id=CJC6PCJ2SR37UH2RCC2QW7566JRTGCY2BWQGFRDJOYCJW3ITSGWQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.414307297 (585h54m14.285390332s) [noise=true , address=145.79.227.17:4971 , piece_id=D2LB4XJUW2T43R33NWNKAF7GSVO5263YH52MIXCODTSFENWSYTIA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.414401076 (585h54m14.285484111s) [noise=true , address=145.79.227.10:4402 , piece_id=IUQ37VKBNTJFHXWSM5JQFZ66ITV5RLODUYH7YFXAHCAMQU53Z3BQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.415385491 (585h54m14.286468526s) [noise=true , address=145.79.227.8:4329 , piece_id=ONUB2LL5SNPIE542HGDODHGW4K6UDZM5YJCPO6KQ3VTZAOAXJX3Q, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.415893828 (585h54m14.286976863s) [noise=true , address=145.79.227.20:4439 , piece_id=RKZO5TT33LMIBOOZIB3IULHHLH3ERVQL3I6JAILFZ2O4X2MIBLDA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.416403217 (585h54m14.287486252s) [noise=true , address=145.79.227.36:4518 , piece_id=ZKDV72FHKAQCSM6HLXAOL6THAIT4UFWH5C2KY7WT76OVDPWAHXLQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.417390857 (585h54m14.288473893s) [noise=true , address=45.91.116.12:4226 , piece_id=ZPVJGN2AREUUJZEW3AT5HOXT73I6FAFVEO43L45EWKVZKGCC6MLA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.417419492 (585h54m14.288502537s) [noise=true , address=145.79.227.15:4874 , piece_id=RLMXGSIKFAUAAUL5HORQ2COJ5736N2IAKYHPMVXRKR77RDQ6QPFA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.418498746 (585h54m14.289581781s) [noise=true , address=45.91.116.6:4974 , piece_id=N2FAP6WJI644OBBSDWXYWXZF7YLUWTLKOKIEHG6ANN4ZJDKNY7JQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.418799899 (585h54m14.289882944s) [noise=true , address=45.91.116.7:4010 , piece_id=C2RGGVV3ETAESIEAUTSGGEXZETXB3BXQFRKMS4TWLSW7IYECY4FQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.420225372 (585h54m14.291308408s) [noise=true , address=45.91.116.13:4155 , piece_id=OLQCP377TKM6YUJYQ2EZWU33AZ6NNYYKVGQMIO53EHWZHODXMVQQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.420748388 (585h54m14.291831423s) [noise=true , address=145.79.227.46:4602 , piece_id=MP4MSU2UMX7P3HRZIRPLI6NCUCGJ475LMXEW3PEWYZ74UH3TNNVQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:16.113634421 (585h54m15.984717466s) [upload=513620]: Next returned nil segment Dec 30 01:23:16.113664088 (585h54m15.984747123s) [upload=513620]: Waiting for error group managing segments... Dec 30 01:23:16.399775522 (585h54m16.270858558s) [noise=true , address=145.79.227.52:4445 , piece_id=VZYM4WDHWOVC2TRZFEDMWJXA4HRCKALIRTPLBLJXKNA6SCGWDFBA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.405286742 (585h54m16.276369778s) [noise=true , address=145.79.227.20:4439 , piece_id=RKZO5TT33LMIBOOZIB3IULHHLH3ERVQL3I6JAILFZ2O4X2MIBLDA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.405369731 (585h54m16.276452766s) [noise=true , address=145.79.227.36:4518 , piece_id=ZKDV72FHKAQCSM6HLXAOL6THAIT4UFWH5C2KY7WT76OVDPWAHXLQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.407602491 (585h54m16.278685536s) [noise=true , address=145.79.227.17:4971 , piece_id=D2LB4XJUW2T43R33NWNKAF7GSVO5263YH52MIXCODTSFENWSYTIA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.412068081 (585h54m16.283151126s) [noise=true , address=145.79.227.8:4329 , piece_id=ONUB2LL5SNPIE542HGDODHGW4K6UDZM5YJCPO6KQ3VTZAOAXJX3Q, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.412197457 (585h54m16.283280492s) [noise=true , address=145.79.227.15:4874 , piece_id=RLMXGSIKFAUAAUL5HORQ2COJ5736N2IAKYHPMVXRKR77RDQ6QPFA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.420180532 (585h54m16.291263567s) [noise=true , address=145.79.227.10:4402 , piece_id=IUQ37VKBNTJFHXWSM5JQFZ66ITV5RLODUYH7YFXAHCAMQU53Z3BQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.426707214 (585h54m16.29779026s) [noise=true , address=145.79.227.46:4602 , piece_id=MP4MSU2UMX7P3HRZIRPLI6NCUCGJ475LMXEW3PEWYZ74UH3TNNVQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.586338838 (585h54m16.457421883s) [noise=true , address=77.90.31.16:4735 , piece_id=CJC6PCJ2SR37UH2RCC2QW7566JRTGCY2BWQGFRDJOYCJW3ITSGWQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.662930924 (585h54m16.534013969s) [noise=true , address=45.91.116.6:4974 , piece_id=N2FAP6WJI644OBBSDWXYWXZF7YLUWTLKOKIEHG6ANN4ZJDKNY7JQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.680218899 (585h54m16.551301934s) [noise=true , address=45.91.116.12:4226 , piece_id=ZPVJGN2AREUUJZEW3AT5HOXT73I6FAFVEO43L45EWKVZKGCC6MLA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.695093331 (585h54m16.566176367s) [noise=true , address=45.91.116.4:4939 , piece_id=6ZOKYXBYAVZ6GOEBGJIXKGRBA2MUTCM6XF7UMZYD3CHW2OFGRNFQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.717354203 (585h54m16.588437248s) [noise=true , address=45.91.116.13:4155 , piece_id=OLQCP377TKM6YUJYQ2EZWU33AZ6NNYYKVGQMIO53EHWZHODXMVQQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.717377557 (585h54m16.588460602s) [seg_pos={0 0}, upload=513620]: Segment reached optimal threshold of 13 pieces. Dec 30 01:23:16.717574872 (585h54m16.588657907s) [noise=true , address=45.91.116.7:4010 , piece_id=C2RGGVV3ETAESIEAUTSGGEXZETXB3BXQFRKMS4TWLSW7IYECY4FQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 30 01:23:16.717581765 (585h54m16.58866481s) [noise=true , address=45.91.116.7:4010 , piece_id=C2RGGVV3ETAESIEAUTSGGEXZETXB3BXQFRKMS4TWLSW7IYECY4FQ, seg_pos={0 0}, upload=513620]: Long tail cancellation due to optimal threshold Dec 30 01:23:16.717711633 (585h54m16.588794668s) [noise=true , address=145.79.227.7:4295 , piece_id=XEUHRX3UMQ3MLLRNKKSBXZJ2KCS5QRLYFLSDV53MR2JCUNDVYCXA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Dec 30 01:23:16.717720750 (585h54m16.588803785s) [noise=true , address=145.79.227.7:4295 , piece_id=XEUHRX3UMQ3MLLRNKKSBXZJ2KCS5QRLYFLSDV53MR2JCUNDVYCXA, seg_pos={0 0}, upload=513620]: Long tail cancellation due to optimal threshold Dec 30 01:23:16.717976106 (585h54m16.589059151s) [noise=true , address=77.90.31.10:4520 , piece_id=LSCPY3E6IB6Y22KX4D24HQ756IVBQCZCZH6NTIOTU23UDQF5GDGA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 30 01:23:16.717992227 (585h54m16.589075272s) [noise=true , address=77.90.31.10:4520 , piece_id=LSCPY3E6IB6Y22KX4D24HQ756IVBQCZCZH6NTIOTU23UDQF5GDGA, seg_pos={0 0}, upload=513620]: Long tail cancellation due to optimal threshold Dec 30 01:23:16.718026112 (585h54m16.589109147s) [seg_pos={0 0}, upload=513620]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:23:16.718109991 (585h54m16.589193036s) [upload=513620]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:23:16.718117305 (585h54m16.58920034s) [upload=513620]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:23:17.016443429 (585h54m16.887526474s) [upload=513620]: Done uploading segments... Dec 30 01:23:17.016467595 (585h54m16.88755064s) [upload=513620]: Upload finished. err: <nil> Duration: 22.983740875s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 26.370698676s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:25:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 26.370725717s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 21.672496376s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:24:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 21.672534207s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X.........X......X.X...X....X.X.....X.................XX..] duration: 6.129504196s trace: https://jaeger.dev.storj.io/trace/3af02f72f9549d7 FAIL (2025-12-30 01:24:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: N2FAP6WJI644OBBSDWXYWXZF7YLUWTLKOKIEHG6ANN4ZJDKNY7JQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: OLQCP377TKM6YUJYQ2EZWU33AZ6NNYYKVGQMIO53EHWZHODXMVQQ) context deadline exceeded ======LOGS====== Dec 30 01:23:14.158861942 (585h54m14.029944987s) [upload=513620]: Starting upload Dec 30 01:23:14.158944258 (585h54m14.030027314s) [upload=513620]: Done starting upload Dec 30 01:23:14.159068506 (585h54m14.030151541s) [upload=513620]: Uploading segments... Dec 30 01:23:14.159156653 (585h54m14.030239688s) [upload=513620]: Got next segment. Inline: false Dec 30 01:23:14.159190828 (585h54m14.030273863s) [upload=513620]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:23:14.159196008 (585h54m14.030279053s) [upload=513620]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:23:14.405670689 (585h54m14.276753724s) [seg_pos={0 0}, upload=513620]: Begin upload segment... Dec 30 01:23:14.405717478 (585h54m14.276800513s) [seg_pos={0 0}, upload=513620]: Done begin upload segment. Dec 30 01:23:14.405741774 (585h54m14.276824809s) [noise=true , address=45.91.116.4:4939 , piece_id=6ZOKYXBYAVZ6GOEBGJIXKGRBA2MUTCM6XF7UMZYD3CHW2OFGRNFQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.405881039 (585h54m14.276964074s) [noise=true , address=145.79.227.7:4295 , piece_id=XEUHRX3UMQ3MLLRNKKSBXZJ2KCS5QRLYFLSDV53MR2JCUNDVYCXA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.406599146 (585h54m14.277682181s) [noise=true , address=145.79.227.52:4445 , piece_id=VZYM4WDHWOVC2TRZFEDMWJXA4HRCKALIRTPLBLJXKNA6SCGWDFBA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.412496261 (585h54m14.283579296s) [noise=true , address=77.90.31.10:4520 , piece_id=LSCPY3E6IB6Y22KX4D24HQ756IVBQCZCZH6NTIOTU23UDQF5GDGA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.412589529 (585h54m14.283672574s) [noise=true , address=77.90.31.16:4735 , piece_id=CJC6PCJ2SR37UH2RCC2QW7566JRTGCY2BWQGFRDJOYCJW3ITSGWQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.414307297 (585h54m14.285390332s) [noise=true , address=145.79.227.17:4971 , piece_id=D2LB4XJUW2T43R33NWNKAF7GSVO5263YH52MIXCODTSFENWSYTIA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.414401076 (585h54m14.285484111s) [noise=true , address=145.79.227.10:4402 , piece_id=IUQ37VKBNTJFHXWSM5JQFZ66ITV5RLODUYH7YFXAHCAMQU53Z3BQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.415385491 (585h54m14.286468526s) [noise=true , address=145.79.227.8:4329 , piece_id=ONUB2LL5SNPIE542HGDODHGW4K6UDZM5YJCPO6KQ3VTZAOAXJX3Q, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.415893828 (585h54m14.286976863s) [noise=true , address=145.79.227.20:4439 , piece_id=RKZO5TT33LMIBOOZIB3IULHHLH3ERVQL3I6JAILFZ2O4X2MIBLDA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.416403217 (585h54m14.287486252s) [noise=true , address=145.79.227.36:4518 , piece_id=ZKDV72FHKAQCSM6HLXAOL6THAIT4UFWH5C2KY7WT76OVDPWAHXLQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.417390857 (585h54m14.288473893s) [noise=true , address=45.91.116.12:4226 , piece_id=ZPVJGN2AREUUJZEW3AT5HOXT73I6FAFVEO43L45EWKVZKGCC6MLA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.417419492 (585h54m14.288502537s) [noise=true , address=145.79.227.15:4874 , piece_id=RLMXGSIKFAUAAUL5HORQ2COJ5736N2IAKYHPMVXRKR77RDQ6QPFA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.418498746 (585h54m14.289581781s) [noise=true , address=45.91.116.6:4974 , piece_id=N2FAP6WJI644OBBSDWXYWXZF7YLUWTLKOKIEHG6ANN4ZJDKNY7JQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.418799899 (585h54m14.289882944s) [noise=true , address=45.91.116.7:4010 , piece_id=C2RGGVV3ETAESIEAUTSGGEXZETXB3BXQFRKMS4TWLSW7IYECY4FQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.420225372 (585h54m14.291308408s) [noise=true , address=45.91.116.13:4155 , piece_id=OLQCP377TKM6YUJYQ2EZWU33AZ6NNYYKVGQMIO53EHWZHODXMVQQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.420748388 (585h54m14.291831423s) [noise=true , address=145.79.227.46:4602 , piece_id=MP4MSU2UMX7P3HRZIRPLI6NCUCGJ475LMXEW3PEWYZ74UH3TNNVQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:16.113634421 (585h54m15.984717466s) [upload=513620]: Next returned nil segment Dec 30 01:23:16.113664088 (585h54m15.984747123s) [upload=513620]: Waiting for error group managing segments... Dec 30 01:23:16.399775522 (585h54m16.270858558s) [noise=true , address=145.79.227.52:4445 , piece_id=VZYM4WDHWOVC2TRZFEDMWJXA4HRCKALIRTPLBLJXKNA6SCGWDFBA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.405286742 (585h54m16.276369778s) [noise=true , address=145.79.227.20:4439 , piece_id=RKZO5TT33LMIBOOZIB3IULHHLH3ERVQL3I6JAILFZ2O4X2MIBLDA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.405369731 (585h54m16.276452766s) [noise=true , address=145.79.227.36:4518 , piece_id=ZKDV72FHKAQCSM6HLXAOL6THAIT4UFWH5C2KY7WT76OVDPWAHXLQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.407602491 (585h54m16.278685536s) [noise=true , address=145.79.227.17:4971 , piece_id=D2LB4XJUW2T43R33NWNKAF7GSVO5263YH52MIXCODTSFENWSYTIA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.412068081 (585h54m16.283151126s) [noise=true , address=145.79.227.8:4329 , piece_id=ONUB2LL5SNPIE542HGDODHGW4K6UDZM5YJCPO6KQ3VTZAOAXJX3Q, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.412197457 (585h54m16.283280492s) [noise=true , address=145.79.227.15:4874 , piece_id=RLMXGSIKFAUAAUL5HORQ2COJ5736N2IAKYHPMVXRKR77RDQ6QPFA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.420180532 (585h54m16.291263567s) [noise=true , address=145.79.227.10:4402 , piece_id=IUQ37VKBNTJFHXWSM5JQFZ66ITV5RLODUYH7YFXAHCAMQU53Z3BQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.426707214 (585h54m16.29779026s) [noise=true , address=145.79.227.46:4602 , piece_id=MP4MSU2UMX7P3HRZIRPLI6NCUCGJ475LMXEW3PEWYZ74UH3TNNVQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.586338838 (585h54m16.457421883s) [noise=true , address=77.90.31.16:4735 , piece_id=CJC6PCJ2SR37UH2RCC2QW7566JRTGCY2BWQGFRDJOYCJW3ITSGWQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.662930924 (585h54m16.534013969s) [noise=true , address=45.91.116.6:4974 , piece_id=N2FAP6WJI644OBBSDWXYWXZF7YLUWTLKOKIEHG6ANN4ZJDKNY7JQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.680218899 (585h54m16.551301934s) [noise=true , address=45.91.116.12:4226 , piece_id=ZPVJGN2AREUUJZEW3AT5HOXT73I6FAFVEO43L45EWKVZKGCC6MLA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.695093331 (585h54m16.566176367s) [noise=true , address=45.91.116.4:4939 , piece_id=6ZOKYXBYAVZ6GOEBGJIXKGRBA2MUTCM6XF7UMZYD3CHW2OFGRNFQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.717354203 (585h54m16.588437248s) [noise=true , address=45.91.116.13:4155 , piece_id=OLQCP377TKM6YUJYQ2EZWU33AZ6NNYYKVGQMIO53EHWZHODXMVQQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.717377557 (585h54m16.588460602s) [seg_pos={0 0}, upload=513620]: Segment reached optimal threshold of 13 pieces. Dec 30 01:23:16.717574872 (585h54m16.588657907s) [noise=true , address=45.91.116.7:4010 , piece_id=C2RGGVV3ETAESIEAUTSGGEXZETXB3BXQFRKMS4TWLSW7IYECY4FQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 30 01:23:16.717581765 (585h54m16.58866481s) [noise=true , address=45.91.116.7:4010 , piece_id=C2RGGVV3ETAESIEAUTSGGEXZETXB3BXQFRKMS4TWLSW7IYECY4FQ, seg_pos={0 0}, upload=513620]: Long tail cancellation due to optimal threshold Dec 30 01:23:16.717711633 (585h54m16.588794668s) [noise=true , address=145.79.227.7:4295 , piece_id=XEUHRX3UMQ3MLLRNKKSBXZJ2KCS5QRLYFLSDV53MR2JCUNDVYCXA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Dec 30 01:23:16.717720750 (585h54m16.588803785s) [noise=true , address=145.79.227.7:4295 , piece_id=XEUHRX3UMQ3MLLRNKKSBXZJ2KCS5QRLYFLSDV53MR2JCUNDVYCXA, seg_pos={0 0}, upload=513620]: Long tail cancellation due to optimal threshold Dec 30 01:23:16.717976106 (585h54m16.589059151s) [noise=true , address=77.90.31.10:4520 , piece_id=LSCPY3E6IB6Y22KX4D24HQ756IVBQCZCZH6NTIOTU23UDQF5GDGA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 30 01:23:16.717992227 (585h54m16.589075272s) [noise=true , address=77.90.31.10:4520 , piece_id=LSCPY3E6IB6Y22KX4D24HQ756IVBQCZCZH6NTIOTU23UDQF5GDGA, seg_pos={0 0}, upload=513620]: Long tail cancellation due to optimal threshold Dec 30 01:23:16.718026112 (585h54m16.589109147s) [seg_pos={0 0}, upload=513620]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:23:16.718109991 (585h54m16.589193036s) [upload=513620]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:23:16.718117305 (585h54m16.58920034s) [upload=513620]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:23:17.016443429 (585h54m16.887526474s) [upload=513620]: Done uploading segments... Dec 30 01:23:17.016467595 (585h54m16.88755064s) [upload=513620]: Upload finished. err: <nil> Duration: 6.129530766s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 23.424743078s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:23:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 23.424784688s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X.........X......X.X...X....X.X.....X.................XX.] duration: 5.987385897s trace: https://jaeger.dev.storj.io/trace/3af02f72f9549d7 FAIL (2025-12-30 01:23:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: N2FAP6WJI644OBBSDWXYWXZF7YLUWTLKOKIEHG6ANN4ZJDKNY7JQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: OLQCP377TKM6YUJYQ2EZWU33AZ6NNYYKVGQMIO53EHWZHODXMVQQ) context deadline exceeded ======LOGS====== Dec 30 01:23:14.158861942 (585h54m14.029944987s) [upload=513620]: Starting upload Dec 30 01:23:14.158944258 (585h54m14.030027314s) [upload=513620]: Done starting upload Dec 30 01:23:14.159068506 (585h54m14.030151541s) [upload=513620]: Uploading segments... Dec 30 01:23:14.159156653 (585h54m14.030239688s) [upload=513620]: Got next segment. Inline: false Dec 30 01:23:14.159190828 (585h54m14.030273863s) [upload=513620]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:23:14.159196008 (585h54m14.030279053s) [upload=513620]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:23:14.405670689 (585h54m14.276753724s) [seg_pos={0 0}, upload=513620]: Begin upload segment... Dec 30 01:23:14.405717478 (585h54m14.276800513s) [seg_pos={0 0}, upload=513620]: Done begin upload segment. Dec 30 01:23:14.405741774 (585h54m14.276824809s) [noise=true , address=45.91.116.4:4939 , piece_id=6ZOKYXBYAVZ6GOEBGJIXKGRBA2MUTCM6XF7UMZYD3CHW2OFGRNFQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.405881039 (585h54m14.276964074s) [noise=true , address=145.79.227.7:4295 , piece_id=XEUHRX3UMQ3MLLRNKKSBXZJ2KCS5QRLYFLSDV53MR2JCUNDVYCXA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.406599146 (585h54m14.277682181s) [noise=true , address=145.79.227.52:4445 , piece_id=VZYM4WDHWOVC2TRZFEDMWJXA4HRCKALIRTPLBLJXKNA6SCGWDFBA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.412496261 (585h54m14.283579296s) [noise=true , address=77.90.31.10:4520 , piece_id=LSCPY3E6IB6Y22KX4D24HQ756IVBQCZCZH6NTIOTU23UDQF5GDGA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.412589529 (585h54m14.283672574s) [noise=true , address=77.90.31.16:4735 , piece_id=CJC6PCJ2SR37UH2RCC2QW7566JRTGCY2BWQGFRDJOYCJW3ITSGWQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.414307297 (585h54m14.285390332s) [noise=true , address=145.79.227.17:4971 , piece_id=D2LB4XJUW2T43R33NWNKAF7GSVO5263YH52MIXCODTSFENWSYTIA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.414401076 (585h54m14.285484111s) [noise=true , address=145.79.227.10:4402 , piece_id=IUQ37VKBNTJFHXWSM5JQFZ66ITV5RLODUYH7YFXAHCAMQU53Z3BQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.415385491 (585h54m14.286468526s) [noise=true , address=145.79.227.8:4329 , piece_id=ONUB2LL5SNPIE542HGDODHGW4K6UDZM5YJCPO6KQ3VTZAOAXJX3Q, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.415893828 (585h54m14.286976863s) [noise=true , address=145.79.227.20:4439 , piece_id=RKZO5TT33LMIBOOZIB3IULHHLH3ERVQL3I6JAILFZ2O4X2MIBLDA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.416403217 (585h54m14.287486252s) [noise=true , address=145.79.227.36:4518 , piece_id=ZKDV72FHKAQCSM6HLXAOL6THAIT4UFWH5C2KY7WT76OVDPWAHXLQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.417390857 (585h54m14.288473893s) [noise=true , address=45.91.116.12:4226 , piece_id=ZPVJGN2AREUUJZEW3AT5HOXT73I6FAFVEO43L45EWKVZKGCC6MLA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.417419492 (585h54m14.288502537s) [noise=true , address=145.79.227.15:4874 , piece_id=RLMXGSIKFAUAAUL5HORQ2COJ5736N2IAKYHPMVXRKR77RDQ6QPFA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.418498746 (585h54m14.289581781s) [noise=true , address=45.91.116.6:4974 , piece_id=N2FAP6WJI644OBBSDWXYWXZF7YLUWTLKOKIEHG6ANN4ZJDKNY7JQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.418799899 (585h54m14.289882944s) [noise=true , address=45.91.116.7:4010 , piece_id=C2RGGVV3ETAESIEAUTSGGEXZETXB3BXQFRKMS4TWLSW7IYECY4FQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.420225372 (585h54m14.291308408s) [noise=true , address=45.91.116.13:4155 , piece_id=OLQCP377TKM6YUJYQ2EZWU33AZ6NNYYKVGQMIO53EHWZHODXMVQQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.420748388 (585h54m14.291831423s) [noise=true , address=145.79.227.46:4602 , piece_id=MP4MSU2UMX7P3HRZIRPLI6NCUCGJ475LMXEW3PEWYZ74UH3TNNVQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:16.113634421 (585h54m15.984717466s) [upload=513620]: Next returned nil segment Dec 30 01:23:16.113664088 (585h54m15.984747123s) [upload=513620]: Waiting for error group managing segments... Dec 30 01:23:16.399775522 (585h54m16.270858558s) [noise=true , address=145.79.227.52:4445 , piece_id=VZYM4WDHWOVC2TRZFEDMWJXA4HRCKALIRTPLBLJXKNA6SCGWDFBA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.405286742 (585h54m16.276369778s) [noise=true , address=145.79.227.20:4439 , piece_id=RKZO5TT33LMIBOOZIB3IULHHLH3ERVQL3I6JAILFZ2O4X2MIBLDA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.405369731 (585h54m16.276452766s) [noise=true , address=145.79.227.36:4518 , piece_id=ZKDV72FHKAQCSM6HLXAOL6THAIT4UFWH5C2KY7WT76OVDPWAHXLQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.407602491 (585h54m16.278685536s) [noise=true , address=145.79.227.17:4971 , piece_id=D2LB4XJUW2T43R33NWNKAF7GSVO5263YH52MIXCODTSFENWSYTIA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.412068081 (585h54m16.283151126s) [noise=true , address=145.79.227.8:4329 , piece_id=ONUB2LL5SNPIE542HGDODHGW4K6UDZM5YJCPO6KQ3VTZAOAXJX3Q, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.412197457 (585h54m16.283280492s) [noise=true , address=145.79.227.15:4874 , piece_id=RLMXGSIKFAUAAUL5HORQ2COJ5736N2IAKYHPMVXRKR77RDQ6QPFA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.420180532 (585h54m16.291263567s) [noise=true , address=145.79.227.10:4402 , piece_id=IUQ37VKBNTJFHXWSM5JQFZ66ITV5RLODUYH7YFXAHCAMQU53Z3BQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.426707214 (585h54m16.29779026s) [noise=true , address=145.79.227.46:4602 , piece_id=MP4MSU2UMX7P3HRZIRPLI6NCUCGJ475LMXEW3PEWYZ74UH3TNNVQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.586338838 (585h54m16.457421883s) [noise=true , address=77.90.31.16:4735 , piece_id=CJC6PCJ2SR37UH2RCC2QW7566JRTGCY2BWQGFRDJOYCJW3ITSGWQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.662930924 (585h54m16.534013969s) [noise=true , address=45.91.116.6:4974 , piece_id=N2FAP6WJI644OBBSDWXYWXZF7YLUWTLKOKIEHG6ANN4ZJDKNY7JQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.680218899 (585h54m16.551301934s) [noise=true , address=45.91.116.12:4226 , piece_id=ZPVJGN2AREUUJZEW3AT5HOXT73I6FAFVEO43L45EWKVZKGCC6MLA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.695093331 (585h54m16.566176367s) [noise=true , address=45.91.116.4:4939 , piece_id=6ZOKYXBYAVZ6GOEBGJIXKGRBA2MUTCM6XF7UMZYD3CHW2OFGRNFQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.717354203 (585h54m16.588437248s) [noise=true , address=45.91.116.13:4155 , piece_id=OLQCP377TKM6YUJYQ2EZWU33AZ6NNYYKVGQMIO53EHWZHODXMVQQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.717377557 (585h54m16.588460602s) [seg_pos={0 0}, upload=513620]: Segment reached optimal threshold of 13 pieces. Dec 30 01:23:16.717574872 (585h54m16.588657907s) [noise=true , address=45.91.116.7:4010 , piece_id=C2RGGVV3ETAESIEAUTSGGEXZETXB3BXQFRKMS4TWLSW7IYECY4FQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 30 01:23:16.717581765 (585h54m16.58866481s) [noise=true , address=45.91.116.7:4010 , piece_id=C2RGGVV3ETAESIEAUTSGGEXZETXB3BXQFRKMS4TWLSW7IYECY4FQ, seg_pos={0 0}, upload=513620]: Long tail cancellation due to optimal threshold Dec 30 01:23:16.717711633 (585h54m16.588794668s) [noise=true , address=145.79.227.7:4295 , piece_id=XEUHRX3UMQ3MLLRNKKSBXZJ2KCS5QRLYFLSDV53MR2JCUNDVYCXA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Dec 30 01:23:16.717720750 (585h54m16.588803785s) [noise=true , address=145.79.227.7:4295 , piece_id=XEUHRX3UMQ3MLLRNKKSBXZJ2KCS5QRLYFLSDV53MR2JCUNDVYCXA, seg_pos={0 0}, upload=513620]: Long tail cancellation due to optimal threshold Dec 30 01:23:16.717976106 (585h54m16.589059151s) [noise=true , address=77.90.31.10:4520 , piece_id=LSCPY3E6IB6Y22KX4D24HQ756IVBQCZCZH6NTIOTU23UDQF5GDGA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 30 01:23:16.717992227 (585h54m16.589075272s) [noise=true , address=77.90.31.10:4520 , piece_id=LSCPY3E6IB6Y22KX4D24HQ756IVBQCZCZH6NTIOTU23UDQF5GDGA, seg_pos={0 0}, upload=513620]: Long tail cancellation due to optimal threshold Dec 30 01:23:16.718026112 (585h54m16.589109147s) [seg_pos={0 0}, upload=513620]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:23:16.718109991 (585h54m16.589193036s) [upload=513620]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:23:16.718117305 (585h54m16.58920034s) [upload=513620]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:23:17.016443429 (585h54m16.887526474s) [upload=513620]: Done uploading segments... Dec 30 01:23:17.016467595 (585h54m16.88755064s) [upload=513620]: Upload finished. err: <nil> Duration: 5.987403049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X...X.........X......X.X...X....X.X.....X.................XX] duration: 30.003464864s trace: https://jaeger.dev.storj.io/trace/3af02f72f9549d7 FAIL (2025-12-30 01:23:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: N2FAP6WJI644OBBSDWXYWXZF7YLUWTLKOKIEHG6ANN4ZJDKNY7JQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: OLQCP377TKM6YUJYQ2EZWU33AZ6NNYYKVGQMIO53EHWZHODXMVQQ) context deadline exceeded ======LOGS====== Dec 30 01:23:14.158861942 (585h54m14.029944987s) [upload=513620]: Starting upload Dec 30 01:23:14.158944258 (585h54m14.030027314s) [upload=513620]: Done starting upload Dec 30 01:23:14.159068506 (585h54m14.030151541s) [upload=513620]: Uploading segments... Dec 30 01:23:14.159156653 (585h54m14.030239688s) [upload=513620]: Got next segment. Inline: false Dec 30 01:23:14.159190828 (585h54m14.030273863s) [upload=513620]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:23:14.159196008 (585h54m14.030279053s) [upload=513620]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:23:14.405670689 (585h54m14.276753724s) [seg_pos={0 0}, upload=513620]: Begin upload segment... Dec 30 01:23:14.405717478 (585h54m14.276800513s) [seg_pos={0 0}, upload=513620]: Done begin upload segment. Dec 30 01:23:14.405741774 (585h54m14.276824809s) [noise=true , address=45.91.116.4:4939 , piece_id=6ZOKYXBYAVZ6GOEBGJIXKGRBA2MUTCM6XF7UMZYD3CHW2OFGRNFQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.405881039 (585h54m14.276964074s) [noise=true , address=145.79.227.7:4295 , piece_id=XEUHRX3UMQ3MLLRNKKSBXZJ2KCS5QRLYFLSDV53MR2JCUNDVYCXA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.406599146 (585h54m14.277682181s) [noise=true , address=145.79.227.52:4445 , piece_id=VZYM4WDHWOVC2TRZFEDMWJXA4HRCKALIRTPLBLJXKNA6SCGWDFBA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.412496261 (585h54m14.283579296s) [noise=true , address=77.90.31.10:4520 , piece_id=LSCPY3E6IB6Y22KX4D24HQ756IVBQCZCZH6NTIOTU23UDQF5GDGA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.412589529 (585h54m14.283672574s) [noise=true , address=77.90.31.16:4735 , piece_id=CJC6PCJ2SR37UH2RCC2QW7566JRTGCY2BWQGFRDJOYCJW3ITSGWQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.414307297 (585h54m14.285390332s) [noise=true , address=145.79.227.17:4971 , piece_id=D2LB4XJUW2T43R33NWNKAF7GSVO5263YH52MIXCODTSFENWSYTIA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.414401076 (585h54m14.285484111s) [noise=true , address=145.79.227.10:4402 , piece_id=IUQ37VKBNTJFHXWSM5JQFZ66ITV5RLODUYH7YFXAHCAMQU53Z3BQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.415385491 (585h54m14.286468526s) [noise=true , address=145.79.227.8:4329 , piece_id=ONUB2LL5SNPIE542HGDODHGW4K6UDZM5YJCPO6KQ3VTZAOAXJX3Q, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.415893828 (585h54m14.286976863s) [noise=true , address=145.79.227.20:4439 , piece_id=RKZO5TT33LMIBOOZIB3IULHHLH3ERVQL3I6JAILFZ2O4X2MIBLDA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.416403217 (585h54m14.287486252s) [noise=true , address=145.79.227.36:4518 , piece_id=ZKDV72FHKAQCSM6HLXAOL6THAIT4UFWH5C2KY7WT76OVDPWAHXLQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.417390857 (585h54m14.288473893s) [noise=true , address=45.91.116.12:4226 , piece_id=ZPVJGN2AREUUJZEW3AT5HOXT73I6FAFVEO43L45EWKVZKGCC6MLA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.417419492 (585h54m14.288502537s) [noise=true , address=145.79.227.15:4874 , piece_id=RLMXGSIKFAUAAUL5HORQ2COJ5736N2IAKYHPMVXRKR77RDQ6QPFA, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.418498746 (585h54m14.289581781s) [noise=true , address=45.91.116.6:4974 , piece_id=N2FAP6WJI644OBBSDWXYWXZF7YLUWTLKOKIEHG6ANN4ZJDKNY7JQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.418799899 (585h54m14.289882944s) [noise=true , address=45.91.116.7:4010 , piece_id=C2RGGVV3ETAESIEAUTSGGEXZETXB3BXQFRKMS4TWLSW7IYECY4FQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.420225372 (585h54m14.291308408s) [noise=true , address=45.91.116.13:4155 , piece_id=OLQCP377TKM6YUJYQ2EZWU33AZ6NNYYKVGQMIO53EHWZHODXMVQQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:14.420748388 (585h54m14.291831423s) [noise=true , address=145.79.227.46:4602 , piece_id=MP4MSU2UMX7P3HRZIRPLI6NCUCGJ475LMXEW3PEWYZ74UH3TNNVQ, seg_pos={0 0}, upload=513620]: Uploading piece... Dec 30 01:23:16.113634421 (585h54m15.984717466s) [upload=513620]: Next returned nil segment Dec 30 01:23:16.113664088 (585h54m15.984747123s) [upload=513620]: Waiting for error group managing segments... Dec 30 01:23:16.399775522 (585h54m16.270858558s) [noise=true , address=145.79.227.52:4445 , piece_id=VZYM4WDHWOVC2TRZFEDMWJXA4HRCKALIRTPLBLJXKNA6SCGWDFBA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.405286742 (585h54m16.276369778s) [noise=true , address=145.79.227.20:4439 , piece_id=RKZO5TT33LMIBOOZIB3IULHHLH3ERVQL3I6JAILFZ2O4X2MIBLDA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.405369731 (585h54m16.276452766s) [noise=true , address=145.79.227.36:4518 , piece_id=ZKDV72FHKAQCSM6HLXAOL6THAIT4UFWH5C2KY7WT76OVDPWAHXLQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.407602491 (585h54m16.278685536s) [noise=true , address=145.79.227.17:4971 , piece_id=D2LB4XJUW2T43R33NWNKAF7GSVO5263YH52MIXCODTSFENWSYTIA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.412068081 (585h54m16.283151126s) [noise=true , address=145.79.227.8:4329 , piece_id=ONUB2LL5SNPIE542HGDODHGW4K6UDZM5YJCPO6KQ3VTZAOAXJX3Q, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.412197457 (585h54m16.283280492s) [noise=true , address=145.79.227.15:4874 , piece_id=RLMXGSIKFAUAAUL5HORQ2COJ5736N2IAKYHPMVXRKR77RDQ6QPFA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.420180532 (585h54m16.291263567s) [noise=true , address=145.79.227.10:4402 , piece_id=IUQ37VKBNTJFHXWSM5JQFZ66ITV5RLODUYH7YFXAHCAMQU53Z3BQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.426707214 (585h54m16.29779026s) [noise=true , address=145.79.227.46:4602 , piece_id=MP4MSU2UMX7P3HRZIRPLI6NCUCGJ475LMXEW3PEWYZ74UH3TNNVQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.586338838 (585h54m16.457421883s) [noise=true , address=77.90.31.16:4735 , piece_id=CJC6PCJ2SR37UH2RCC2QW7566JRTGCY2BWQGFRDJOYCJW3ITSGWQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.662930924 (585h54m16.534013969s) [noise=true , address=45.91.116.6:4974 , piece_id=N2FAP6WJI644OBBSDWXYWXZF7YLUWTLKOKIEHG6ANN4ZJDKNY7JQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.680218899 (585h54m16.551301934s) [noise=true , address=45.91.116.12:4226 , piece_id=ZPVJGN2AREUUJZEW3AT5HOXT73I6FAFVEO43L45EWKVZKGCC6MLA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.695093331 (585h54m16.566176367s) [noise=true , address=45.91.116.4:4939 , piece_id=6ZOKYXBYAVZ6GOEBGJIXKGRBA2MUTCM6XF7UMZYD3CHW2OFGRNFQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.717354203 (585h54m16.588437248s) [noise=true , address=45.91.116.13:4155 , piece_id=OLQCP377TKM6YUJYQ2EZWU33AZ6NNYYKVGQMIO53EHWZHODXMVQQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: <nil> Dec 30 01:23:16.717377557 (585h54m16.588460602s) [seg_pos={0 0}, upload=513620]: Segment reached optimal threshold of 13 pieces. Dec 30 01:23:16.717574872 (585h54m16.588657907s) [noise=true , address=45.91.116.7:4010 , piece_id=C2RGGVV3ETAESIEAUTSGGEXZETXB3BXQFRKMS4TWLSW7IYECY4FQ, seg_pos={0 0}, upload=513620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 30 01:23:16.717581765 (585h54m16.58866481s) [noise=true , address=45.91.116.7:4010 , piece_id=C2RGGVV3ETAESIEAUTSGGEXZETXB3BXQFRKMS4TWLSW7IYECY4FQ, seg_pos={0 0}, upload=513620]: Long tail cancellation due to optimal threshold Dec 30 01:23:16.717711633 (585h54m16.588794668s) [noise=true , address=145.79.227.7:4295 , piece_id=XEUHRX3UMQ3MLLRNKKSBXZJ2KCS5QRLYFLSDV53MR2JCUNDVYCXA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Dec 30 01:23:16.717720750 (585h54m16.588803785s) [noise=true , address=145.79.227.7:4295 , piece_id=XEUHRX3UMQ3MLLRNKKSBXZJ2KCS5QRLYFLSDV53MR2JCUNDVYCXA, seg_pos={0 0}, upload=513620]: Long tail cancellation due to optimal threshold Dec 30 01:23:16.717976106 (585h54m16.589059151s) [noise=true , address=77.90.31.10:4520 , piece_id=LSCPY3E6IB6Y22KX4D24HQ756IVBQCZCZH6NTIOTU23UDQF5GDGA, seg_pos={0 0}, upload=513620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 30 01:23:16.717992227 (585h54m16.589075272s) [noise=true , address=77.90.31.10:4520 , piece_id=LSCPY3E6IB6Y22KX4D24HQ756IVBQCZCZH6NTIOTU23UDQF5GDGA, seg_pos={0 0}, upload=513620]: Long tail cancellation due to optimal threshold Dec 30 01:23:16.718026112 (585h54m16.589109147s) [seg_pos={0 0}, upload=513620]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:23:16.718109991 (585h54m16.589193036s) [upload=513620]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:23:16.718117305 (585h54m16.58920034s) [upload=513620]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:23:17.016443429 (585h54m16.887526474s) [upload=513620]: Done uploading segments... Dec 30 01:23:17.016467595 (585h54m16.88755064s) [upload=513620]: Upload finished. err: <nil> Duration: 30.003489252s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X...X.........X......X.X...X....X.X.....X.................X] duration: 30.002839961s trace: https://jaeger.dev.storj.io/trace/9b662ceaa37afdd FAIL (2025-12-30 01:23:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: HE4NLEYMFHTSTBIGSWZSNI7E4JMKGCS3OYLZSCVB7XJGSH5VSVRA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: FM32B3EIUIHJGUCUO7MNM3ZILIC2GXAPABYATPJ3HUDA5VJSAYQA) context deadline exceeded ======LOGS====== 00409 (585h53m38.492583454s) [upload=513612]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:22:38.873355973 (585h53m38.744439018s) [seg_pos={0 0}, upload=513612]: Begin upload segment... Dec 30 01:22:38.873411087 (585h53m38.744494132s) [seg_pos={0 0}, upload=513612]: Done begin upload segment. Dec 30 01:22:38.873445743 (585h53m38.744528778s) [noise=true , address=145.79.227.16:4910 , piece_id=244AQVMIDBQQKUKTU6PMBNY2BHRR6NACA7RJWYPTZE3AWU474Y5A, seg_pos={0 0}, upload=513612]: Uploading piece... Dec 30 01:22:38.873607351 (585h53m38.744690386s) [noise=true , address=45.91.116.14:4192 , piece_id=R5IGU5FYGN4NWUWZWW46WA45SGEKDUP5ZJ7IWMACWPTGUIPSWRBQ, seg_pos={0 0}, upload=513612]: Uploading piece... Dec 30 01:22:38.875399071 (585h53m38.746482107s) [noise=true , address=77.90.31.16:4736 , piece_id=TGHSGDUR42UJJPZM5WZ2BTX2QYST5WQCJOW2PUKPCBAS37YPFFDQ, seg_pos={0 0}, upload=513612]: Uploading piece... Dec 30 01:22:38.875715443 (585h53m38.746798488s) [noise=true , address=145.79.227.5:4221 , piece_id=YSLBNK7IOG2TO3E3MX7XLMXSBLI2JN3QLGP5O23AA4REPRDP536A, seg_pos={0 0}, upload=513612]: Uploading piece... Dec 30 01:22:38.875986078 (585h53m38.747069113s) [noise=true , address=145.79.227.13:4804 , piece_id=EWPYMMJNIWKQA7NIYM74L53ZO4ONMFOCOFQGDUPPZGQXZQO3MXQA, seg_pos={0 0}, upload=513612]: Uploading piece... Dec 30 01:22:38.876405046 (585h53m38.747488091s) [noise=true , address=145.79.227.50:4373 , piece_id=ANMRKKKUYFF7TJPBKDJFIGBWGYB7UJXUY45H24MB5TZOPUXC6HVQ, seg_pos={0 0}, upload=513612]: Uploading piece... Dec 30 01:22:38.876459279 (585h53m38.747542314s) [noise=true , address=45.91.116.4:4940 , piece_id=5ZPO27EF3P5AQT3KLY6P4IW5JFLVALKBZD2KRTP42L4VX5DLYXPQ, seg_pos={0 0}, upload=513612]: Uploading piece... Dec 30 01:22:38.877040254 (585h53m38.748123309s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6PQ6G7HQVQXBWB7L3TLL6FB6Y5SC44K5M53H3GWJJOCJ4H4RB6A, seg_pos={0 0}, upload=513612]: Uploading piece... Dec 30 01:22:38.877215137 (585h53m38.748298182s) [noise=true , address=45.91.116.9:4263 , piece_id=HE4NLEYMFHTSTBIGSWZSNI7E4JMKGCS3OYLZSCVB7XJGSH5VSVRA, seg_pos={0 0}, upload=513612]: Uploading piece... Dec 30 01:22:38.877650055 (585h53m38.74873309s) [noise=true , address=145.79.227.14:4839 , piece_id=MJLBYWBA7EUWE7ASQ5HOUX7T2KY6EI46DCNWE7POSGHCJPPXNWWA, seg_pos={0 0}, upload=513612]: Uploading piece... Dec 30 01:22:38.877929297 (585h53m38.749012342s) [noise=true , address=145.79.227.48:4303 , piece_id=MGVWWKMSF4QTK4LZLHD3ODMI3TDW3X2YP6G7KW6WSWHJD66ECCZA, seg_pos={0 0}, upload=513612]: Uploading piece... Dec 30 01:22:38.878198389 (585h53m38.749281585s) [noise=true , address=77.90.31.21:4878 , piece_id=FM32B3EIUIHJGUCUO7MNM3ZILIC2GXAPABYATPJ3HUDA5VJSAYQA, seg_pos={0 0}, upload=513612]: Uploading piece... Dec 30 01:22:38.878623559 (585h53m38.749706594s) [noise=true , address=45.91.116.5:4298 , piece_id=JAQF75ESX7CUX72ZLRI6AXNL6ZPABGEEOXFKU3F6N52JJVZPF3MA, seg_pos={0 0}, upload=513612]: Uploading piece... Dec 30 01:22:38.878725242 (585h53m38.749808287s) [noise=true , address=145.79.227.11:4729 , piece_id=AQV2VIFJIAGOSGNHWT2PBOX6ZZ7E2MXQRJO6NR2D63N5HP4U75RA, seg_pos={0 0}, upload=513612]: Uploading piece... Dec 30 01:22:38.879154529 (585h53m38.750237574s) [noise=true , address=145.79.227.3:4149 , piece_id=SPHFRPG4KAA7L56BVY2HLQI5BVRUXU5MVUTEEC2VUYP66RCJBHHA, seg_pos={0 0}, upload=513612]: Uploading piece... Dec 30 01:22:38.879212519 (585h53m38.750295554s) [noise=true , address=45.91.116.10:4082 , piece_id=VAORAZICTSVOW7UHFS5CEF27LFN6MIDJP4BZS54Q3UQ6NLISBHTQ, seg_pos={0 0}, upload=513612]: Uploading piece... Dec 30 01:22:40.479817375 (585h53m40.350900411s) [upload=513612]: Next returned nil segment Dec 30 01:22:40.479830049 (585h53m40.350913094s) [upload=513612]: Waiting for error group managing segments... Dec 30 01:22:40.866086300 (585h53m40.737169336s) [noise=true , address=145.79.227.14:4839 , piece_id=MJLBYWBA7EUWE7ASQ5HOUX7T2KY6EI46DCNWE7POSGHCJPPXNWWA, seg_pos={0 0}, upload=513612]: Done uploading piece. err: <nil> Dec 30 01:22:40.872990752 (585h53m40.744073787s) [noise=true , address=145.79.227.13:4804 , piece_id=EWPYMMJNIWKQA7NIYM74L53ZO4ONMFOCOFQGDUPPZGQXZQO3MXQA, seg_pos={0 0}, upload=513612]: Done uploading piece. err: <nil> Dec 30 01:22:40.873942514 (585h53m40.745025559s) [noise=true , address=145.79.227.11:4729 , piece_id=AQV2VIFJIAGOSGNHWT2PBOX6ZZ7E2MXQRJO6NR2D63N5HP4U75RA, seg_pos={0 0}, upload=513612]: Done uploading piece. err: <nil> Dec 30 01:22:40.874480468 (585h53m40.745563503s) [noise=true , address=145.79.227.16:4910 , piece_id=244AQVMIDBQQKUKTU6PMBNY2BHRR6NACA7RJWYPTZE3AWU474Y5A, seg_pos={0 0}, upload=513612]: Done uploading piece. err: <nil> Dec 30 01:22:40.879192427 (585h53m40.750275472s) [noise=true , address=145.79.227.48:4303 , piece_id=MGVWWKMSF4QTK4LZLHD3ODMI3TDW3X2YP6G7KW6WSWHJD66ECCZA, seg_pos={0 0}, upload=513612]: Done uploading piece. err: <nil> Dec 30 01:22:40.893440357 (585h53m40.764523392s) [noise=true , address=145.79.227.50:4373 , piece_id=ANMRKKKUYFF7TJPBKDJFIGBWGYB7UJXUY45H24MB5TZOPUXC6HVQ, seg_pos={0 0}, upload=513612]: Done uploading piece. err: <nil> Dec 30 01:22:40.894298861 (585h53m40.765381896s) [noise=true , address=145.79.227.5:4221 , piece_id=YSLBNK7IOG2TO3E3MX7XLMXSBLI2JN3QLGP5O23AA4REPRDP536A, seg_pos={0 0}, upload=513612]: Done uploading piece. err: <nil> Dec 30 01:22:41.043755907 (585h53m40.914838943s) [noise=true , address=77.90.31.21:4878 , piece_id=FM32B3EIUIHJGUCUO7MNM3ZILIC2GXAPABYATPJ3HUDA5VJSAYQA, seg_pos={0 0}, upload=513612]: Done uploading piece. err: <nil> Dec 30 01:22:41.102523135 (585h53m40.97360617s) [noise=true , address=77.90.31.16:4736 , piece_id=TGHSGDUR42UJJPZM5WZ2BTX2QYST5WQCJOW2PUKPCBAS37YPFFDQ, seg_pos={0 0}, upload=513612]: Done uploading piece. err: <nil> Dec 30 01:22:41.164629288 (585h53m41.035712333s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6PQ6G7HQVQXBWB7L3TLL6FB6Y5SC44K5M53H3GWJJOCJ4H4RB6A, seg_pos={0 0}, upload=513612]: Done uploading piece. err: <nil> Dec 30 01:22:41.205459745 (585h53m41.076542781s) [noise=true , address=45.91.116.14:4192 , piece_id=R5IGU5FYGN4NWUWZWW46WA45SGEKDUP5ZJ7IWMACWPTGUIPSWRBQ, seg_pos={0 0}, upload=513612]: Done uploading piece. err: <nil> Dec 30 01:22:41.221699887 (585h53m41.092782932s) [noise=true , address=45.91.116.5:4298 , piece_id=JAQF75ESX7CUX72ZLRI6AXNL6ZPABGEEOXFKU3F6N52JJVZPF3MA, seg_pos={0 0}, upload=513612]: Done uploading piece. err: <nil> Dec 30 01:22:41.376088811 (585h53m41.247171846s) [noise=true , address=45.91.116.9:4263 , piece_id=HE4NLEYMFHTSTBIGSWZSNI7E4JMKGCS3OYLZSCVB7XJGSH5VSVRA, seg_pos={0 0}, upload=513612]: Done uploading piece. err: <nil> Dec 30 01:22:41.376112846 (585h53m41.247195911s) [seg_pos={0 0}, upload=513612]: Segment reached optimal threshold of 13 pieces. Dec 30 01:22:41.376321945 (585h53m41.24740498s) [noise=true , address=45.91.116.4:4940 , piece_id=5ZPO27EF3P5AQT3KLY6P4IW5JFLVALKBZD2KRTP42L4VX5DLYXPQ, seg_pos={0 0}, upload=513612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:40582->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:40582->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40582->45.91.116.4:4940: use of closed network connection Dec 30 01:22:41.376329308 (585h53m41.247412353s) [noise=true , address=45.91.116.4:4940 , piece_id=5ZPO27EF3P5AQT3KLY6P4IW5JFLVALKBZD2KRTP42L4VX5DLYXPQ, seg_pos={0 0}, upload=513612]: Long tail cancellation due to optimal threshold Dec 30 01:22:41.376583413 (585h53m41.247666458s) [noise=true , address=45.91.116.10:4082 , piece_id=VAORAZICTSVOW7UHFS5CEF27LFN6MIDJP4BZS54Q3UQ6NLISBHTQ, seg_pos={0 0}, upload=513612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:33316->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:33316->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33316->45.91.116.10:4082: use of closed network connection Dec 30 01:22:41.376606556 (585h53m41.247689611s) [noise=true , address=45.91.116.10:4082 , piece_id=VAORAZICTSVOW7UHFS5CEF27LFN6MIDJP4BZS54Q3UQ6NLISBHTQ, seg_pos={0 0}, upload=513612]: Long tail cancellation due to optimal threshold Dec 30 01:22:41.376675738 (585h53m41.247758773s) [noise=true , address=145.79.227.3:4149 , piece_id=SPHFRPG4KAA7L56BVY2HLQI5BVRUXU5MVUTEEC2VUYP66RCJBHHA, seg_pos={0 0}, upload=513612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: write tcp 188.166.225.170:39672->145.79.227.3:4149: use of closed network connection; write tcp 188.166.225.170:39672->145.79.227.3:4149: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39672->145.79.227.3:4149: use of closed network connection Dec 30 01:22:41.376684585 (585h53m41.24776762s) [noise=true , address=145.79.227.3:4149 , piece_id=SPHFRPG4KAA7L56BVY2HLQI5BVRUXU5MVUTEEC2VUYP66RCJBHHA, seg_pos={0 0}, upload=513612]: Long tail cancellation due to optimal threshold Dec 30 01:22:41.376702309 (585h53m41.247785344s) [seg_pos={0 0}, upload=513612]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:22:41.376776290 (585h53m41.247859325s) [upload=513612]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:22:41.376781990 (585h53m41.247865025s) [upload=513612]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:22:41.668223270 (585h53m41.539306305s) [upload=513612]: Done uploading segments... Dec 30 01:22:41.668247096 (585h53m41.539330131s) [upload=513612]: Upload finished. err: <nil> Duration:* 30.002864618s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 38.566405106s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:22:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 38.566430496s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X...X.........X......X.X...X....X.X.....X.................] duration: 5.891154667s trace: https://jaeger.dev.storj.io/trace/16af4eed1c703455 FAIL (2025-12-30 01:22:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: RJKGGPIZQ3YU5HLNNUF46TUGQFBR2JX62CUZEDGGBJWUTWXQJFBA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: KNWLZSLZASPIPNIRJF5BPR6XDWGISRGRLQ7YS5I7YJ2HCPB23RMA) context deadline exceeded ======LOGS====== g segments... Dec 30 01:08:22.484294554 (585h39m22.355377599s) [upload=513404]: Got next segment. Inline: false Dec 30 01:08:22.484338407 (585h39m22.355421442s) [upload=513404]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:08:22.484343197 (585h39m22.355426232s) [upload=513404]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:08:22.732399804 (585h39m22.603482839s) [seg_pos={0 0}, upload=513404]: Begin upload segment... Dec 30 01:08:22.732449068 (585h39m22.603532113s) [seg_pos={0 0}, upload=513404]: Done begin upload segment. Dec 30 01:08:22.732475969 (585h39m22.603559014s) [noise=true , address=145.79.227.2:4113 , piece_id=TBXZDUTJGXLB5WUFQUC5GV6AC3JRXNRRPF2YXNRC4VICCKNQMBHQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.732515504 (585h39m22.603598539s) [noise=true , address=145.79.227.44:4194 , piece_id=GFBCVZH3EN54KVCOA4HW7M7URVVXLLO37KKTBVP53AHMTXM4DNGQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.733197663 (585h39m22.604280698s) [noise=true , address=145.79.227.19:4078 , piece_id=B3LIUGAXP6O6EERR3LV3ABFHRQ7U4FNYQOGWWFUQAGOBHJBBDLDQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.733334544 (585h39m22.604417579s) [noise=true , address=45.91.116.7:4012 , piece_id=DC7YL6DHDQQOOLBLSHJ2NZOCZDBBDXG5HJUC3KYALUMYVEWC5C4Q, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.733717734 (585h39m22.604800779s) [noise=true , address=45.91.116.12:4226 , piece_id=RJKGGPIZQ3YU5HLNNUF46TUGQFBR2JX62CUZEDGGBJWUTWXQJFBA, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.734067009 (585h39m22.605150054s) [noise=true , address=145.79.227.51:4410 , piece_id=TEGW7H66VA3QQWKZDX7HIHDH7TCALOZWAVHA56G4YQX3I4O7V5MQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.734190263 (585h39m22.605273308s) [noise=true , address=45.91.116.14:4192 , piece_id=5BHW2RX7XHTYXUIFALQBE6FEN3G6PCB475XFYO43E5QTCWN5AGUQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.734493841 (585h39m22.605576876s) [noise=true , address=77.90.31.14:4661 , piece_id=CWJC67GO7HPCNOI57H7BJMGGVXWOY3FF2YWS6T7V65CMLD7DPKTA, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.734795215 (585h39m22.60587826s) [noise=true , address=145.79.227.46:4601 , piece_id=4MYGT6TZ2TDT6LMEP4YARQUMC5GJAFWC6G4NKHQZSQKETBA5UNNA, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.734812428 (585h39m22.605895473s) [noise=true , address=145.79.227.15:4873 , piece_id=BRIGIUKVPNWKUP4I5DDSRJRXJLYGYGIRAY7XQJDFPG55ZPYNXJUA, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.735186771 (585h39m22.606269816s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7I35P5AFVSGWYW7MQG6N7SJQ4ZAH24VB5NW6CUWQLZPXDYW26UA, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.735262475 (585h39m22.60634552s) [noise=true , address=145.79.227.52:4447 , piece_id=IAADPQ7CYD5JYLUVN52EDGHNU7AG6TBPI5OXODY3RRPVY3KEWUMQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.735531347 (585h39m22.606614382s) [noise=true , address=45.91.116.5:4300 , piece_id=KNWLZSLZASPIPNIRJF5BPR6XDWGISRGRLQ7YS5I7YJ2HCPB23RMA, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.735684438 (585h39m22.606767483s) [noise=true , address=145.79.227.17:4972 , piece_id=ZFAPAJQ5ZPOUMBG7UYVTWANJJKHDLGG7RFJ2KXPQ2JBT7UDOOJUQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.736100951 (585h39m22.607183996s) [noise=true , address=145.79.227.6:4257 , piece_id=IOJAITMSTXKQSUOPNTYK5GQFNUOMTNMDQXPHHUFEM36TIBE4JRCQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.736552080 (585h39m22.607635115s) [noise=true , address=145.79.227.47:4267 , piece_id=RMXSL3ZILRYQUQ3OKHYCLQBEK5TX7QJDZIKXWHE62WSVXO3L46NQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:24.423934888 (585h39m24.295017933s) [upload=513404]: Next returned nil segment Dec 30 01:08:24.423954085 (585h39m24.29503713s) [upload=513404]: Waiting for error group managing segments... Dec 30 01:08:24.703932355 (585h39m24.5750154s) [noise=true , address=145.79.227.51:4410 , piece_id=TEGW7H66VA3QQWKZDX7HIHDH7TCALOZWAVHA56G4YQX3I4O7V5MQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.708283540 (585h39m24.579366585s) [noise=true , address=145.79.227.15:4873 , piece_id=BRIGIUKVPNWKUP4I5DDSRJRXJLYGYGIRAY7XQJDFPG55ZPYNXJUA, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.714480530 (585h39m24.585563565s) [noise=true , address=145.79.227.46:4601 , piece_id=4MYGT6TZ2TDT6LMEP4YARQUMC5GJAFWC6G4NKHQZSQKETBA5UNNA, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.715434436 (585h39m24.586517481s) [noise=true , address=145.79.227.47:4267 , piece_id=RMXSL3ZILRYQUQ3OKHYCLQBEK5TX7QJDZIKXWHE62WSVXO3L46NQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.716158786 (585h39m24.587241831s) [noise=true , address=145.79.227.17:4972 , piece_id=ZFAPAJQ5ZPOUMBG7UYVTWANJJKHDLGG7RFJ2KXPQ2JBT7UDOOJUQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.728524702 (585h39m24.599607747s) [noise=true , address=145.79.227.6:4257 , piece_id=IOJAITMSTXKQSUOPNTYK5GQFNUOMTNMDQXPHHUFEM36TIBE4JRCQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.729139773 (585h39m24.600222818s) [noise=true , address=145.79.227.52:4447 , piece_id=IAADPQ7CYD5JYLUVN52EDGHNU7AG6TBPI5OXODY3RRPVY3KEWUMQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.735310563 (585h39m24.606393598s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7I35P5AFVSGWYW7MQG6N7SJQ4ZAH24VB5NW6CUWQLZPXDYW26UA, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.739217192 (585h39m24.610300247s) [noise=true , address=145.79.227.19:4078 , piece_id=B3LIUGAXP6O6EERR3LV3ABFHRQ7U4FNYQOGWWFUQAGOBHJBBDLDQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.999002984 (585h39m24.870086019s) [noise=true , address=45.91.116.14:4192 , piece_id=5BHW2RX7XHTYXUIFALQBE6FEN3G6PCB475XFYO43E5QTCWN5AGUQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:25.090676861 (585h39m24.961759906s) [noise=true , address=145.79.227.2:4113 , piece_id=TBXZDUTJGXLB5WUFQUC5GV6AC3JRXNRRPF2YXNRC4VICCKNQMBHQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:25.143840390 (585h39m25.014923426s) [noise=true , address=45.91.116.12:4226 , piece_id=RJKGGPIZQ3YU5HLNNUF46TUGQFBR2JX62CUZEDGGBJWUTWXQJFBA, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:25.306778804 (585h39m25.177861839s) [noise=true , address=45.91.116.5:4300 , piece_id=KNWLZSLZASPIPNIRJF5BPR6XDWGISRGRLQ7YS5I7YJ2HCPB23RMA, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:25.306799223 (585h39m25.177882258s) [seg_pos={0 0}, upload=513404]: Segment reached optimal threshold of 13 pieces. Dec 30 01:08:25.307018811 (585h39m25.178101846s) [noise=true , address=45.91.116.7:4012 , piece_id=DC7YL6DHDQQOOLBLSHJ2NZOCZDBBDXG5HJUC3KYALUMYVEWC5C4Q, seg_pos={0 0}, upload=513404]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:59146->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:59146->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59146->45.91.116.7:4012: use of closed network connection Dec 30 01:08:25.307035833 (585h39m25.178118868s) [noise=true , address=45.91.116.7:4012 , piece_id=DC7YL6DHDQQOOLBLSHJ2NZOCZDBBDXG5HJUC3KYALUMYVEWC5C4Q, seg_pos={0 0}, upload=513404]: Long tail cancellation due to optimal threshold Dec 30 01:08:25.307214483 (585h39m25.178297518s) [noise=true , address=145.79.227.44:4194 , piece_id=GFBCVZH3EN54KVCOA4HW7M7URVVXLLO37KKTBVP53AHMTXM4DNGQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 30 01:08:25.307220625 (585h39m25.17830367s) [noise=true , address=145.79.227.44:4194 , piece_id=GFBCVZH3EN54KVCOA4HW7M7URVVXLLO37KKTBVP53AHMTXM4DNGQ, seg_pos={0 0}, upload=513404]: Long tail cancellation due to optimal threshold Dec 30 01:08:25.307628461 (585h39m25.178711496s) [noise=true , address=77.90.31.14:4661 , piece_id=CWJC67GO7HPCNOI57H7BJMGGVXWOY3FF2YWS6T7V65CMLD7DPKTA, seg_pos={0 0}, upload=513404]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:53704->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:53704->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53704->77.90.31.14:4661: use of closed network connection Dec 30 01:08:25.307636506 (585h39m25.178719551s) [noise=true , address=77.90.31.14:4661 , piece_id=CWJC67GO7HPCNOI57H7BJMGGVXWOY3FF2YWS6T7V65CMLD7DPKTA, seg_pos={0 0}, upload=513404]: Long tail cancellation due to optimal threshold Dec 30 01:08:25.307663077 (585h39m25.178746112s) [seg_pos={0 0}, upload=513404]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:08:25.307744743 (585h39m25.178827778s) [upload=513404]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:08:25.307750013 (585h39m25.178833048s) [upload=513404]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:08:25.604467427 (585h39m25.475550472s) [upload=513404]: Done uploading segments... Dec 30 01:08:25.604490570 (585h39m25.475573605s) [upload=513404]: Upload finished. err: <nil> Duration: 5.891179957s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 15.662238861s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:21:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 15.662258247s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X...X.........X......X.X...X....X.X.....X................] duration: 5.965703215s trace: https://jaeger.dev.storj.io/trace/16af4eed1c703455 FAIL (2025-12-30 01:21:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: RJKGGPIZQ3YU5HLNNUF46TUGQFBR2JX62CUZEDGGBJWUTWXQJFBA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: KNWLZSLZASPIPNIRJF5BPR6XDWGISRGRLQ7YS5I7YJ2HCPB23RMA) context deadline exceeded ======LOGS====== g segments... Dec 30 01:08:22.484294554 (585h39m22.355377599s) [upload=513404]: Got next segment. Inline: false Dec 30 01:08:22.484338407 (585h39m22.355421442s) [upload=513404]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:08:22.484343197 (585h39m22.355426232s) [upload=513404]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:08:22.732399804 (585h39m22.603482839s) [seg_pos={0 0}, upload=513404]: Begin upload segment... Dec 30 01:08:22.732449068 (585h39m22.603532113s) [seg_pos={0 0}, upload=513404]: Done begin upload segment. Dec 30 01:08:22.732475969 (585h39m22.603559014s) [noise=true , address=145.79.227.2:4113 , piece_id=TBXZDUTJGXLB5WUFQUC5GV6AC3JRXNRRPF2YXNRC4VICCKNQMBHQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.732515504 (585h39m22.603598539s) [noise=true , address=145.79.227.44:4194 , piece_id=GFBCVZH3EN54KVCOA4HW7M7URVVXLLO37KKTBVP53AHMTXM4DNGQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.733197663 (585h39m22.604280698s) [noise=true , address=145.79.227.19:4078 , piece_id=B3LIUGAXP6O6EERR3LV3ABFHRQ7U4FNYQOGWWFUQAGOBHJBBDLDQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.733334544 (585h39m22.604417579s) [noise=true , address=45.91.116.7:4012 , piece_id=DC7YL6DHDQQOOLBLSHJ2NZOCZDBBDXG5HJUC3KYALUMYVEWC5C4Q, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.733717734 (585h39m22.604800779s) [noise=true , address=45.91.116.12:4226 , piece_id=RJKGGPIZQ3YU5HLNNUF46TUGQFBR2JX62CUZEDGGBJWUTWXQJFBA, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.734067009 (585h39m22.605150054s) [noise=true , address=145.79.227.51:4410 , piece_id=TEGW7H66VA3QQWKZDX7HIHDH7TCALOZWAVHA56G4YQX3I4O7V5MQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.734190263 (585h39m22.605273308s) [noise=true , address=45.91.116.14:4192 , piece_id=5BHW2RX7XHTYXUIFALQBE6FEN3G6PCB475XFYO43E5QTCWN5AGUQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.734493841 (585h39m22.605576876s) [noise=true , address=77.90.31.14:4661 , piece_id=CWJC67GO7HPCNOI57H7BJMGGVXWOY3FF2YWS6T7V65CMLD7DPKTA, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.734795215 (585h39m22.60587826s) [noise=true , address=145.79.227.46:4601 , piece_id=4MYGT6TZ2TDT6LMEP4YARQUMC5GJAFWC6G4NKHQZSQKETBA5UNNA, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.734812428 (585h39m22.605895473s) [noise=true , address=145.79.227.15:4873 , piece_id=BRIGIUKVPNWKUP4I5DDSRJRXJLYGYGIRAY7XQJDFPG55ZPYNXJUA, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.735186771 (585h39m22.606269816s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7I35P5AFVSGWYW7MQG6N7SJQ4ZAH24VB5NW6CUWQLZPXDYW26UA, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.735262475 (585h39m22.60634552s) [noise=true , address=145.79.227.52:4447 , piece_id=IAADPQ7CYD5JYLUVN52EDGHNU7AG6TBPI5OXODY3RRPVY3KEWUMQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.735531347 (585h39m22.606614382s) [noise=true , address=45.91.116.5:4300 , piece_id=KNWLZSLZASPIPNIRJF5BPR6XDWGISRGRLQ7YS5I7YJ2HCPB23RMA, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.735684438 (585h39m22.606767483s) [noise=true , address=145.79.227.17:4972 , piece_id=ZFAPAJQ5ZPOUMBG7UYVTWANJJKHDLGG7RFJ2KXPQ2JBT7UDOOJUQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.736100951 (585h39m22.607183996s) [noise=true , address=145.79.227.6:4257 , piece_id=IOJAITMSTXKQSUOPNTYK5GQFNUOMTNMDQXPHHUFEM36TIBE4JRCQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.736552080 (585h39m22.607635115s) [noise=true , address=145.79.227.47:4267 , piece_id=RMXSL3ZILRYQUQ3OKHYCLQBEK5TX7QJDZIKXWHE62WSVXO3L46NQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:24.423934888 (585h39m24.295017933s) [upload=513404]: Next returned nil segment Dec 30 01:08:24.423954085 (585h39m24.29503713s) [upload=513404]: Waiting for error group managing segments... Dec 30 01:08:24.703932355 (585h39m24.5750154s) [noise=true , address=145.79.227.51:4410 , piece_id=TEGW7H66VA3QQWKZDX7HIHDH7TCALOZWAVHA56G4YQX3I4O7V5MQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.708283540 (585h39m24.579366585s) [noise=true , address=145.79.227.15:4873 , piece_id=BRIGIUKVPNWKUP4I5DDSRJRXJLYGYGIRAY7XQJDFPG55ZPYNXJUA, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.714480530 (585h39m24.585563565s) [noise=true , address=145.79.227.46:4601 , piece_id=4MYGT6TZ2TDT6LMEP4YARQUMC5GJAFWC6G4NKHQZSQKETBA5UNNA, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.715434436 (585h39m24.586517481s) [noise=true , address=145.79.227.47:4267 , piece_id=RMXSL3ZILRYQUQ3OKHYCLQBEK5TX7QJDZIKXWHE62WSVXO3L46NQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.716158786 (585h39m24.587241831s) [noise=true , address=145.79.227.17:4972 , piece_id=ZFAPAJQ5ZPOUMBG7UYVTWANJJKHDLGG7RFJ2KXPQ2JBT7UDOOJUQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.728524702 (585h39m24.599607747s) [noise=true , address=145.79.227.6:4257 , piece_id=IOJAITMSTXKQSUOPNTYK5GQFNUOMTNMDQXPHHUFEM36TIBE4JRCQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.729139773 (585h39m24.600222818s) [noise=true , address=145.79.227.52:4447 , piece_id=IAADPQ7CYD5JYLUVN52EDGHNU7AG6TBPI5OXODY3RRPVY3KEWUMQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.735310563 (585h39m24.606393598s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7I35P5AFVSGWYW7MQG6N7SJQ4ZAH24VB5NW6CUWQLZPXDYW26UA, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.739217192 (585h39m24.610300247s) [noise=true , address=145.79.227.19:4078 , piece_id=B3LIUGAXP6O6EERR3LV3ABFHRQ7U4FNYQOGWWFUQAGOBHJBBDLDQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.999002984 (585h39m24.870086019s) [noise=true , address=45.91.116.14:4192 , piece_id=5BHW2RX7XHTYXUIFALQBE6FEN3G6PCB475XFYO43E5QTCWN5AGUQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:25.090676861 (585h39m24.961759906s) [noise=true , address=145.79.227.2:4113 , piece_id=TBXZDUTJGXLB5WUFQUC5GV6AC3JRXNRRPF2YXNRC4VICCKNQMBHQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:25.143840390 (585h39m25.014923426s) [noise=true , address=45.91.116.12:4226 , piece_id=RJKGGPIZQ3YU5HLNNUF46TUGQFBR2JX62CUZEDGGBJWUTWXQJFBA, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:25.306778804 (585h39m25.177861839s) [noise=true , address=45.91.116.5:4300 , piece_id=KNWLZSLZASPIPNIRJF5BPR6XDWGISRGRLQ7YS5I7YJ2HCPB23RMA, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:25.306799223 (585h39m25.177882258s) [seg_pos={0 0}, upload=513404]: Segment reached optimal threshold of 13 pieces. Dec 30 01:08:25.307018811 (585h39m25.178101846s) [noise=true , address=45.91.116.7:4012 , piece_id=DC7YL6DHDQQOOLBLSHJ2NZOCZDBBDXG5HJUC3KYALUMYVEWC5C4Q, seg_pos={0 0}, upload=513404]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:59146->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:59146->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59146->45.91.116.7:4012: use of closed network connection Dec 30 01:08:25.307035833 (585h39m25.178118868s) [noise=true , address=45.91.116.7:4012 , piece_id=DC7YL6DHDQQOOLBLSHJ2NZOCZDBBDXG5HJUC3KYALUMYVEWC5C4Q, seg_pos={0 0}, upload=513404]: Long tail cancellation due to optimal threshold Dec 30 01:08:25.307214483 (585h39m25.178297518s) [noise=true , address=145.79.227.44:4194 , piece_id=GFBCVZH3EN54KVCOA4HW7M7URVVXLLO37KKTBVP53AHMTXM4DNGQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 30 01:08:25.307220625 (585h39m25.17830367s) [noise=true , address=145.79.227.44:4194 , piece_id=GFBCVZH3EN54KVCOA4HW7M7URVVXLLO37KKTBVP53AHMTXM4DNGQ, seg_pos={0 0}, upload=513404]: Long tail cancellation due to optimal threshold Dec 30 01:08:25.307628461 (585h39m25.178711496s) [noise=true , address=77.90.31.14:4661 , piece_id=CWJC67GO7HPCNOI57H7BJMGGVXWOY3FF2YWS6T7V65CMLD7DPKTA, seg_pos={0 0}, upload=513404]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:53704->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:53704->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53704->77.90.31.14:4661: use of closed network connection Dec 30 01:08:25.307636506 (585h39m25.178719551s) [noise=true , address=77.90.31.14:4661 , piece_id=CWJC67GO7HPCNOI57H7BJMGGVXWOY3FF2YWS6T7V65CMLD7DPKTA, seg_pos={0 0}, upload=513404]: Long tail cancellation due to optimal threshold Dec 30 01:08:25.307663077 (585h39m25.178746112s) [seg_pos={0 0}, upload=513404]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:08:25.307744743 (585h39m25.178827778s) [upload=513404]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:08:25.307750013 (585h39m25.178833048s) [upload=513404]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:08:25.604467427 (585h39m25.475550472s) [upload=513404]: Done uploading segments... Dec 30 01:08:25.604490570 (585h39m25.475573605s) [upload=513404]: Upload finished. err: <nil> Duration: 5.965734303s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 23.5417721s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:20:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 23.541792107s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X...X.........X......X.X...X....X.X.....X...............] duration: 6.031735433s trace: https://jaeger.dev.storj.io/trace/16af4eed1c703455 FAIL (2025-12-30 01:20:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: RJKGGPIZQ3YU5HLNNUF46TUGQFBR2JX62CUZEDGGBJWUTWXQJFBA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: KNWLZSLZASPIPNIRJF5BPR6XDWGISRGRLQ7YS5I7YJ2HCPB23RMA) context deadline exceeded ======LOGS====== g segments... Dec 30 01:08:22.484294554 (585h39m22.355377599s) [upload=513404]: Got next segment. Inline: false Dec 30 01:08:22.484338407 (585h39m22.355421442s) [upload=513404]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:08:22.484343197 (585h39m22.355426232s) [upload=513404]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:08:22.732399804 (585h39m22.603482839s) [seg_pos={0 0}, upload=513404]: Begin upload segment... Dec 30 01:08:22.732449068 (585h39m22.603532113s) [seg_pos={0 0}, upload=513404]: Done begin upload segment. Dec 30 01:08:22.732475969 (585h39m22.603559014s) [noise=true , address=145.79.227.2:4113 , piece_id=TBXZDUTJGXLB5WUFQUC5GV6AC3JRXNRRPF2YXNRC4VICCKNQMBHQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.732515504 (585h39m22.603598539s) [noise=true , address=145.79.227.44:4194 , piece_id=GFBCVZH3EN54KVCOA4HW7M7URVVXLLO37KKTBVP53AHMTXM4DNGQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.733197663 (585h39m22.604280698s) [noise=true , address=145.79.227.19:4078 , piece_id=B3LIUGAXP6O6EERR3LV3ABFHRQ7U4FNYQOGWWFUQAGOBHJBBDLDQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.733334544 (585h39m22.604417579s) [noise=true , address=45.91.116.7:4012 , piece_id=DC7YL6DHDQQOOLBLSHJ2NZOCZDBBDXG5HJUC3KYALUMYVEWC5C4Q, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.733717734 (585h39m22.604800779s) [noise=true , address=45.91.116.12:4226 , piece_id=RJKGGPIZQ3YU5HLNNUF46TUGQFBR2JX62CUZEDGGBJWUTWXQJFBA, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.734067009 (585h39m22.605150054s) [noise=true , address=145.79.227.51:4410 , piece_id=TEGW7H66VA3QQWKZDX7HIHDH7TCALOZWAVHA56G4YQX3I4O7V5MQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.734190263 (585h39m22.605273308s) [noise=true , address=45.91.116.14:4192 , piece_id=5BHW2RX7XHTYXUIFALQBE6FEN3G6PCB475XFYO43E5QTCWN5AGUQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.734493841 (585h39m22.605576876s) [noise=true , address=77.90.31.14:4661 , piece_id=CWJC67GO7HPCNOI57H7BJMGGVXWOY3FF2YWS6T7V65CMLD7DPKTA, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.734795215 (585h39m22.60587826s) [noise=true , address=145.79.227.46:4601 , piece_id=4MYGT6TZ2TDT6LMEP4YARQUMC5GJAFWC6G4NKHQZSQKETBA5UNNA, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.734812428 (585h39m22.605895473s) [noise=true , address=145.79.227.15:4873 , piece_id=BRIGIUKVPNWKUP4I5DDSRJRXJLYGYGIRAY7XQJDFPG55ZPYNXJUA, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.735186771 (585h39m22.606269816s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7I35P5AFVSGWYW7MQG6N7SJQ4ZAH24VB5NW6CUWQLZPXDYW26UA, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.735262475 (585h39m22.60634552s) [noise=true , address=145.79.227.52:4447 , piece_id=IAADPQ7CYD5JYLUVN52EDGHNU7AG6TBPI5OXODY3RRPVY3KEWUMQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.735531347 (585h39m22.606614382s) [noise=true , address=45.91.116.5:4300 , piece_id=KNWLZSLZASPIPNIRJF5BPR6XDWGISRGRLQ7YS5I7YJ2HCPB23RMA, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.735684438 (585h39m22.606767483s) [noise=true , address=145.79.227.17:4972 , piece_id=ZFAPAJQ5ZPOUMBG7UYVTWANJJKHDLGG7RFJ2KXPQ2JBT7UDOOJUQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.736100951 (585h39m22.607183996s) [noise=true , address=145.79.227.6:4257 , piece_id=IOJAITMSTXKQSUOPNTYK5GQFNUOMTNMDQXPHHUFEM36TIBE4JRCQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:22.736552080 (585h39m22.607635115s) [noise=true , address=145.79.227.47:4267 , piece_id=RMXSL3ZILRYQUQ3OKHYCLQBEK5TX7QJDZIKXWHE62WSVXO3L46NQ, seg_pos={0 0}, upload=513404]: Uploading piece... Dec 30 01:08:24.423934888 (585h39m24.295017933s) [upload=513404]: Next returned nil segment Dec 30 01:08:24.423954085 (585h39m24.29503713s) [upload=513404]: Waiting for error group managing segments... Dec 30 01:08:24.703932355 (585h39m24.5750154s) [noise=true , address=145.79.227.51:4410 , piece_id=TEGW7H66VA3QQWKZDX7HIHDH7TCALOZWAVHA56G4YQX3I4O7V5MQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.708283540 (585h39m24.579366585s) [noise=true , address=145.79.227.15:4873 , piece_id=BRIGIUKVPNWKUP4I5DDSRJRXJLYGYGIRAY7XQJDFPG55ZPYNXJUA, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.714480530 (585h39m24.585563565s) [noise=true , address=145.79.227.46:4601 , piece_id=4MYGT6TZ2TDT6LMEP4YARQUMC5GJAFWC6G4NKHQZSQKETBA5UNNA, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.715434436 (585h39m24.586517481s) [noise=true , address=145.79.227.47:4267 , piece_id=RMXSL3ZILRYQUQ3OKHYCLQBEK5TX7QJDZIKXWHE62WSVXO3L46NQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.716158786 (585h39m24.587241831s) [noise=true , address=145.79.227.17:4972 , piece_id=ZFAPAJQ5ZPOUMBG7UYVTWANJJKHDLGG7RFJ2KXPQ2JBT7UDOOJUQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.728524702 (585h39m24.599607747s) [noise=true , address=145.79.227.6:4257 , piece_id=IOJAITMSTXKQSUOPNTYK5GQFNUOMTNMDQXPHHUFEM36TIBE4JRCQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.729139773 (585h39m24.600222818s) [noise=true , address=145.79.227.52:4447 , piece_id=IAADPQ7CYD5JYLUVN52EDGHNU7AG6TBPI5OXODY3RRPVY3KEWUMQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.735310563 (585h39m24.606393598s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7I35P5AFVSGWYW7MQG6N7SJQ4ZAH24VB5NW6CUWQLZPXDYW26UA, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.739217192 (585h39m24.610300247s) [noise=true , address=145.79.227.19:4078 , piece_id=B3LIUGAXP6O6EERR3LV3ABFHRQ7U4FNYQOGWWFUQAGOBHJBBDLDQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:24.999002984 (585h39m24.870086019s) [noise=true , address=45.91.116.14:4192 , piece_id=5BHW2RX7XHTYXUIFALQBE6FEN3G6PCB475XFYO43E5QTCWN5AGUQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:25.090676861 (585h39m24.961759906s) [noise=true , address=145.79.227.2:4113 , piece_id=TBXZDUTJGXLB5WUFQUC5GV6AC3JRXNRRPF2YXNRC4VICCKNQMBHQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:25.143840390 (585h39m25.014923426s) [noise=true , address=45.91.116.12:4226 , piece_id=RJKGGPIZQ3YU5HLNNUF46TUGQFBR2JX62CUZEDGGBJWUTWXQJFBA, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:25.306778804 (585h39m25.177861839s) [noise=true , address=45.91.116.5:4300 , piece_id=KNWLZSLZASPIPNIRJF5BPR6XDWGISRGRLQ7YS5I7YJ2HCPB23RMA, seg_pos={0 0}, upload=513404]: Done uploading piece. err: <nil> Dec 30 01:08:25.306799223 (585h39m25.177882258s) [seg_pos={0 0}, upload=513404]: Segment reached optimal threshold of 13 pieces. Dec 30 01:08:25.307018811 (585h39m25.178101846s) [noise=true , address=45.91.116.7:4012 , piece_id=DC7YL6DHDQQOOLBLSHJ2NZOCZDBBDXG5HJUC3KYALUMYVEWC5C4Q, seg_pos={0 0}, upload=513404]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:59146->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:59146->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59146->45.91.116.7:4012: use of closed network connection Dec 30 01:08:25.307035833 (585h39m25.178118868s) [noise=true , address=45.91.116.7:4012 , piece_id=DC7YL6DHDQQOOLBLSHJ2NZOCZDBBDXG5HJUC3KYALUMYVEWC5C4Q, seg_pos={0 0}, upload=513404]: Long tail cancellation due to optimal threshold Dec 30 01:08:25.307214483 (585h39m25.178297518s) [noise=true , address=145.79.227.44:4194 , piece_id=GFBCVZH3EN54KVCOA4HW7M7URVVXLLO37KKTBVP53AHMTXM4DNGQ, seg_pos={0 0}, upload=513404]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 30 01:08:25.307220625 (585h39m25.17830367s) [noise=true , address=145.79.227.44:4194 , piece_id=GFBCVZH3EN54KVCOA4HW7M7URVVXLLO37KKTBVP53AHMTXM4DNGQ, seg_pos={0 0}, upload=513404]: Long tail cancellation due to optimal threshold Dec 30 01:08:25.307628461 (585h39m25.178711496s) [noise=true , address=77.90.31.14:4661 , piece_id=CWJC67GO7HPCNOI57H7BJMGGVXWOY3FF2YWS6T7V65CMLD7DPKTA, seg_pos={0 0}, upload=513404]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:53704->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:53704->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53704->77.90.31.14:4661: use of closed network connection Dec 30 01:08:25.307636506 (585h39m25.178719551s) [noise=true , address=77.90.31.14:4661 , piece_id=CWJC67GO7HPCNOI57H7BJMGGVXWOY3FF2YWS6T7V65CMLD7DPKTA, seg_pos={0 0}, upload=513404]: Long tail cancellation due to optimal threshold Dec 30 01:08:25.307663077 (585h39m25.178746112s) [seg_pos={0 0}, upload=513404]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:08:25.307744743 (585h39m25.178827778s) [upload=513404]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:08:25.307750013 (585h39m25.178833048s) [upload=513404]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:08:25.604467427 (585h39m25.475550472s) [upload=513404]: Done uploading segments... Dec 30 01:08:25.604490570 (585h39m25.475573605s) [upload=513404]: Upload finished. err: <nil> Duration: 6.031764128s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 27.173775134s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:20:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 27.173805432s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)