Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 15.389230257s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:43:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 15.389252509s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.......XX..........X.........X..X...............X.......X..] duration: 23.674888785s trace: https://jaeger.dev.storj.io/trace/14e86a12188d224b FAIL (2025-12-26 16:43:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W, Piece ID: 6TIM4CCDRYOS2RIB4MJXG55NHN5AK5IM2WCJBRFZQTFGFBR7RUXQ) context deadline exceeded; piecestore: (Node ID: 12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, Piece ID: 44TLRO3WIRAF45DAXO3TPQ2GHC2U3OFN2KLU73HG563552DGWUMA) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: BRJT6UUDGWS5T3KWTAW4ESJXFULND4FB6VQVYOSYRECDNNJ56SJA) context deadline exceeded ======LOGS====== 6 , piece_id=BRJT6UUDGWS5T3KWTAW4ESJXFULND4FB6VQVYOSYRECDNNJ56SJA, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.639224730 (505h11m56.510307775s) [noise=true , address=145.79.227.40:4067 , piece_id=7ATX5TWB3NK4BARIMVPL4SYF2KUOCUR7JGYBCAK4W46EZASZIPSQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.639702510 (505h11m56.510785555s) [noise=true , address=145.79.227.33:4897 , piece_id=QLFZSNPKDCEFA5TQ5TCUIT3T7A4JQ3CX3AT5T4N7M64GEYZ647LA, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.640116127 (505h11m56.511199173s) [noise=true , address=77.90.31.18:4812 , piece_id=4YFXYQYUW3FBVXMVJX47PLUU6QBJWXKSWULXTO35CHVZXPLVPQEQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.640171784 (505h11m56.511254829s) [noise=true , address=5.231.73.2:4027 , piece_id=Y6HUQVZS4RDZFGJCFHXHGY5GDBVNGZUNUAC5OXORLLD4GWA3DRLQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.640518985 (505h11m56.51160204s) [noise=true , address=5.231.73.5:4029 , piece_id=HJICME4ZINBCPHVF2KUXILTQQIAYKQGIL5E4FT7WQHPX23YOEQ4A, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.640819537 (505h11m56.511902582s) [noise=true , address=64.49.13.27:4965 , piece_id=JNYCPACXREVCJYKSXHEUAFEGGEI5QPW4VSWLJFLPSLUTT65AYLXQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.640975103 (505h11m56.512058149s) [noise=true , address=64.49.13.155:4924 , piece_id=5WEKSQI423JPQAZYOB27OV4RHWFVWSNH3A3XB3CLCS7T3DPRKPAQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.641317956 (505h11m56.512401002s) [noise=true , address=77.90.31.21:4905 , piece_id=CARCPTBOEX4JRA3JHLGBMMSHYRZTFVVQALZ4WHBVJV7LOC5JCM5A, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.641413087 (505h11m56.512496132s) [noise=true , address=145.79.227.17:4978 , piece_id=JDFCSH37HLE7RXUJVIC4HU2TNDVNPCKFWFUNOL7AZCALMIPZJWPA, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.642007388 (505h11m56.513090434s) [noise=true , address=145.79.227.43:4175 , piece_id=T63E5RK7U3N7GHEAOQAFHCV5UPWPC2RNGBCTKD7XB5ODY44P5TMQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.642606971 (505h11m56.513690016s) [noise=true , address=145.79.227.8:4340 , piece_id=2XM5IYMXC3T4RZGP5STFD7X44GA5LLJ2YBLIZKZCMPV2EGMITP5Q, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.643110820 (505h11m56.514193865s) [noise=true , address=193.149.164.143:4703 , piece_id=EG5EHWBVYP4P3CCCDTO2IWZDNTQCMI7JTSRECYNMQVDBXQKN2WCQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.643616243 (505h11m56.514699288s) [noise=true , address=145.79.227.5:4255 , piece_id=2BXGXSZ76NKMPKLDE5NRJ622SFOLZT3FYAKOJ7EQSGKO2JBYDTJA, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.644160849 (505h11m56.515243894s) [noise=true , address=77.90.31.14:4671 , piece_id=6TIM4CCDRYOS2RIB4MJXG55NHN5AK5IM2WCJBRFZQTFGFBR7RUXQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:57.909019390 (505h11m57.780102425s) [upload=443308]: Next returned nil segment Dec 26 16:40:57.909046512 (505h11m57.780129547s) [upload=443308]: Waiting for error group managing segments... Dec 26 16:40:58.279921182 (505h11m58.151004217s) [noise=true , address=64.49.13.27:4965 , piece_id=JNYCPACXREVCJYKSXHEUAFEGGEI5QPW4VSWLJFLPSLUTT65AYLXQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.509473179 (505h11m58.380556214s) [noise=true , address=145.79.227.43:4175 , piece_id=T63E5RK7U3N7GHEAOQAFHCV5UPWPC2RNGBCTKD7XB5ODY44P5TMQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.526249174 (505h11m58.397332209s) [noise=true , address=145.79.227.17:4978 , piece_id=JDFCSH37HLE7RXUJVIC4HU2TNDVNPCKFWFUNOL7AZCALMIPZJWPA, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.534222977 (505h11m58.405306012s) [noise=true , address=145.79.227.8:4340 , piece_id=2XM5IYMXC3T4RZGP5STFD7X44GA5LLJ2YBLIZKZCMPV2EGMITP5Q, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.624481034 (505h11m58.495564069s) [noise=true , address=64.49.13.155:4924 , piece_id=5WEKSQI423JPQAZYOB27OV4RHWFVWSNH3A3XB3CLCS7T3DPRKPAQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.650754864 (505h11m58.521837909s) [noise=true , address=5.231.73.5:4029 , piece_id=HJICME4ZINBCPHVF2KUXILTQQIAYKQGIL5E4FT7WQHPX23YOEQ4A, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.673199081 (505h11m58.544282117s) [noise=true , address=77.90.31.13:4646 , piece_id=BRJT6UUDGWS5T3KWTAW4ESJXFULND4FB6VQVYOSYRECDNNJ56SJA, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.745181288 (505h11m58.616264323s) [noise=true , address=77.90.31.12:4610 , piece_id=44TLRO3WIRAF45DAXO3TPQ2GHC2U3OFN2KLU73HG563552DGWUMA, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.779227909 (505h11m58.650310954s) [noise=true , address=5.231.73.2:4027 , piece_id=Y6HUQVZS4RDZFGJCFHXHGY5GDBVNGZUNUAC5OXORLLD4GWA3DRLQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.073598071 (505h11m58.944681106s) [noise=true , address=77.90.31.17:4784 , piece_id=7CMDKQQHDP3P7M23U3ZP2GJVQC4KUCKCB4S6BBXHUNWBX6KS3DHQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.116295695 (505h11m58.98737874s) [noise=true , address=77.90.31.14:4671 , piece_id=6TIM4CCDRYOS2RIB4MJXG55NHN5AK5IM2WCJBRFZQTFGFBR7RUXQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.196854433 (505h11m59.067937478s) [noise=true , address=193.149.164.143:4703 , piece_id=EG5EHWBVYP4P3CCCDTO2IWZDNTQCMI7JTSRECYNMQVDBXQKN2WCQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.426409626 (505h11m59.297492671s) [noise=true , address=145.79.227.44:4200 , piece_id=732NRQEMNGUE6WNMUFDRB3IEOOZKGLRPDLGOID54ADB4WHQW6QCQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.535650219 (505h11m59.406733264s) [noise=true , address=145.79.227.33:4897 , piece_id=QLFZSNPKDCEFA5TQ5TCUIT3T7A4JQ3CX3AT5T4N7M64GEYZ647LA, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.535675147 (505h11m59.406758192s) [seg_pos={0 0}, upload=443308]: Segment reached optimal threshold of 14 pieces. Dec 26 16:40:59.535893853 (505h11m59.406976898s) [noise=true , address=145.79.227.5:4255 , piece_id=2BXGXSZ76NKMPKLDE5NRJ622SFOLZT3FYAKOJ7EQSGKO2JBYDTJA, seg_pos={0 0}, upload=443308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fTNKdc6AZbkfsKRFZwTwV6wPz9j9ZD9t2eNkHBJNLMNcGh9gR): protocol: write tcp 188.166.225.170:34966->145.79.227.5:4255: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.5:4255: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.5:4255: use of closed network connection Dec 26 16:40:59.535902018 (505h11m59.406985053s) [noise=true , address=145.79.227.5:4255 , piece_id=2BXGXSZ76NKMPKLDE5NRJ622SFOLZT3FYAKOJ7EQSGKO2JBYDTJA, seg_pos={0 0}, upload=443308]: Long tail cancellation due to optimal threshold Dec 26 16:40:59.536140953 (505h11m59.407224008s) [noise=true , address=145.79.227.40:4067 , piece_id=7ATX5TWB3NK4BARIMVPL4SYF2KUOCUR7JGYBCAK4W46EZASZIPSQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BPYCUuXmyUAZWEgWk2GWTnmT4Sh2pvnvHLHaFaArX3iSvkrnq): protocol: expected piece hash; context canceled; EOF Dec 26 16:40:59.536148949 (505h11m59.407231994s) [noise=true , address=145.79.227.40:4067 , piece_id=7ATX5TWB3NK4BARIMVPL4SYF2KUOCUR7JGYBCAK4W46EZASZIPSQ, seg_pos={0 0}, upload=443308]: Long tail cancellation due to optimal threshold Dec 26 16:40:59.536341756 (505h11m59.407424791s) [noise=true , address=77.90.31.21:4905 , piece_id=CARCPTBOEX4JRA3JHLGBMMSHYRZTFVVQALZ4WHBVJV7LOC5JCM5A, seg_pos={0 0}, upload=443308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK): protocol: write tcp 188.166.225.170:33344->77.90.31.21:4905: use of closed network connection; write tcp 188.166.225.170:33344->77.90.31.21:4905: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33344->77.90.31.21:4905: use of closed network connection Dec 26 16:40:59.536349891 (505h11m59.407432936s) [noise=true , address=77.90.31.21:4905 , piece_id=CARCPTBOEX4JRA3JHLGBMMSHYRZTFVVQALZ4WHBVJV7LOC5JCM5A, seg_pos={0 0}, upload=443308]: Long tail cancellation due to optimal threshold Dec 26 16:40:59.536580220 (505h11m59.407663265s) [noise=true , address=77.90.31.18:4812 , piece_id=4YFXYQYUW3FBVXMVJX47PLUU6QBJWXKSWULXTO35CHVZXPLVPQEQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn): protocol: expected piece hash; context canceled; EOF Dec 26 16:40:59.536592623 (505h11m59.407675658s) [noise=true , address=77.90.31.18:4812 , piece_id=4YFXYQYUW3FBVXMVJX47PLUU6QBJWXKSWULXTO35CHVZXPLVPQEQ, seg_pos={0 0}, upload=443308]: Long tail cancellation due to optimal threshold Dec 26 16:40:59.536637570 (505h11m59.407720605s) [seg_pos={0 0}, upload=443308]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:40:59.536765813 (505h11m59.407848858s) [upload=443308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:40:59.536773297 (505h11m59.407856342s) [upload=443308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:40:59.833034932 (505h11m59.704117977s) [upload=443308]: Done uploading segments... Dec 26 16:40:59.833058236 (505h11m59.704141271s) [upload=443308]: Upload finished. err: <nil> Duration: 23.674918021s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [....X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X..X...X] duration: 30.002576816s trace: https://jaeger.dev.storj.io/trace/7c1ebe55b0448509 FAIL (2025-12-26 16:42:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ZHLJZL553VYDC76SRKSAL5IKSVMOV6BZOH3KCW2VWC7UE24H7HJQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: GXJMYIVD4PWIRAZEHIELQVFCSO5BC4ZTVSC4NXW3ELBMGZCOJIDA) context deadline exceeded ======LOGS====== ents... Dec 26 16:42:15.287484886 (505h13m15.158567931s) [upload=443326]: Got next segment. Inline: false Dec 26 16:42:15.287525874 (505h13m15.158608919s) [upload=443326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:42:15.287531806 (505h13m15.158614841s) [upload=443326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:42:15.539659110 (505h13m15.410742165s) [seg_pos={0 0}, upload=443326]: Begin upload segment... Dec 26 16:42:15.539714665 (505h13m15.41079771s) [seg_pos={0 0}, upload=443326]: Done begin upload segment. Dec 26 16:42:15.539742378 (505h13m15.410825413s) [noise=true , address=145.79.227.50:4374 , piece_id=DS7MCCAYIDWG6IBGLGU3GWJK4GZGDJJAH2ELP7CP4WUYR2KPPWZA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.539810518 (505h13m15.410893553s) [noise=true , address=145.79.227.12:4767 , piece_id=AMDHJU7TH4G2HVM2OSJ4OTSJJWBNPZKNCGEUJ3QHH232YIWQ5WFA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.540298437 (505h13m15.411381482s) [noise=true , address=145.79.227.11:4732 , piece_id=YFPFZKKHILV4Q2674MJY4QFFJVJ6ILBDLAI5EINQQ5XFXUMHK3DQ, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.540575104 (505h13m15.411658139s) [noise=true , address=45.91.116.9:4263 , piece_id=ZHLJZL553VYDC76SRKSAL5IKSVMOV6BZOH3KCW2VWC7UE24H7HJQ, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.540939098 (505h13m15.412022133s) [noise=true , address=145.79.227.48:4301 , piece_id=DDGMKU3SO3OG7N26A46CGRFIR6NDW5HQRQYVLBDIEAABH4PIFGKQ, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.541288553 (505h13m15.412371588s) [noise=true , address=145.79.227.32:4836 , piece_id=DSY3GJEJOP75QHYX6M655MSOXVYTLLPYB3AMVC7GAIGBM3LNCL2A, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.541665580 (505h13m15.412748615s) [noise=true , address=145.79.227.5:4222 , piece_id=C42KVZ2ZVNQJCU65NQDKLREJKCFJUUOV6X2KIGLDN75UBRK6JWTA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.542056155 (505h13m15.41313919s) [noise=true , address=145.79.227.47:4265 , piece_id=AVVNURIXLPZ33ZLYIR45L25PPBIHXP27B7SKKM7AYAMXQQ55QTNA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.542386443 (505h13m15.413469488s) [noise=true , address=45.91.116.5:4297 , piece_id=GXJMYIVD4PWIRAZEHIELQVFCSO5BC4ZTVSC4NXW3ELBMGZCOJIDA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.543833599 (505h13m15.414916634s) [noise=true , address=145.79.227.40:4051 , piece_id=N7XR6BGZI7VYKAEATP24L2I3HNQKG3FOX772MX6GMIOMVY4ZC7PA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.544814948 (505h13m15.415897993s) [noise=true , address=45.91.116.12:4227 , piece_id=MV2DAMZE6FY5NR27HPKNGETBZEN34LEQSOOUIT5N6OTXXGBXZQOA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.546040822 (505h13m15.417123857s) [noise=true , address=145.79.227.15:4875 , piece_id=NODF22YJ2FI6VIZFOPMZF4NESXP3QN5PQU5NP7EIGNWP5A24CJNQ, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.547299118 (505h13m15.418382163s) [noise=true , address=145.79.227.52:4446 , piece_id=2OFNORXTUF3ZM5LSXMGU3QM2M4ICVVXZHSJK7EBOZGF7HTQZML3A, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.548767193 (505h13m15.419850238s) [noise=true , address=145.79.227.13:4803 , piece_id=ETOIFVMCRWWLU3CEHKROI6FK6GM7CI3CLYMSEBQDLV765MHYDJBA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.550118606 (505h13m15.421201641s) [noise=true , address=45.91.116.13:4154 , piece_id=2NGV2JXC7MIXF3M26TO4URHR4LX4CBWJLETY5FDFYCJWNGMAAKBQ, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.551361323 (505h13m15.422444358s) [noise=true , address=145.79.227.42:4122 , piece_id=5DBWMUF6ZB6T4BTMWBWOL347GMEFH4KEUSV3ECKFYWMW6DQCWD6Q, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:17.259877936 (505h13m17.130960981s) [upload=443326]: Next returned nil segment Dec 26 16:42:17.259896352 (505h13m17.130979397s) [upload=443326]: Waiting for error group managing segments... Dec 26 16:42:17.539066701 (505h13m17.410149736s) [noise=true , address=145.79.227.32:4836 , piece_id=DSY3GJEJOP75QHYX6M655MSOXVYTLLPYB3AMVC7GAIGBM3LNCL2A, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.548192668 (505h13m17.419275713s) [noise=true , address=145.79.227.48:4301 , piece_id=DDGMKU3SO3OG7N26A46CGRFIR6NDW5HQRQYVLBDIEAABH4PIFGKQ, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.555516845 (505h13m17.42659989s) [noise=true , address=145.79.227.52:4446 , piece_id=2OFNORXTUF3ZM5LSXMGU3QM2M4ICVVXZHSJK7EBOZGF7HTQZML3A, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.558493903 (505h13m17.429576939s) [noise=true , address=145.79.227.47:4265 , piece_id=AVVNURIXLPZ33ZLYIR45L25PPBIHXP27B7SKKM7AYAMXQQ55QTNA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.563433349 (505h13m17.434516394s) [noise=true , address=145.79.227.11:4732 , piece_id=YFPFZKKHILV4Q2674MJY4QFFJVJ6ILBDLAI5EINQQ5XFXUMHK3DQ, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.568861816 (505h13m17.439944861s) [noise=true , address=145.79.227.15:4875 , piece_id=NODF22YJ2FI6VIZFOPMZF4NESXP3QN5PQU5NP7EIGNWP5A24CJNQ, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.572164695 (505h13m17.44324773s) [noise=true , address=145.79.227.50:4374 , piece_id=DS7MCCAYIDWG6IBGLGU3GWJK4GZGDJJAH2ELP7CP4WUYR2KPPWZA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.572447454 (505h13m17.443530499s) [noise=true , address=145.79.227.42:4122 , piece_id=5DBWMUF6ZB6T4BTMWBWOL347GMEFH4KEUSV3ECKFYWMW6DQCWD6Q, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.619080281 (505h13m17.490163326s) [noise=true , address=145.79.227.5:4222 , piece_id=C42KVZ2ZVNQJCU65NQDKLREJKCFJUUOV6X2KIGLDN75UBRK6JWTA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.630157514 (505h13m17.501240549s) [noise=true , address=145.79.227.13:4803 , piece_id=ETOIFVMCRWWLU3CEHKROI6FK6GM7CI3CLYMSEBQDLV765MHYDJBA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.769512389 (505h13m17.640595424s) [noise=true , address=145.79.227.12:4767 , piece_id=AMDHJU7TH4G2HVM2OSJ4OTSJJWBNPZKNCGEUJ3QHH232YIWQ5WFA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:18.093770894 (505h13m17.96485393s) [noise=true , address=45.91.116.5:4297 , piece_id=GXJMYIVD4PWIRAZEHIELQVFCSO5BC4ZTVSC4NXW3ELBMGZCOJIDA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:18.109269015 (505h13m17.98035206s) [noise=true , address=45.91.116.9:4263 , piece_id=ZHLJZL553VYDC76SRKSAL5IKSVMOV6BZOH3KCW2VWC7UE24H7HJQ, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:18.109288753 (505h13m17.980371798s) [seg_pos={0 0}, upload=443326]: Segment reached optimal threshold of 13 pieces. Dec 26 16:42:18.109480638 (505h13m17.980563683s) [noise=true , address=45.91.116.13:4154 , piece_id=2NGV2JXC7MIXF3M26TO4URHR4LX4CBWJLETY5FDFYCJWNGMAAKBQ, seg_pos={0 0}, upload=443326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:37840->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:37840->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37840->45.91.116.13:4154: use of closed network connection Dec 26 16:42:18.109490316 (505h13m17.980573382s) [noise=true , address=45.91.116.13:4154 , piece_id=2NGV2JXC7MIXF3M26TO4URHR4LX4CBWJLETY5FDFYCJWNGMAAKBQ, seg_pos={0 0}, upload=443326]: Long tail cancellation due to optimal threshold Dec 26 16:42:18.109549650 (505h13m17.980632685s) [noise=true , address=45.91.116.12:4227 , piece_id=MV2DAMZE6FY5NR27HPKNGETBZEN34LEQSOOUIT5N6OTXXGBXZQOA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:48030->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:48030->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48030->45.91.116.12:4227: use of closed network connection Dec 26 16:42:18.109556072 (505h13m17.980639107s) [noise=true , address=45.91.116.12:4227 , piece_id=MV2DAMZE6FY5NR27HPKNGETBZEN34LEQSOOUIT5N6OTXXGBXZQOA, seg_pos={0 0}, upload=443326]: Long tail cancellation due to optimal threshold Dec 26 16:42:18.109739211 (505h13m17.980822266s) [noise=true , address=145.79.227.40:4051 , piece_id=N7XR6BGZI7VYKAEATP24L2I3HNQKG3FOX772MX6GMIOMVY4ZC7PA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 16:42:18.109750192 (505h13m17.980833227s) [noise=true , address=145.79.227.40:4051 , piece_id=N7XR6BGZI7VYKAEATP24L2I3HNQKG3FOX772MX6GMIOMVY4ZC7PA, seg_pos={0 0}, upload=443326]: Long tail cancellation due to optimal threshold Dec 26 16:42:18.109769067 (505h13m17.980852102s) [seg_pos={0 0}, upload=443326]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:42:18.109965982 (505h13m17.981049028s) [upload=443326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:42:18.109975440 (505h13m17.981058505s) [upload=443326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:42:18.398418937 (505h13m18.269501982s) [upload=443326]: Done uploading segments... Dec 26 16:42:18.398441982 (505h13m18.269525027s) [upload=443326]: Upload finished. err: <nil> Duration: 30.002595202s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 8.710356416s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:42:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 8.710382065s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.......XX..........X.........X..X...............X.......X.] duration: 30.74927478s trace: https://jaeger.dev.storj.io/trace/14e86a12188d224b FAIL (2025-12-26 16:42:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W, Piece ID: 6TIM4CCDRYOS2RIB4MJXG55NHN5AK5IM2WCJBRFZQTFGFBR7RUXQ) context deadline exceeded; piecestore: (Node ID: 12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, Piece ID: 44TLRO3WIRAF45DAXO3TPQ2GHC2U3OFN2KLU73HG563552DGWUMA) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: BRJT6UUDGWS5T3KWTAW4ESJXFULND4FB6VQVYOSYRECDNNJ56SJA) context deadline exceeded ======LOGS====== 6 , piece_id=BRJT6UUDGWS5T3KWTAW4ESJXFULND4FB6VQVYOSYRECDNNJ56SJA, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.639224730 (505h11m56.510307775s) [noise=true , address=145.79.227.40:4067 , piece_id=7ATX5TWB3NK4BARIMVPL4SYF2KUOCUR7JGYBCAK4W46EZASZIPSQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.639702510 (505h11m56.510785555s) [noise=true , address=145.79.227.33:4897 , piece_id=QLFZSNPKDCEFA5TQ5TCUIT3T7A4JQ3CX3AT5T4N7M64GEYZ647LA, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.640116127 (505h11m56.511199173s) [noise=true , address=77.90.31.18:4812 , piece_id=4YFXYQYUW3FBVXMVJX47PLUU6QBJWXKSWULXTO35CHVZXPLVPQEQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.640171784 (505h11m56.511254829s) [noise=true , address=5.231.73.2:4027 , piece_id=Y6HUQVZS4RDZFGJCFHXHGY5GDBVNGZUNUAC5OXORLLD4GWA3DRLQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.640518985 (505h11m56.51160204s) [noise=true , address=5.231.73.5:4029 , piece_id=HJICME4ZINBCPHVF2KUXILTQQIAYKQGIL5E4FT7WQHPX23YOEQ4A, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.640819537 (505h11m56.511902582s) [noise=true , address=64.49.13.27:4965 , piece_id=JNYCPACXREVCJYKSXHEUAFEGGEI5QPW4VSWLJFLPSLUTT65AYLXQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.640975103 (505h11m56.512058149s) [noise=true , address=64.49.13.155:4924 , piece_id=5WEKSQI423JPQAZYOB27OV4RHWFVWSNH3A3XB3CLCS7T3DPRKPAQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.641317956 (505h11m56.512401002s) [noise=true , address=77.90.31.21:4905 , piece_id=CARCPTBOEX4JRA3JHLGBMMSHYRZTFVVQALZ4WHBVJV7LOC5JCM5A, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.641413087 (505h11m56.512496132s) [noise=true , address=145.79.227.17:4978 , piece_id=JDFCSH37HLE7RXUJVIC4HU2TNDVNPCKFWFUNOL7AZCALMIPZJWPA, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.642007388 (505h11m56.513090434s) [noise=true , address=145.79.227.43:4175 , piece_id=T63E5RK7U3N7GHEAOQAFHCV5UPWPC2RNGBCTKD7XB5ODY44P5TMQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.642606971 (505h11m56.513690016s) [noise=true , address=145.79.227.8:4340 , piece_id=2XM5IYMXC3T4RZGP5STFD7X44GA5LLJ2YBLIZKZCMPV2EGMITP5Q, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.643110820 (505h11m56.514193865s) [noise=true , address=193.149.164.143:4703 , piece_id=EG5EHWBVYP4P3CCCDTO2IWZDNTQCMI7JTSRECYNMQVDBXQKN2WCQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.643616243 (505h11m56.514699288s) [noise=true , address=145.79.227.5:4255 , piece_id=2BXGXSZ76NKMPKLDE5NRJ622SFOLZT3FYAKOJ7EQSGKO2JBYDTJA, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.644160849 (505h11m56.515243894s) [noise=true , address=77.90.31.14:4671 , piece_id=6TIM4CCDRYOS2RIB4MJXG55NHN5AK5IM2WCJBRFZQTFGFBR7RUXQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:57.909019390 (505h11m57.780102425s) [upload=443308]: Next returned nil segment Dec 26 16:40:57.909046512 (505h11m57.780129547s) [upload=443308]: Waiting for error group managing segments... Dec 26 16:40:58.279921182 (505h11m58.151004217s) [noise=true , address=64.49.13.27:4965 , piece_id=JNYCPACXREVCJYKSXHEUAFEGGEI5QPW4VSWLJFLPSLUTT65AYLXQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.509473179 (505h11m58.380556214s) [noise=true , address=145.79.227.43:4175 , piece_id=T63E5RK7U3N7GHEAOQAFHCV5UPWPC2RNGBCTKD7XB5ODY44P5TMQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.526249174 (505h11m58.397332209s) [noise=true , address=145.79.227.17:4978 , piece_id=JDFCSH37HLE7RXUJVIC4HU2TNDVNPCKFWFUNOL7AZCALMIPZJWPA, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.534222977 (505h11m58.405306012s) [noise=true , address=145.79.227.8:4340 , piece_id=2XM5IYMXC3T4RZGP5STFD7X44GA5LLJ2YBLIZKZCMPV2EGMITP5Q, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.624481034 (505h11m58.495564069s) [noise=true , address=64.49.13.155:4924 , piece_id=5WEKSQI423JPQAZYOB27OV4RHWFVWSNH3A3XB3CLCS7T3DPRKPAQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.650754864 (505h11m58.521837909s) [noise=true , address=5.231.73.5:4029 , piece_id=HJICME4ZINBCPHVF2KUXILTQQIAYKQGIL5E4FT7WQHPX23YOEQ4A, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.673199081 (505h11m58.544282117s) [noise=true , address=77.90.31.13:4646 , piece_id=BRJT6UUDGWS5T3KWTAW4ESJXFULND4FB6VQVYOSYRECDNNJ56SJA, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.745181288 (505h11m58.616264323s) [noise=true , address=77.90.31.12:4610 , piece_id=44TLRO3WIRAF45DAXO3TPQ2GHC2U3OFN2KLU73HG563552DGWUMA, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.779227909 (505h11m58.650310954s) [noise=true , address=5.231.73.2:4027 , piece_id=Y6HUQVZS4RDZFGJCFHXHGY5GDBVNGZUNUAC5OXORLLD4GWA3DRLQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.073598071 (505h11m58.944681106s) [noise=true , address=77.90.31.17:4784 , piece_id=7CMDKQQHDP3P7M23U3ZP2GJVQC4KUCKCB4S6BBXHUNWBX6KS3DHQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.116295695 (505h11m58.98737874s) [noise=true , address=77.90.31.14:4671 , piece_id=6TIM4CCDRYOS2RIB4MJXG55NHN5AK5IM2WCJBRFZQTFGFBR7RUXQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.196854433 (505h11m59.067937478s) [noise=true , address=193.149.164.143:4703 , piece_id=EG5EHWBVYP4P3CCCDTO2IWZDNTQCMI7JTSRECYNMQVDBXQKN2WCQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.426409626 (505h11m59.297492671s) [noise=true , address=145.79.227.44:4200 , piece_id=732NRQEMNGUE6WNMUFDRB3IEOOZKGLRPDLGOID54ADB4WHQW6QCQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.535650219 (505h11m59.406733264s) [noise=true , address=145.79.227.33:4897 , piece_id=QLFZSNPKDCEFA5TQ5TCUIT3T7A4JQ3CX3AT5T4N7M64GEYZ647LA, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.535675147 (505h11m59.406758192s) [seg_pos={0 0}, upload=443308]: Segment reached optimal threshold of 14 pieces. Dec 26 16:40:59.535893853 (505h11m59.406976898s) [noise=true , address=145.79.227.5:4255 , piece_id=2BXGXSZ76NKMPKLDE5NRJ622SFOLZT3FYAKOJ7EQSGKO2JBYDTJA, seg_pos={0 0}, upload=443308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fTNKdc6AZbkfsKRFZwTwV6wPz9j9ZD9t2eNkHBJNLMNcGh9gR): protocol: write tcp 188.166.225.170:34966->145.79.227.5:4255: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.5:4255: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.5:4255: use of closed network connection Dec 26 16:40:59.535902018 (505h11m59.406985053s) [noise=true , address=145.79.227.5:4255 , piece_id=2BXGXSZ76NKMPKLDE5NRJ622SFOLZT3FYAKOJ7EQSGKO2JBYDTJA, seg_pos={0 0}, upload=443308]: Long tail cancellation due to optimal threshold Dec 26 16:40:59.536140953 (505h11m59.407224008s) [noise=true , address=145.79.227.40:4067 , piece_id=7ATX5TWB3NK4BARIMVPL4SYF2KUOCUR7JGYBCAK4W46EZASZIPSQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BPYCUuXmyUAZWEgWk2GWTnmT4Sh2pvnvHLHaFaArX3iSvkrnq): protocol: expected piece hash; context canceled; EOF Dec 26 16:40:59.536148949 (505h11m59.407231994s) [noise=true , address=145.79.227.40:4067 , piece_id=7ATX5TWB3NK4BARIMVPL4SYF2KUOCUR7JGYBCAK4W46EZASZIPSQ, seg_pos={0 0}, upload=443308]: Long tail cancellation due to optimal threshold Dec 26 16:40:59.536341756 (505h11m59.407424791s) [noise=true , address=77.90.31.21:4905 , piece_id=CARCPTBOEX4JRA3JHLGBMMSHYRZTFVVQALZ4WHBVJV7LOC5JCM5A, seg_pos={0 0}, upload=443308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK): protocol: write tcp 188.166.225.170:33344->77.90.31.21:4905: use of closed network connection; write tcp 188.166.225.170:33344->77.90.31.21:4905: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33344->77.90.31.21:4905: use of closed network connection Dec 26 16:40:59.536349891 (505h11m59.407432936s) [noise=true , address=77.90.31.21:4905 , piece_id=CARCPTBOEX4JRA3JHLGBMMSHYRZTFVVQALZ4WHBVJV7LOC5JCM5A, seg_pos={0 0}, upload=443308]: Long tail cancellation due to optimal threshold Dec 26 16:40:59.536580220 (505h11m59.407663265s) [noise=true , address=77.90.31.18:4812 , piece_id=4YFXYQYUW3FBVXMVJX47PLUU6QBJWXKSWULXTO35CHVZXPLVPQEQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn): protocol: expected piece hash; context canceled; EOF Dec 26 16:40:59.536592623 (505h11m59.407675658s) [noise=true , address=77.90.31.18:4812 , piece_id=4YFXYQYUW3FBVXMVJX47PLUU6QBJWXKSWULXTO35CHVZXPLVPQEQ, seg_pos={0 0}, upload=443308]: Long tail cancellation due to optimal threshold Dec 26 16:40:59.536637570 (505h11m59.407720605s) [seg_pos={0 0}, upload=443308]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:40:59.536765813 (505h11m59.407848858s) [upload=443308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:40:59.536773297 (505h11m59.407856342s) [upload=443308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:40:59.833034932 (505h11m59.704117977s) [upload=443308]: Done uploading segments... Dec 26 16:40:59.833058236 (505h11m59.704141271s) [upload=443308]: Upload finished. err: <nil> Duration: 30.749305188s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.X.............XXX...X...XX.....XX..........X......XXX...X..] duration: 4.942821494s trace: https://jaeger.dev.storj.io/trace/69cd2adc1b6c0e6d FAIL (2025-12-26 16:42:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.942846872s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X.......XX..........X.........X..X...............X.......X] duration: 1m0.002047236s trace: https://jaeger.dev.storj.io/trace/14e86a12188d224b FAIL (2025-12-26 16:41:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W, Piece ID: 6TIM4CCDRYOS2RIB4MJXG55NHN5AK5IM2WCJBRFZQTFGFBR7RUXQ) context deadline exceeded; piecestore: (Node ID: 12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, Piece ID: 44TLRO3WIRAF45DAXO3TPQ2GHC2U3OFN2KLU73HG563552DGWUMA) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: BRJT6UUDGWS5T3KWTAW4ESJXFULND4FB6VQVYOSYRECDNNJ56SJA) context deadline exceeded ======LOGS====== 6 , piece_id=BRJT6UUDGWS5T3KWTAW4ESJXFULND4FB6VQVYOSYRECDNNJ56SJA, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.639224730 (505h11m56.510307775s) [noise=true , address=145.79.227.40:4067 , piece_id=7ATX5TWB3NK4BARIMVPL4SYF2KUOCUR7JGYBCAK4W46EZASZIPSQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.639702510 (505h11m56.510785555s) [noise=true , address=145.79.227.33:4897 , piece_id=QLFZSNPKDCEFA5TQ5TCUIT3T7A4JQ3CX3AT5T4N7M64GEYZ647LA, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.640116127 (505h11m56.511199173s) [noise=true , address=77.90.31.18:4812 , piece_id=4YFXYQYUW3FBVXMVJX47PLUU6QBJWXKSWULXTO35CHVZXPLVPQEQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.640171784 (505h11m56.511254829s) [noise=true , address=5.231.73.2:4027 , piece_id=Y6HUQVZS4RDZFGJCFHXHGY5GDBVNGZUNUAC5OXORLLD4GWA3DRLQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.640518985 (505h11m56.51160204s) [noise=true , address=5.231.73.5:4029 , piece_id=HJICME4ZINBCPHVF2KUXILTQQIAYKQGIL5E4FT7WQHPX23YOEQ4A, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.640819537 (505h11m56.511902582s) [noise=true , address=64.49.13.27:4965 , piece_id=JNYCPACXREVCJYKSXHEUAFEGGEI5QPW4VSWLJFLPSLUTT65AYLXQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.640975103 (505h11m56.512058149s) [noise=true , address=64.49.13.155:4924 , piece_id=5WEKSQI423JPQAZYOB27OV4RHWFVWSNH3A3XB3CLCS7T3DPRKPAQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.641317956 (505h11m56.512401002s) [noise=true , address=77.90.31.21:4905 , piece_id=CARCPTBOEX4JRA3JHLGBMMSHYRZTFVVQALZ4WHBVJV7LOC5JCM5A, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.641413087 (505h11m56.512496132s) [noise=true , address=145.79.227.17:4978 , piece_id=JDFCSH37HLE7RXUJVIC4HU2TNDVNPCKFWFUNOL7AZCALMIPZJWPA, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.642007388 (505h11m56.513090434s) [noise=true , address=145.79.227.43:4175 , piece_id=T63E5RK7U3N7GHEAOQAFHCV5UPWPC2RNGBCTKD7XB5ODY44P5TMQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.642606971 (505h11m56.513690016s) [noise=true , address=145.79.227.8:4340 , piece_id=2XM5IYMXC3T4RZGP5STFD7X44GA5LLJ2YBLIZKZCMPV2EGMITP5Q, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.643110820 (505h11m56.514193865s) [noise=true , address=193.149.164.143:4703 , piece_id=EG5EHWBVYP4P3CCCDTO2IWZDNTQCMI7JTSRECYNMQVDBXQKN2WCQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.643616243 (505h11m56.514699288s) [noise=true , address=145.79.227.5:4255 , piece_id=2BXGXSZ76NKMPKLDE5NRJ622SFOLZT3FYAKOJ7EQSGKO2JBYDTJA, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.644160849 (505h11m56.515243894s) [noise=true , address=77.90.31.14:4671 , piece_id=6TIM4CCDRYOS2RIB4MJXG55NHN5AK5IM2WCJBRFZQTFGFBR7RUXQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:57.909019390 (505h11m57.780102425s) [upload=443308]: Next returned nil segment Dec 26 16:40:57.909046512 (505h11m57.780129547s) [upload=443308]: Waiting for error group managing segments... Dec 26 16:40:58.279921182 (505h11m58.151004217s) [noise=true , address=64.49.13.27:4965 , piece_id=JNYCPACXREVCJYKSXHEUAFEGGEI5QPW4VSWLJFLPSLUTT65AYLXQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.509473179 (505h11m58.380556214s) [noise=true , address=145.79.227.43:4175 , piece_id=T63E5RK7U3N7GHEAOQAFHCV5UPWPC2RNGBCTKD7XB5ODY44P5TMQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.526249174 (505h11m58.397332209s) [noise=true , address=145.79.227.17:4978 , piece_id=JDFCSH37HLE7RXUJVIC4HU2TNDVNPCKFWFUNOL7AZCALMIPZJWPA, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.534222977 (505h11m58.405306012s) [noise=true , address=145.79.227.8:4340 , piece_id=2XM5IYMXC3T4RZGP5STFD7X44GA5LLJ2YBLIZKZCMPV2EGMITP5Q, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.624481034 (505h11m58.495564069s) [noise=true , address=64.49.13.155:4924 , piece_id=5WEKSQI423JPQAZYOB27OV4RHWFVWSNH3A3XB3CLCS7T3DPRKPAQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.650754864 (505h11m58.521837909s) [noise=true , address=5.231.73.5:4029 , piece_id=HJICME4ZINBCPHVF2KUXILTQQIAYKQGIL5E4FT7WQHPX23YOEQ4A, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.673199081 (505h11m58.544282117s) [noise=true , address=77.90.31.13:4646 , piece_id=BRJT6UUDGWS5T3KWTAW4ESJXFULND4FB6VQVYOSYRECDNNJ56SJA, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.745181288 (505h11m58.616264323s) [noise=true , address=77.90.31.12:4610 , piece_id=44TLRO3WIRAF45DAXO3TPQ2GHC2U3OFN2KLU73HG563552DGWUMA, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.779227909 (505h11m58.650310954s) [noise=true , address=5.231.73.2:4027 , piece_id=Y6HUQVZS4RDZFGJCFHXHGY5GDBVNGZUNUAC5OXORLLD4GWA3DRLQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.073598071 (505h11m58.944681106s) [noise=true , address=77.90.31.17:4784 , piece_id=7CMDKQQHDP3P7M23U3ZP2GJVQC4KUCKCB4S6BBXHUNWBX6KS3DHQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.116295695 (505h11m58.98737874s) [noise=true , address=77.90.31.14:4671 , piece_id=6TIM4CCDRYOS2RIB4MJXG55NHN5AK5IM2WCJBRFZQTFGFBR7RUXQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.196854433 (505h11m59.067937478s) [noise=true , address=193.149.164.143:4703 , piece_id=EG5EHWBVYP4P3CCCDTO2IWZDNTQCMI7JTSRECYNMQVDBXQKN2WCQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.426409626 (505h11m59.297492671s) [noise=true , address=145.79.227.44:4200 , piece_id=732NRQEMNGUE6WNMUFDRB3IEOOZKGLRPDLGOID54ADB4WHQW6QCQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.535650219 (505h11m59.406733264s) [noise=true , address=145.79.227.33:4897 , piece_id=QLFZSNPKDCEFA5TQ5TCUIT3T7A4JQ3CX3AT5T4N7M64GEYZ647LA, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.535675147 (505h11m59.406758192s) [seg_pos={0 0}, upload=443308]: Segment reached optimal threshold of 14 pieces. Dec 26 16:40:59.535893853 (505h11m59.406976898s) [noise=true , address=145.79.227.5:4255 , piece_id=2BXGXSZ76NKMPKLDE5NRJ622SFOLZT3FYAKOJ7EQSGKO2JBYDTJA, seg_pos={0 0}, upload=443308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fTNKdc6AZbkfsKRFZwTwV6wPz9j9ZD9t2eNkHBJNLMNcGh9gR): protocol: write tcp 188.166.225.170:34966->145.79.227.5:4255: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.5:4255: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.5:4255: use of closed network connection Dec 26 16:40:59.535902018 (505h11m59.406985053s) [noise=true , address=145.79.227.5:4255 , piece_id=2BXGXSZ76NKMPKLDE5NRJ622SFOLZT3FYAKOJ7EQSGKO2JBYDTJA, seg_pos={0 0}, upload=443308]: Long tail cancellation due to optimal threshold Dec 26 16:40:59.536140953 (505h11m59.407224008s) [noise=true , address=145.79.227.40:4067 , piece_id=7ATX5TWB3NK4BARIMVPL4SYF2KUOCUR7JGYBCAK4W46EZASZIPSQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BPYCUuXmyUAZWEgWk2GWTnmT4Sh2pvnvHLHaFaArX3iSvkrnq): protocol: expected piece hash; context canceled; EOF Dec 26 16:40:59.536148949 (505h11m59.407231994s) [noise=true , address=145.79.227.40:4067 , piece_id=7ATX5TWB3NK4BARIMVPL4SYF2KUOCUR7JGYBCAK4W46EZASZIPSQ, seg_pos={0 0}, upload=443308]: Long tail cancellation due to optimal threshold Dec 26 16:40:59.536341756 (505h11m59.407424791s) [noise=true , address=77.90.31.21:4905 , piece_id=CARCPTBOEX4JRA3JHLGBMMSHYRZTFVVQALZ4WHBVJV7LOC5JCM5A, seg_pos={0 0}, upload=443308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK): protocol: write tcp 188.166.225.170:33344->77.90.31.21:4905: use of closed network connection; write tcp 188.166.225.170:33344->77.90.31.21:4905: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33344->77.90.31.21:4905: use of closed network connection Dec 26 16:40:59.536349891 (505h11m59.407432936s) [noise=true , address=77.90.31.21:4905 , piece_id=CARCPTBOEX4JRA3JHLGBMMSHYRZTFVVQALZ4WHBVJV7LOC5JCM5A, seg_pos={0 0}, upload=443308]: Long tail cancellation due to optimal threshold Dec 26 16:40:59.536580220 (505h11m59.407663265s) [noise=true , address=77.90.31.18:4812 , piece_id=4YFXYQYUW3FBVXMVJX47PLUU6QBJWXKSWULXTO35CHVZXPLVPQEQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn): protocol: expected piece hash; context canceled; EOF Dec 26 16:40:59.536592623 (505h11m59.407675658s) [noise=true , address=77.90.31.18:4812 , piece_id=4YFXYQYUW3FBVXMVJX47PLUU6QBJWXKSWULXTO35CHVZXPLVPQEQ, seg_pos={0 0}, upload=443308]: Long tail cancellation due to optimal threshold Dec 26 16:40:59.536637570 (505h11m59.407720605s) [seg_pos={0 0}, upload=443308]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:40:59.536765813 (505h11m59.407848858s) [upload=443308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:40:59.536773297 (505h11m59.407856342s) [upload=443308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:40:59.833034932 (505h11m59.704117977s) [upload=443308]: Done uploading segments... Dec 26 16:40:59.833058236 (505h11m59.704141271s) [upload=443308]: Upload finished. err: <nil> Duration: 1m0.002057425s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 11.695836501s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:41:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 11.695866537s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [..X.............XXX...X...XX.....XX..........X......XXX...X.] duration: 3.600311219s trace: https://jaeger.dev.storj.io/trace/69cd2adc1b6c0e6d FAIL (2025-12-26 16:41:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.600361404s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.....X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X..X...] duration: 6.816400365s trace: https://jaeger.dev.storj.io/trace/3080d1fa188efb2e FAIL (2025-12-26 16:41:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: DIKBZ34FZCFLGFBHJJDX5BBZYNL2FTHNG3V6TSDCBPN5TXHVZMRQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: XINQ4KY3TRQ675HEJRVUQXMHMCP4EJWKDMIIKXDOXSMQUQXAPJYA) context deadline exceeded ======LOGS====== Dec 26 16:37:29.297178880 (505h8m29.168261925s) [upload=443263]: Starting upload Dec 26 16:37:29.297263381 (505h8m29.168346426s) [upload=443263]: Done starting upload Dec 26 16:37:29.297386335 (505h8m29.16846937s) [upload=443263]: Uploading segments... Dec 26 16:37:29.297473902 (505h8m29.168556947s) [upload=443263]: Got next segment. Inline: false Dec 26 16:37:29.297515431 (505h8m29.168598467s) [upload=443263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:37:29.297521964 (505h8m29.168605009s) [upload=443263]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:37:29.544301414 (505h8m29.415384449s) [seg_pos={0 0}, upload=443263]: Begin upload segment... Dec 26 16:37:29.544359325 (505h8m29.41544236s) [seg_pos={0 0}, upload=443263]: Done begin upload segment. Dec 26 16:37:29.544388360 (505h8m29.415471395s) [noise=true , address=45.91.116.14:4191 , piece_id=CRS5MFAUUM4QAMCSA27GIUNK2DYGCPZUQ45OI75BSALOPW32UCKQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.544452883 (505h8m29.415535918s) [noise=true , address=145.79.227.51:4409 , piece_id=TP5KPGXQT46CH5O2QUWAPAQQFHQTXVVE465C7NMTWDHYO2Q2RA3A, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.544975058 (505h8m29.416058093s) [noise=true , address=145.79.227.37:4566 , piece_id=MDTJ7G6ZAGFR42S5API3IBD2FGOZXI7NLE3I2OIVCF3P4T4NQ3LA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.545259659 (505h8m29.416342704s) [noise=true , address=145.79.227.13:4804 , piece_id=UOL2F3ULLGMJYUM3DDMLS37CYGAAVYAAXEJXULC4ZM7QIMLKOCSA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.545535645 (505h8m29.41661868s) [noise=true , address=45.91.116.4:4940 , piece_id=RZ4UPCTN4INRLKKB2Z7NAPMFGJROLNPBCGR463MO7OOQ6QDKVGNQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.545944193 (505h8m29.417027228s) [noise=true , address=45.91.116.5:4300 , piece_id=VN3C7NU2X2Q4CEOABPEDYTGLSACEAIUKOAPPAY7576JVUH6JI6DA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.546222914 (505h8m29.417305959s) [noise=true , address=77.90.31.10:4520 , piece_id=XINQ4KY3TRQ675HEJRVUQXMHMCP4EJWKDMIIKXDOXSMQUQXAPJYA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.546501424 (505h8m29.417584459s) [noise=true , address=145.79.227.22:4509 , piece_id=JTVULVGIRWP6YWVB2LRWKTDPW3LTVAWE4KN3LDPOPCAETV3UQMLA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.546824910 (505h8m29.417907955s) [noise=true , address=77.90.31.19:4841 , piece_id=AFOELLKOI6X6SCWVF23QRDYWFSUP2AM5DCKNL5NHDRUUKODIRUHA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.548078477 (505h8m29.419161512s) [noise=true , address=145.79.227.43:4157 , piece_id=A5H2QYKRN3HHWQUMWWUKGJLILKOQVLGQ5MM4CQRH5ECTOBVROTDA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.548184078 (505h8m29.419267113s) [noise=true , address=145.79.227.6:4257 , piece_id=XLHDMVMFGODI5L7LBBMD7MZSREBO7HLMR4LIL4IPMVK2HMTBJWUQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.549250759 (505h8m29.420333794s) [noise=true , address=145.79.227.39:4014 , piece_id=ONUVY2Q26V64DMVZ36UFD7WESAKWRPCKXO2LCB22EVOA3QOTTV4Q, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.549380546 (505h8m29.420463591s) [noise=true , address=45.91.116.7:4009 , piece_id=GAMH75PUG45WLH7K2FYMBYPVOTLDDBCJSP6Z7MB4NXYLEZPRP2VQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.550718123 (505h8m29.421801168s) [noise=true , address=145.79.227.45:4229 , piece_id=ZQYVHRSFNR5LSRQDKHN7JFZWFPZCLKK3D4DP7F3FRS2NSUXJ2THA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.551967743 (505h8m29.423050788s) [noise=true , address=145.79.227.46:4601 , piece_id=MWAIQ23QBDSQ5H7SE7OTI55T5C3KO7OJG6AGGKNT56KIOX2PDWPA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.553114406 (505h8m29.424197452s) [noise=true , address=77.90.31.14:4663 , piece_id=DIKBZ34FZCFLGFBHJJDX5BBZYNL2FTHNG3V6TSDCBPN5TXHVZMRQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:31.267104923 (505h8m31.138187968s) [upload=443263]: Next returned nil segment Dec 26 16:37:31.267129039 (505h8m31.138212084s) [upload=443263]: Waiting for error group managing segments... Dec 26 16:37:31.373605980 (505h8m31.244689025s) [noise=true , address=77.90.31.19:4841 , piece_id=AFOELLKOI6X6SCWVF23QRDYWFSUP2AM5DCKNL5NHDRUUKODIRUHA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 26 16:37:31.547383522 (505h8m31.418466567s) [noise=true , address=145.79.227.13:4804 , piece_id=UOL2F3ULLGMJYUM3DDMLS37CYGAAVYAAXEJXULC4ZM7QIMLKOCSA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.556110039 (505h8m31.427193074s) [noise=true , address=145.79.227.6:4257 , piece_id=XLHDMVMFGODI5L7LBBMD7MZSREBO7HLMR4LIL4IPMVK2HMTBJWUQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.556833557 (505h8m31.427916602s) [noise=true , address=145.79.227.39:4014 , piece_id=ONUVY2Q26V64DMVZ36UFD7WESAKWRPCKXO2LCB22EVOA3QOTTV4Q, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.558425728 (505h8m31.429508773s) [noise=true , address=145.79.227.46:4601 , piece_id=MWAIQ23QBDSQ5H7SE7OTI55T5C3KO7OJG6AGGKNT56KIOX2PDWPA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.559746073 (505h8m31.430829108s) [noise=true , address=145.79.227.22:4509 , piece_id=JTVULVGIRWP6YWVB2LRWKTDPW3LTVAWE4KN3LDPOPCAETV3UQMLA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.560533582 (505h8m31.431616628s) [noise=true , address=145.79.227.45:4229 , piece_id=ZQYVHRSFNR5LSRQDKHN7JFZWFPZCLKK3D4DP7F3FRS2NSUXJ2THA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.560664382 (505h8m31.431747417s) [noise=true , address=145.79.227.51:4409 , piece_id=TP5KPGXQT46CH5O2QUWAPAQQFHQTXVVE465C7NMTWDHYO2Q2RA3A, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.571627928 (505h8m31.442710973s) [noise=true , address=145.79.227.37:4566 , piece_id=MDTJ7G6ZAGFR42S5API3IBD2FGOZXI7NLE3I2OIVCF3P4T4NQ3LA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.843361235 (505h8m31.714444281s) [noise=true , address=77.90.31.14:4663 , piece_id=DIKBZ34FZCFLGFBHJJDX5BBZYNL2FTHNG3V6TSDCBPN5TXHVZMRQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.864291030 (505h8m31.735374065s) [noise=true , address=77.90.31.10:4520 , piece_id=XINQ4KY3TRQ675HEJRVUQXMHMCP4EJWKDMIIKXDOXSMQUQXAPJYA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.878425975 (505h8m31.749509011s) [noise=true , address=45.91.116.7:4009 , piece_id=GAMH75PUG45WLH7K2FYMBYPVOTLDDBCJSP6Z7MB4NXYLEZPRP2VQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.908628876 (505h8m31.779711911s) [noise=true , address=45.91.116.4:4940 , piece_id=RZ4UPCTN4INRLKKB2Z7NAPMFGJROLNPBCGR463MO7OOQ6QDKVGNQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:32.819947576 (505h8m32.691030611s) [noise=true , address=45.91.116.5:4300 , piece_id=VN3C7NU2X2Q4CEOABPEDYTGLSACEAIUKOAPPAY7576JVUH6JI6DA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:32.819983785 (505h8m32.69106683s) [seg_pos={0 0}, upload=443263]: Segment reached optimal threshold of 13 pieces. Dec 26 16:37:32.820176242 (505h8m32.691259277s) [noise=true , address=145.79.227.43:4157 , piece_id=A5H2QYKRN3HHWQUMWWUKGJLILKOQVLGQ5MM4CQRH5ECTOBVROTDA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 26 16:37:32.820182814 (505h8m32.691265849s) [noise=true , address=145.79.227.43:4157 , piece_id=A5H2QYKRN3HHWQUMWWUKGJLILKOQVLGQ5MM4CQRH5ECTOBVROTDA, seg_pos={0 0}, upload=443263]: Long tail cancellation due to optimal threshold Dec 26 16:37:32.820296280 (505h8m32.691379315s) [noise=true , address=45.91.116.14:4191 , piece_id=CRS5MFAUUM4QAMCSA27GIUNK2DYGCPZUQ45OI75BSALOPW32UCKQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 16:37:32.820301560 (505h8m32.691384615s) [noise=true , address=45.91.116.14:4191 , piece_id=CRS5MFAUUM4QAMCSA27GIUNK2DYGCPZUQ45OI75BSALOPW32UCKQ, seg_pos={0 0}, upload=443263]: Long tail cancellation due to optimal threshold Dec 26 16:37:32.820339582 (505h8m32.691422618s) [seg_pos={0 0}, upload=443263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:37:32.820416289 (505h8m32.691499324s) [upload=443263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:37:32.820420507 (505h8m32.691503542s) [upload=443263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:37:33.108070794 (505h8m32.979153829s) [upload=443263]: Done uploading segments... Dec 26 16:37:33.108097985 (505h8m32.97918102s) [upload=443263]: Upload finished. err: <nil> Duration: 6.816452634s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.......XX..........X.........X..X...............X.......] duration: 47.588251296s trace: https://jaeger.dev.storj.io/trace/4bc9becdddcc2037 FAIL (2025-12-26 16:40:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443223]: Uploading piece... Dec 26 16:34:46.586854691 (505h5m46.457937736s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.587812465 (505h5m46.4588955s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.588277791 (505h5m46.459360826s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590149876 (505h5m46.461232911s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590674354 (505h5m46.461757399s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591254680 (505h5m46.462337715s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591632860 (505h5m46.462715895s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591778888 (505h5m46.462861923s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592107854 (505h5m46.463190889s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592306482 (505h5m46.463389517s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592535257 (505h5m46.463618302s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592867591 (505h5m46.463950636s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592905783 (505h5m46.463988828s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593325633 (505h5m46.464408678s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593417989 (505h5m46.464501034s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593721267 (505h5m46.464804302s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.594046005 (505h5m46.46512905s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:47.885134462 (505h5m47.756217497s) [upload=443223]: Next returned nil segment Dec 26 16:34:47.885151225 (505h5m47.75623426s) [upload=443223]: Waiting for error group managing segments... Dec 26 16:34:48.239296449 (505h5m48.110379494s) [noise=true , address=64.49.13.140:4505 , piece_id=FXVVLCPQXGJLYWDGHDOQECN3S4ZZXTEEF7HNVLQW33IUEIPYW6OA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465120747 (505h5m48.336203792s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465174059 (505h5m48.336257094s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.468068951 (505h5m48.339151986s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.481407861 (505h5m48.352490896s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.482107242 (505h5m48.353190277s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483127506 (505h5m48.354210551s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483731115 (505h5m48.35481415s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.639322688 (505h5m48.510405733s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.760495594 (505h5m48.631578639s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.763286658 (505h5m48.634369693s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.771028379 (505h5m48.642111414s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.341293323 (505h5m49.212376368s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446731015 (505h5m49.31781406s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446757516 (505h5m49.317840561s) [seg_pos={0 0}, upload=443223]: Segment reached optimal threshold of 14 pieces. Dec 26 16:34:49.446947537 (505h5m49.318030572s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.446954180 (505h5m49.318037216s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447148209 (505h5m49.318231244s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447154361 (505h5m49.318237396s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447201702 (505h5m49.318284767s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447232551 (505h5m49.318315606s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447285240 (505h5m49.318368275s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447291793 (505h5m49.318374838s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447309938 (505h5m49.318392973s) [seg_pos={0 0}, upload=443223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:34:49.447375853 (505h5m49.318458898s) [upload=443223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:34:49.447380402 (505h5m49.318463437s) [upload=443223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:34:49.756168298 (505h5m49.627251333s) [upload=443223]: Done uploading segments... Dec 26 16:34:49.756192103 (505h5m49.627275149s) [upload=443223]: Upload finished. err: <nil> Duration: 47.588269361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 25.162209676s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:40:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 25.162235926s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [...X.............XXX...X...XX.....XX..........X......XXX...X] duration: 5.001966587s trace: https://jaeger.dev.storj.io/trace/69cd2adc1b6c0e6d FAIL (2025-12-26 16:40:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002001574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [X...X.......XX..........X.........X..X...............X......] duration: 53.916171139s trace: https://jaeger.dev.storj.io/trace/4bc9becdddcc2037 FAIL (2025-12-26 16:40:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443223]: Uploading piece... Dec 26 16:34:46.586854691 (505h5m46.457937736s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.587812465 (505h5m46.4588955s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.588277791 (505h5m46.459360826s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590149876 (505h5m46.461232911s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590674354 (505h5m46.461757399s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591254680 (505h5m46.462337715s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591632860 (505h5m46.462715895s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591778888 (505h5m46.462861923s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592107854 (505h5m46.463190889s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592306482 (505h5m46.463389517s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592535257 (505h5m46.463618302s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592867591 (505h5m46.463950636s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592905783 (505h5m46.463988828s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593325633 (505h5m46.464408678s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593417989 (505h5m46.464501034s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593721267 (505h5m46.464804302s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.594046005 (505h5m46.46512905s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:47.885134462 (505h5m47.756217497s) [upload=443223]: Next returned nil segment Dec 26 16:34:47.885151225 (505h5m47.75623426s) [upload=443223]: Waiting for error group managing segments... Dec 26 16:34:48.239296449 (505h5m48.110379494s) [noise=true , address=64.49.13.140:4505 , piece_id=FXVVLCPQXGJLYWDGHDOQECN3S4ZZXTEEF7HNVLQW33IUEIPYW6OA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465120747 (505h5m48.336203792s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465174059 (505h5m48.336257094s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.468068951 (505h5m48.339151986s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.481407861 (505h5m48.352490896s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.482107242 (505h5m48.353190277s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483127506 (505h5m48.354210551s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483731115 (505h5m48.35481415s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.639322688 (505h5m48.510405733s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.760495594 (505h5m48.631578639s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.763286658 (505h5m48.634369693s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.771028379 (505h5m48.642111414s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.341293323 (505h5m49.212376368s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446731015 (505h5m49.31781406s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446757516 (505h5m49.317840561s) [seg_pos={0 0}, upload=443223]: Segment reached optimal threshold of 14 pieces. Dec 26 16:34:49.446947537 (505h5m49.318030572s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.446954180 (505h5m49.318037216s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447148209 (505h5m49.318231244s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447154361 (505h5m49.318237396s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447201702 (505h5m49.318284767s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447232551 (505h5m49.318315606s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447285240 (505h5m49.318368275s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447291793 (505h5m49.318374838s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447309938 (505h5m49.318392973s) [seg_pos={0 0}, upload=443223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:34:49.447375853 (505h5m49.318458898s) [upload=443223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:34:49.447380402 (505h5m49.318463437s) [upload=443223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:34:49.756168298 (505h5m49.627251333s) [upload=443223]: Done uploading segments... Dec 26 16:34:49.756192103 (505h5m49.627275149s) [upload=443223]: Upload finished. err: <nil> Duration: 53.91620248s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [......X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X..X..] duration: 7.18217336s trace: https://jaeger.dev.storj.io/trace/3080d1fa188efb2e FAIL (2025-12-26 16:39:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: DIKBZ34FZCFLGFBHJJDX5BBZYNL2FTHNG3V6TSDCBPN5TXHVZMRQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: XINQ4KY3TRQ675HEJRVUQXMHMCP4EJWKDMIIKXDOXSMQUQXAPJYA) context deadline exceeded ======LOGS====== Dec 26 16:37:29.297178880 (505h8m29.168261925s) [upload=443263]: Starting upload Dec 26 16:37:29.297263381 (505h8m29.168346426s) [upload=443263]: Done starting upload Dec 26 16:37:29.297386335 (505h8m29.16846937s) [upload=443263]: Uploading segments... Dec 26 16:37:29.297473902 (505h8m29.168556947s) [upload=443263]: Got next segment. Inline: false Dec 26 16:37:29.297515431 (505h8m29.168598467s) [upload=443263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:37:29.297521964 (505h8m29.168605009s) [upload=443263]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:37:29.544301414 (505h8m29.415384449s) [seg_pos={0 0}, upload=443263]: Begin upload segment... Dec 26 16:37:29.544359325 (505h8m29.41544236s) [seg_pos={0 0}, upload=443263]: Done begin upload segment. Dec 26 16:37:29.544388360 (505h8m29.415471395s) [noise=true , address=45.91.116.14:4191 , piece_id=CRS5MFAUUM4QAMCSA27GIUNK2DYGCPZUQ45OI75BSALOPW32UCKQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.544452883 (505h8m29.415535918s) [noise=true , address=145.79.227.51:4409 , piece_id=TP5KPGXQT46CH5O2QUWAPAQQFHQTXVVE465C7NMTWDHYO2Q2RA3A, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.544975058 (505h8m29.416058093s) [noise=true , address=145.79.227.37:4566 , piece_id=MDTJ7G6ZAGFR42S5API3IBD2FGOZXI7NLE3I2OIVCF3P4T4NQ3LA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.545259659 (505h8m29.416342704s) [noise=true , address=145.79.227.13:4804 , piece_id=UOL2F3ULLGMJYUM3DDMLS37CYGAAVYAAXEJXULC4ZM7QIMLKOCSA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.545535645 (505h8m29.41661868s) [noise=true , address=45.91.116.4:4940 , piece_id=RZ4UPCTN4INRLKKB2Z7NAPMFGJROLNPBCGR463MO7OOQ6QDKVGNQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.545944193 (505h8m29.417027228s) [noise=true , address=45.91.116.5:4300 , piece_id=VN3C7NU2X2Q4CEOABPEDYTGLSACEAIUKOAPPAY7576JVUH6JI6DA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.546222914 (505h8m29.417305959s) [noise=true , address=77.90.31.10:4520 , piece_id=XINQ4KY3TRQ675HEJRVUQXMHMCP4EJWKDMIIKXDOXSMQUQXAPJYA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.546501424 (505h8m29.417584459s) [noise=true , address=145.79.227.22:4509 , piece_id=JTVULVGIRWP6YWVB2LRWKTDPW3LTVAWE4KN3LDPOPCAETV3UQMLA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.546824910 (505h8m29.417907955s) [noise=true , address=77.90.31.19:4841 , piece_id=AFOELLKOI6X6SCWVF23QRDYWFSUP2AM5DCKNL5NHDRUUKODIRUHA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.548078477 (505h8m29.419161512s) [noise=true , address=145.79.227.43:4157 , piece_id=A5H2QYKRN3HHWQUMWWUKGJLILKOQVLGQ5MM4CQRH5ECTOBVROTDA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.548184078 (505h8m29.419267113s) [noise=true , address=145.79.227.6:4257 , piece_id=XLHDMVMFGODI5L7LBBMD7MZSREBO7HLMR4LIL4IPMVK2HMTBJWUQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.549250759 (505h8m29.420333794s) [noise=true , address=145.79.227.39:4014 , piece_id=ONUVY2Q26V64DMVZ36UFD7WESAKWRPCKXO2LCB22EVOA3QOTTV4Q, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.549380546 (505h8m29.420463591s) [noise=true , address=45.91.116.7:4009 , piece_id=GAMH75PUG45WLH7K2FYMBYPVOTLDDBCJSP6Z7MB4NXYLEZPRP2VQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.550718123 (505h8m29.421801168s) [noise=true , address=145.79.227.45:4229 , piece_id=ZQYVHRSFNR5LSRQDKHN7JFZWFPZCLKK3D4DP7F3FRS2NSUXJ2THA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.551967743 (505h8m29.423050788s) [noise=true , address=145.79.227.46:4601 , piece_id=MWAIQ23QBDSQ5H7SE7OTI55T5C3KO7OJG6AGGKNT56KIOX2PDWPA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.553114406 (505h8m29.424197452s) [noise=true , address=77.90.31.14:4663 , piece_id=DIKBZ34FZCFLGFBHJJDX5BBZYNL2FTHNG3V6TSDCBPN5TXHVZMRQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:31.267104923 (505h8m31.138187968s) [upload=443263]: Next returned nil segment Dec 26 16:37:31.267129039 (505h8m31.138212084s) [upload=443263]: Waiting for error group managing segments... Dec 26 16:37:31.373605980 (505h8m31.244689025s) [noise=true , address=77.90.31.19:4841 , piece_id=AFOELLKOI6X6SCWVF23QRDYWFSUP2AM5DCKNL5NHDRUUKODIRUHA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 26 16:37:31.547383522 (505h8m31.418466567s) [noise=true , address=145.79.227.13:4804 , piece_id=UOL2F3ULLGMJYUM3DDMLS37CYGAAVYAAXEJXULC4ZM7QIMLKOCSA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.556110039 (505h8m31.427193074s) [noise=true , address=145.79.227.6:4257 , piece_id=XLHDMVMFGODI5L7LBBMD7MZSREBO7HLMR4LIL4IPMVK2HMTBJWUQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.556833557 (505h8m31.427916602s) [noise=true , address=145.79.227.39:4014 , piece_id=ONUVY2Q26V64DMVZ36UFD7WESAKWRPCKXO2LCB22EVOA3QOTTV4Q, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.558425728 (505h8m31.429508773s) [noise=true , address=145.79.227.46:4601 , piece_id=MWAIQ23QBDSQ5H7SE7OTI55T5C3KO7OJG6AGGKNT56KIOX2PDWPA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.559746073 (505h8m31.430829108s) [noise=true , address=145.79.227.22:4509 , piece_id=JTVULVGIRWP6YWVB2LRWKTDPW3LTVAWE4KN3LDPOPCAETV3UQMLA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.560533582 (505h8m31.431616628s) [noise=true , address=145.79.227.45:4229 , piece_id=ZQYVHRSFNR5LSRQDKHN7JFZWFPZCLKK3D4DP7F3FRS2NSUXJ2THA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.560664382 (505h8m31.431747417s) [noise=true , address=145.79.227.51:4409 , piece_id=TP5KPGXQT46CH5O2QUWAPAQQFHQTXVVE465C7NMTWDHYO2Q2RA3A, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.571627928 (505h8m31.442710973s) [noise=true , address=145.79.227.37:4566 , piece_id=MDTJ7G6ZAGFR42S5API3IBD2FGOZXI7NLE3I2OIVCF3P4T4NQ3LA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.843361235 (505h8m31.714444281s) [noise=true , address=77.90.31.14:4663 , piece_id=DIKBZ34FZCFLGFBHJJDX5BBZYNL2FTHNG3V6TSDCBPN5TXHVZMRQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.864291030 (505h8m31.735374065s) [noise=true , address=77.90.31.10:4520 , piece_id=XINQ4KY3TRQ675HEJRVUQXMHMCP4EJWKDMIIKXDOXSMQUQXAPJYA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.878425975 (505h8m31.749509011s) [noise=true , address=45.91.116.7:4009 , piece_id=GAMH75PUG45WLH7K2FYMBYPVOTLDDBCJSP6Z7MB4NXYLEZPRP2VQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.908628876 (505h8m31.779711911s) [noise=true , address=45.91.116.4:4940 , piece_id=RZ4UPCTN4INRLKKB2Z7NAPMFGJROLNPBCGR463MO7OOQ6QDKVGNQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:32.819947576 (505h8m32.691030611s) [noise=true , address=45.91.116.5:4300 , piece_id=VN3C7NU2X2Q4CEOABPEDYTGLSACEAIUKOAPPAY7576JVUH6JI6DA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:32.819983785 (505h8m32.69106683s) [seg_pos={0 0}, upload=443263]: Segment reached optimal threshold of 13 pieces. Dec 26 16:37:32.820176242 (505h8m32.691259277s) [noise=true , address=145.79.227.43:4157 , piece_id=A5H2QYKRN3HHWQUMWWUKGJLILKOQVLGQ5MM4CQRH5ECTOBVROTDA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 26 16:37:32.820182814 (505h8m32.691265849s) [noise=true , address=145.79.227.43:4157 , piece_id=A5H2QYKRN3HHWQUMWWUKGJLILKOQVLGQ5MM4CQRH5ECTOBVROTDA, seg_pos={0 0}, upload=443263]: Long tail cancellation due to optimal threshold Dec 26 16:37:32.820296280 (505h8m32.691379315s) [noise=true , address=45.91.116.14:4191 , piece_id=CRS5MFAUUM4QAMCSA27GIUNK2DYGCPZUQ45OI75BSALOPW32UCKQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 16:37:32.820301560 (505h8m32.691384615s) [noise=true , address=45.91.116.14:4191 , piece_id=CRS5MFAUUM4QAMCSA27GIUNK2DYGCPZUQ45OI75BSALOPW32UCKQ, seg_pos={0 0}, upload=443263]: Long tail cancellation due to optimal threshold Dec 26 16:37:32.820339582 (505h8m32.691422618s) [seg_pos={0 0}, upload=443263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:37:32.820416289 (505h8m32.691499324s) [upload=443263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:37:32.820420507 (505h8m32.691503542s) [upload=443263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:37:33.108070794 (505h8m32.979153829s) [upload=443263]: Done uploading segments... Dec 26 16:37:33.108097985 (505h8m32.97918102s) [upload=443263]: Upload finished. err: <nil> Duration: 7.182268662s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.096346286s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:39:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 6.096375453s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.10697743s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:39:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 6.107011554s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [....X.............XXX...X...XX.....XX..........X......XXX...] duration: 3.507456259s trace: https://jaeger.dev.storj.io/trace/5b9c7e160a55f0bd FAIL (2025-12-26 16:39:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.507584072s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X...X.......XX..........X.........X..X...............X.....] duration: 6.447269177s trace: https://jaeger.dev.storj.io/trace/4bc9becdddcc2037 FAIL (2025-12-26 16:38:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443223]: Uploading piece... Dec 26 16:34:46.586854691 (505h5m46.457937736s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.587812465 (505h5m46.4588955s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.588277791 (505h5m46.459360826s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590149876 (505h5m46.461232911s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590674354 (505h5m46.461757399s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591254680 (505h5m46.462337715s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591632860 (505h5m46.462715895s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591778888 (505h5m46.462861923s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592107854 (505h5m46.463190889s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592306482 (505h5m46.463389517s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592535257 (505h5m46.463618302s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592867591 (505h5m46.463950636s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592905783 (505h5m46.463988828s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593325633 (505h5m46.464408678s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593417989 (505h5m46.464501034s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593721267 (505h5m46.464804302s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.594046005 (505h5m46.46512905s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:47.885134462 (505h5m47.756217497s) [upload=443223]: Next returned nil segment Dec 26 16:34:47.885151225 (505h5m47.75623426s) [upload=443223]: Waiting for error group managing segments... Dec 26 16:34:48.239296449 (505h5m48.110379494s) [noise=true , address=64.49.13.140:4505 , piece_id=FXVVLCPQXGJLYWDGHDOQECN3S4ZZXTEEF7HNVLQW33IUEIPYW6OA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465120747 (505h5m48.336203792s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465174059 (505h5m48.336257094s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.468068951 (505h5m48.339151986s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.481407861 (505h5m48.352490896s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.482107242 (505h5m48.353190277s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483127506 (505h5m48.354210551s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483731115 (505h5m48.35481415s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.639322688 (505h5m48.510405733s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.760495594 (505h5m48.631578639s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.763286658 (505h5m48.634369693s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.771028379 (505h5m48.642111414s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.341293323 (505h5m49.212376368s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446731015 (505h5m49.31781406s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446757516 (505h5m49.317840561s) [seg_pos={0 0}, upload=443223]: Segment reached optimal threshold of 14 pieces. Dec 26 16:34:49.446947537 (505h5m49.318030572s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.446954180 (505h5m49.318037216s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447148209 (505h5m49.318231244s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447154361 (505h5m49.318237396s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447201702 (505h5m49.318284767s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447232551 (505h5m49.318315606s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447285240 (505h5m49.318368275s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447291793 (505h5m49.318374838s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447309938 (505h5m49.318392973s) [seg_pos={0 0}, upload=443223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:34:49.447375853 (505h5m49.318458898s) [upload=443223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:34:49.447380402 (505h5m49.318463437s) [upload=443223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:34:49.756168298 (505h5m49.627251333s) [upload=443223]: Done uploading segments... Dec 26 16:34:49.756192103 (505h5m49.627275149s) [upload=443223]: Upload finished. err: <nil> Duration: 6.447304422s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.......X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X..X.] duration: 6.135197657s trace: https://jaeger.dev.storj.io/trace/3080d1fa188efb2e FAIL (2025-12-26 16:38:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: DIKBZ34FZCFLGFBHJJDX5BBZYNL2FTHNG3V6TSDCBPN5TXHVZMRQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: XINQ4KY3TRQ675HEJRVUQXMHMCP4EJWKDMIIKXDOXSMQUQXAPJYA) context deadline exceeded ======LOGS====== Dec 26 16:37:29.297178880 (505h8m29.168261925s) [upload=443263]: Starting upload Dec 26 16:37:29.297263381 (505h8m29.168346426s) [upload=443263]: Done starting upload Dec 26 16:37:29.297386335 (505h8m29.16846937s) [upload=443263]: Uploading segments... Dec 26 16:37:29.297473902 (505h8m29.168556947s) [upload=443263]: Got next segment. Inline: false Dec 26 16:37:29.297515431 (505h8m29.168598467s) [upload=443263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:37:29.297521964 (505h8m29.168605009s) [upload=443263]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:37:29.544301414 (505h8m29.415384449s) [seg_pos={0 0}, upload=443263]: Begin upload segment... Dec 26 16:37:29.544359325 (505h8m29.41544236s) [seg_pos={0 0}, upload=443263]: Done begin upload segment. Dec 26 16:37:29.544388360 (505h8m29.415471395s) [noise=true , address=45.91.116.14:4191 , piece_id=CRS5MFAUUM4QAMCSA27GIUNK2DYGCPZUQ45OI75BSALOPW32UCKQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.544452883 (505h8m29.415535918s) [noise=true , address=145.79.227.51:4409 , piece_id=TP5KPGXQT46CH5O2QUWAPAQQFHQTXVVE465C7NMTWDHYO2Q2RA3A, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.544975058 (505h8m29.416058093s) [noise=true , address=145.79.227.37:4566 , piece_id=MDTJ7G6ZAGFR42S5API3IBD2FGOZXI7NLE3I2OIVCF3P4T4NQ3LA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.545259659 (505h8m29.416342704s) [noise=true , address=145.79.227.13:4804 , piece_id=UOL2F3ULLGMJYUM3DDMLS37CYGAAVYAAXEJXULC4ZM7QIMLKOCSA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.545535645 (505h8m29.41661868s) [noise=true , address=45.91.116.4:4940 , piece_id=RZ4UPCTN4INRLKKB2Z7NAPMFGJROLNPBCGR463MO7OOQ6QDKVGNQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.545944193 (505h8m29.417027228s) [noise=true , address=45.91.116.5:4300 , piece_id=VN3C7NU2X2Q4CEOABPEDYTGLSACEAIUKOAPPAY7576JVUH6JI6DA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.546222914 (505h8m29.417305959s) [noise=true , address=77.90.31.10:4520 , piece_id=XINQ4KY3TRQ675HEJRVUQXMHMCP4EJWKDMIIKXDOXSMQUQXAPJYA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.546501424 (505h8m29.417584459s) [noise=true , address=145.79.227.22:4509 , piece_id=JTVULVGIRWP6YWVB2LRWKTDPW3LTVAWE4KN3LDPOPCAETV3UQMLA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.546824910 (505h8m29.417907955s) [noise=true , address=77.90.31.19:4841 , piece_id=AFOELLKOI6X6SCWVF23QRDYWFSUP2AM5DCKNL5NHDRUUKODIRUHA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.548078477 (505h8m29.419161512s) [noise=true , address=145.79.227.43:4157 , piece_id=A5H2QYKRN3HHWQUMWWUKGJLILKOQVLGQ5MM4CQRH5ECTOBVROTDA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.548184078 (505h8m29.419267113s) [noise=true , address=145.79.227.6:4257 , piece_id=XLHDMVMFGODI5L7LBBMD7MZSREBO7HLMR4LIL4IPMVK2HMTBJWUQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.549250759 (505h8m29.420333794s) [noise=true , address=145.79.227.39:4014 , piece_id=ONUVY2Q26V64DMVZ36UFD7WESAKWRPCKXO2LCB22EVOA3QOTTV4Q, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.549380546 (505h8m29.420463591s) [noise=true , address=45.91.116.7:4009 , piece_id=GAMH75PUG45WLH7K2FYMBYPVOTLDDBCJSP6Z7MB4NXYLEZPRP2VQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.550718123 (505h8m29.421801168s) [noise=true , address=145.79.227.45:4229 , piece_id=ZQYVHRSFNR5LSRQDKHN7JFZWFPZCLKK3D4DP7F3FRS2NSUXJ2THA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.551967743 (505h8m29.423050788s) [noise=true , address=145.79.227.46:4601 , piece_id=MWAIQ23QBDSQ5H7SE7OTI55T5C3KO7OJG6AGGKNT56KIOX2PDWPA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.553114406 (505h8m29.424197452s) [noise=true , address=77.90.31.14:4663 , piece_id=DIKBZ34FZCFLGFBHJJDX5BBZYNL2FTHNG3V6TSDCBPN5TXHVZMRQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:31.267104923 (505h8m31.138187968s) [upload=443263]: Next returned nil segment Dec 26 16:37:31.267129039 (505h8m31.138212084s) [upload=443263]: Waiting for error group managing segments... Dec 26 16:37:31.373605980 (505h8m31.244689025s) [noise=true , address=77.90.31.19:4841 , piece_id=AFOELLKOI6X6SCWVF23QRDYWFSUP2AM5DCKNL5NHDRUUKODIRUHA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 26 16:37:31.547383522 (505h8m31.418466567s) [noise=true , address=145.79.227.13:4804 , piece_id=UOL2F3ULLGMJYUM3DDMLS37CYGAAVYAAXEJXULC4ZM7QIMLKOCSA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.556110039 (505h8m31.427193074s) [noise=true , address=145.79.227.6:4257 , piece_id=XLHDMVMFGODI5L7LBBMD7MZSREBO7HLMR4LIL4IPMVK2HMTBJWUQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.556833557 (505h8m31.427916602s) [noise=true , address=145.79.227.39:4014 , piece_id=ONUVY2Q26V64DMVZ36UFD7WESAKWRPCKXO2LCB22EVOA3QOTTV4Q, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.558425728 (505h8m31.429508773s) [noise=true , address=145.79.227.46:4601 , piece_id=MWAIQ23QBDSQ5H7SE7OTI55T5C3KO7OJG6AGGKNT56KIOX2PDWPA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.559746073 (505h8m31.430829108s) [noise=true , address=145.79.227.22:4509 , piece_id=JTVULVGIRWP6YWVB2LRWKTDPW3LTVAWE4KN3LDPOPCAETV3UQMLA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.560533582 (505h8m31.431616628s) [noise=true , address=145.79.227.45:4229 , piece_id=ZQYVHRSFNR5LSRQDKHN7JFZWFPZCLKK3D4DP7F3FRS2NSUXJ2THA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.560664382 (505h8m31.431747417s) [noise=true , address=145.79.227.51:4409 , piece_id=TP5KPGXQT46CH5O2QUWAPAQQFHQTXVVE465C7NMTWDHYO2Q2RA3A, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.571627928 (505h8m31.442710973s) [noise=true , address=145.79.227.37:4566 , piece_id=MDTJ7G6ZAGFR42S5API3IBD2FGOZXI7NLE3I2OIVCF3P4T4NQ3LA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.843361235 (505h8m31.714444281s) [noise=true , address=77.90.31.14:4663 , piece_id=DIKBZ34FZCFLGFBHJJDX5BBZYNL2FTHNG3V6TSDCBPN5TXHVZMRQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.864291030 (505h8m31.735374065s) [noise=true , address=77.90.31.10:4520 , piece_id=XINQ4KY3TRQ675HEJRVUQXMHMCP4EJWKDMIIKXDOXSMQUQXAPJYA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.878425975 (505h8m31.749509011s) [noise=true , address=45.91.116.7:4009 , piece_id=GAMH75PUG45WLH7K2FYMBYPVOTLDDBCJSP6Z7MB4NXYLEZPRP2VQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.908628876 (505h8m31.779711911s) [noise=true , address=45.91.116.4:4940 , piece_id=RZ4UPCTN4INRLKKB2Z7NAPMFGJROLNPBCGR463MO7OOQ6QDKVGNQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:32.819947576 (505h8m32.691030611s) [noise=true , address=45.91.116.5:4300 , piece_id=VN3C7NU2X2Q4CEOABPEDYTGLSACEAIUKOAPPAY7576JVUH6JI6DA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:32.819983785 (505h8m32.69106683s) [seg_pos={0 0}, upload=443263]: Segment reached optimal threshold of 13 pieces. Dec 26 16:37:32.820176242 (505h8m32.691259277s) [noise=true , address=145.79.227.43:4157 , piece_id=A5H2QYKRN3HHWQUMWWUKGJLILKOQVLGQ5MM4CQRH5ECTOBVROTDA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 26 16:37:32.820182814 (505h8m32.691265849s) [noise=true , address=145.79.227.43:4157 , piece_id=A5H2QYKRN3HHWQUMWWUKGJLILKOQVLGQ5MM4CQRH5ECTOBVROTDA, seg_pos={0 0}, upload=443263]: Long tail cancellation due to optimal threshold Dec 26 16:37:32.820296280 (505h8m32.691379315s) [noise=true , address=45.91.116.14:4191 , piece_id=CRS5MFAUUM4QAMCSA27GIUNK2DYGCPZUQ45OI75BSALOPW32UCKQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 16:37:32.820301560 (505h8m32.691384615s) [noise=true , address=45.91.116.14:4191 , piece_id=CRS5MFAUUM4QAMCSA27GIUNK2DYGCPZUQ45OI75BSALOPW32UCKQ, seg_pos={0 0}, upload=443263]: Long tail cancellation due to optimal threshold Dec 26 16:37:32.820339582 (505h8m32.691422618s) [seg_pos={0 0}, upload=443263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:37:32.820416289 (505h8m32.691499324s) [upload=443263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:37:32.820420507 (505h8m32.691503542s) [upload=443263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:37:33.108070794 (505h8m32.979153829s) [upload=443263]: Done uploading segments... Dec 26 16:37:33.108097985 (505h8m32.97918102s) [upload=443263]: Upload finished. err: <nil> Duration: 6.1352196s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X...X.......XX..........X.........X..X...............X....] duration: 11.510925963s trace: https://jaeger.dev.storj.io/trace/4bc9becdddcc2037 FAIL (2025-12-26 16:38:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443223]: Uploading piece... Dec 26 16:34:46.586854691 (505h5m46.457937736s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.587812465 (505h5m46.4588955s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.588277791 (505h5m46.459360826s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590149876 (505h5m46.461232911s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590674354 (505h5m46.461757399s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591254680 (505h5m46.462337715s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591632860 (505h5m46.462715895s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591778888 (505h5m46.462861923s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592107854 (505h5m46.463190889s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592306482 (505h5m46.463389517s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592535257 (505h5m46.463618302s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592867591 (505h5m46.463950636s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592905783 (505h5m46.463988828s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593325633 (505h5m46.464408678s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593417989 (505h5m46.464501034s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593721267 (505h5m46.464804302s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.594046005 (505h5m46.46512905s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:47.885134462 (505h5m47.756217497s) [upload=443223]: Next returned nil segment Dec 26 16:34:47.885151225 (505h5m47.75623426s) [upload=443223]: Waiting for error group managing segments... Dec 26 16:34:48.239296449 (505h5m48.110379494s) [noise=true , address=64.49.13.140:4505 , piece_id=FXVVLCPQXGJLYWDGHDOQECN3S4ZZXTEEF7HNVLQW33IUEIPYW6OA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465120747 (505h5m48.336203792s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465174059 (505h5m48.336257094s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.468068951 (505h5m48.339151986s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.481407861 (505h5m48.352490896s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.482107242 (505h5m48.353190277s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483127506 (505h5m48.354210551s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483731115 (505h5m48.35481415s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.639322688 (505h5m48.510405733s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.760495594 (505h5m48.631578639s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.763286658 (505h5m48.634369693s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.771028379 (505h5m48.642111414s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.341293323 (505h5m49.212376368s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446731015 (505h5m49.31781406s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446757516 (505h5m49.317840561s) [seg_pos={0 0}, upload=443223]: Segment reached optimal threshold of 14 pieces. Dec 26 16:34:49.446947537 (505h5m49.318030572s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.446954180 (505h5m49.318037216s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447148209 (505h5m49.318231244s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447154361 (505h5m49.318237396s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447201702 (505h5m49.318284767s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447232551 (505h5m49.318315606s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447285240 (505h5m49.318368275s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447291793 (505h5m49.318374838s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447309938 (505h5m49.318392973s) [seg_pos={0 0}, upload=443223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:34:49.447375853 (505h5m49.318458898s) [upload=443223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:34:49.447380402 (505h5m49.318463437s) [upload=443223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:34:49.756168298 (505h5m49.627251333s) [upload=443223]: Done uploading segments... Dec 26 16:34:49.756192103 (505h5m49.627275149s) [upload=443223]: Upload finished. err: <nil> Duration: 11.51096657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 7.69404768s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:38:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 7.694078459s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.....X.............XXX...X...XX.....XX..........X......XXX..] duration: 5.000690287s trace: https://jaeger.dev.storj.io/trace/5b9c7e160a55f0bd FAIL (2025-12-26 16:38:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000719061s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [X..X...X.......XX..........X.........X..X...............X...] duration: 54.569939901s trace: https://jaeger.dev.storj.io/trace/4bc9becdddcc2037 FAIL (2025-12-26 16:38:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443223]: Uploading piece... Dec 26 16:34:46.586854691 (505h5m46.457937736s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.587812465 (505h5m46.4588955s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.588277791 (505h5m46.459360826s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590149876 (505h5m46.461232911s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590674354 (505h5m46.461757399s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591254680 (505h5m46.462337715s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591632860 (505h5m46.462715895s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591778888 (505h5m46.462861923s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592107854 (505h5m46.463190889s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592306482 (505h5m46.463389517s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592535257 (505h5m46.463618302s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592867591 (505h5m46.463950636s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592905783 (505h5m46.463988828s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593325633 (505h5m46.464408678s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593417989 (505h5m46.464501034s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593721267 (505h5m46.464804302s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.594046005 (505h5m46.46512905s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:47.885134462 (505h5m47.756217497s) [upload=443223]: Next returned nil segment Dec 26 16:34:47.885151225 (505h5m47.75623426s) [upload=443223]: Waiting for error group managing segments... Dec 26 16:34:48.239296449 (505h5m48.110379494s) [noise=true , address=64.49.13.140:4505 , piece_id=FXVVLCPQXGJLYWDGHDOQECN3S4ZZXTEEF7HNVLQW33IUEIPYW6OA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465120747 (505h5m48.336203792s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465174059 (505h5m48.336257094s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.468068951 (505h5m48.339151986s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.481407861 (505h5m48.352490896s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.482107242 (505h5m48.353190277s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483127506 (505h5m48.354210551s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483731115 (505h5m48.35481415s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.639322688 (505h5m48.510405733s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.760495594 (505h5m48.631578639s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.763286658 (505h5m48.634369693s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.771028379 (505h5m48.642111414s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.341293323 (505h5m49.212376368s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446731015 (505h5m49.31781406s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446757516 (505h5m49.317840561s) [seg_pos={0 0}, upload=443223]: Segment reached optimal threshold of 14 pieces. Dec 26 16:34:49.446947537 (505h5m49.318030572s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.446954180 (505h5m49.318037216s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447148209 (505h5m49.318231244s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447154361 (505h5m49.318237396s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447201702 (505h5m49.318284767s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447232551 (505h5m49.318315606s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447285240 (505h5m49.318368275s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447291793 (505h5m49.318374838s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447309938 (505h5m49.318392973s) [seg_pos={0 0}, upload=443223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:34:49.447375853 (505h5m49.318458898s) [upload=443223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:34:49.447380402 (505h5m49.318463437s) [upload=443223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:34:49.756168298 (505h5m49.627251333s) [upload=443223]: Done uploading segments... Dec 26 16:34:49.756192103 (505h5m49.627275149s) [upload=443223]: Upload finished. err: <nil> Duration: 54.569958016s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [........X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X..X] duration: 30.00317459s trace: https://jaeger.dev.storj.io/trace/3080d1fa188efb2e FAIL (2025-12-26 16:37:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: DIKBZ34FZCFLGFBHJJDX5BBZYNL2FTHNG3V6TSDCBPN5TXHVZMRQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: XINQ4KY3TRQ675HEJRVUQXMHMCP4EJWKDMIIKXDOXSMQUQXAPJYA) context deadline exceeded ======LOGS====== Dec 26 16:37:29.297178880 (505h8m29.168261925s) [upload=443263]: Starting upload Dec 26 16:37:29.297263381 (505h8m29.168346426s) [upload=443263]: Done starting upload Dec 26 16:37:29.297386335 (505h8m29.16846937s) [upload=443263]: Uploading segments... Dec 26 16:37:29.297473902 (505h8m29.168556947s) [upload=443263]: Got next segment. Inline: false Dec 26 16:37:29.297515431 (505h8m29.168598467s) [upload=443263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:37:29.297521964 (505h8m29.168605009s) [upload=443263]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:37:29.544301414 (505h8m29.415384449s) [seg_pos={0 0}, upload=443263]: Begin upload segment... Dec 26 16:37:29.544359325 (505h8m29.41544236s) [seg_pos={0 0}, upload=443263]: Done begin upload segment. Dec 26 16:37:29.544388360 (505h8m29.415471395s) [noise=true , address=45.91.116.14:4191 , piece_id=CRS5MFAUUM4QAMCSA27GIUNK2DYGCPZUQ45OI75BSALOPW32UCKQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.544452883 (505h8m29.415535918s) [noise=true , address=145.79.227.51:4409 , piece_id=TP5KPGXQT46CH5O2QUWAPAQQFHQTXVVE465C7NMTWDHYO2Q2RA3A, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.544975058 (505h8m29.416058093s) [noise=true , address=145.79.227.37:4566 , piece_id=MDTJ7G6ZAGFR42S5API3IBD2FGOZXI7NLE3I2OIVCF3P4T4NQ3LA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.545259659 (505h8m29.416342704s) [noise=true , address=145.79.227.13:4804 , piece_id=UOL2F3ULLGMJYUM3DDMLS37CYGAAVYAAXEJXULC4ZM7QIMLKOCSA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.545535645 (505h8m29.41661868s) [noise=true , address=45.91.116.4:4940 , piece_id=RZ4UPCTN4INRLKKB2Z7NAPMFGJROLNPBCGR463MO7OOQ6QDKVGNQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.545944193 (505h8m29.417027228s) [noise=true , address=45.91.116.5:4300 , piece_id=VN3C7NU2X2Q4CEOABPEDYTGLSACEAIUKOAPPAY7576JVUH6JI6DA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.546222914 (505h8m29.417305959s) [noise=true , address=77.90.31.10:4520 , piece_id=XINQ4KY3TRQ675HEJRVUQXMHMCP4EJWKDMIIKXDOXSMQUQXAPJYA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.546501424 (505h8m29.417584459s) [noise=true , address=145.79.227.22:4509 , piece_id=JTVULVGIRWP6YWVB2LRWKTDPW3LTVAWE4KN3LDPOPCAETV3UQMLA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.546824910 (505h8m29.417907955s) [noise=true , address=77.90.31.19:4841 , piece_id=AFOELLKOI6X6SCWVF23QRDYWFSUP2AM5DCKNL5NHDRUUKODIRUHA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.548078477 (505h8m29.419161512s) [noise=true , address=145.79.227.43:4157 , piece_id=A5H2QYKRN3HHWQUMWWUKGJLILKOQVLGQ5MM4CQRH5ECTOBVROTDA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.548184078 (505h8m29.419267113s) [noise=true , address=145.79.227.6:4257 , piece_id=XLHDMVMFGODI5L7LBBMD7MZSREBO7HLMR4LIL4IPMVK2HMTBJWUQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.549250759 (505h8m29.420333794s) [noise=true , address=145.79.227.39:4014 , piece_id=ONUVY2Q26V64DMVZ36UFD7WESAKWRPCKXO2LCB22EVOA3QOTTV4Q, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.549380546 (505h8m29.420463591s) [noise=true , address=45.91.116.7:4009 , piece_id=GAMH75PUG45WLH7K2FYMBYPVOTLDDBCJSP6Z7MB4NXYLEZPRP2VQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.550718123 (505h8m29.421801168s) [noise=true , address=145.79.227.45:4229 , piece_id=ZQYVHRSFNR5LSRQDKHN7JFZWFPZCLKK3D4DP7F3FRS2NSUXJ2THA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.551967743 (505h8m29.423050788s) [noise=true , address=145.79.227.46:4601 , piece_id=MWAIQ23QBDSQ5H7SE7OTI55T5C3KO7OJG6AGGKNT56KIOX2PDWPA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.553114406 (505h8m29.424197452s) [noise=true , address=77.90.31.14:4663 , piece_id=DIKBZ34FZCFLGFBHJJDX5BBZYNL2FTHNG3V6TSDCBPN5TXHVZMRQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:31.267104923 (505h8m31.138187968s) [upload=443263]: Next returned nil segment Dec 26 16:37:31.267129039 (505h8m31.138212084s) [upload=443263]: Waiting for error group managing segments... Dec 26 16:37:31.373605980 (505h8m31.244689025s) [noise=true , address=77.90.31.19:4841 , piece_id=AFOELLKOI6X6SCWVF23QRDYWFSUP2AM5DCKNL5NHDRUUKODIRUHA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 26 16:37:31.547383522 (505h8m31.418466567s) [noise=true , address=145.79.227.13:4804 , piece_id=UOL2F3ULLGMJYUM3DDMLS37CYGAAVYAAXEJXULC4ZM7QIMLKOCSA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.556110039 (505h8m31.427193074s) [noise=true , address=145.79.227.6:4257 , piece_id=XLHDMVMFGODI5L7LBBMD7MZSREBO7HLMR4LIL4IPMVK2HMTBJWUQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.556833557 (505h8m31.427916602s) [noise=true , address=145.79.227.39:4014 , piece_id=ONUVY2Q26V64DMVZ36UFD7WESAKWRPCKXO2LCB22EVOA3QOTTV4Q, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.558425728 (505h8m31.429508773s) [noise=true , address=145.79.227.46:4601 , piece_id=MWAIQ23QBDSQ5H7SE7OTI55T5C3KO7OJG6AGGKNT56KIOX2PDWPA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.559746073 (505h8m31.430829108s) [noise=true , address=145.79.227.22:4509 , piece_id=JTVULVGIRWP6YWVB2LRWKTDPW3LTVAWE4KN3LDPOPCAETV3UQMLA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.560533582 (505h8m31.431616628s) [noise=true , address=145.79.227.45:4229 , piece_id=ZQYVHRSFNR5LSRQDKHN7JFZWFPZCLKK3D4DP7F3FRS2NSUXJ2THA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.560664382 (505h8m31.431747417s) [noise=true , address=145.79.227.51:4409 , piece_id=TP5KPGXQT46CH5O2QUWAPAQQFHQTXVVE465C7NMTWDHYO2Q2RA3A, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.571627928 (505h8m31.442710973s) [noise=true , address=145.79.227.37:4566 , piece_id=MDTJ7G6ZAGFR42S5API3IBD2FGOZXI7NLE3I2OIVCF3P4T4NQ3LA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.843361235 (505h8m31.714444281s) [noise=true , address=77.90.31.14:4663 , piece_id=DIKBZ34FZCFLGFBHJJDX5BBZYNL2FTHNG3V6TSDCBPN5TXHVZMRQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.864291030 (505h8m31.735374065s) [noise=true , address=77.90.31.10:4520 , piece_id=XINQ4KY3TRQ675HEJRVUQXMHMCP4EJWKDMIIKXDOXSMQUQXAPJYA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.878425975 (505h8m31.749509011s) [noise=true , address=45.91.116.7:4009 , piece_id=GAMH75PUG45WLH7K2FYMBYPVOTLDDBCJSP6Z7MB4NXYLEZPRP2VQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.908628876 (505h8m31.779711911s) [noise=true , address=45.91.116.4:4940 , piece_id=RZ4UPCTN4INRLKKB2Z7NAPMFGJROLNPBCGR463MO7OOQ6QDKVGNQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:32.819947576 (505h8m32.691030611s) [noise=true , address=45.91.116.5:4300 , piece_id=VN3C7NU2X2Q4CEOABPEDYTGLSACEAIUKOAPPAY7576JVUH6JI6DA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:32.819983785 (505h8m32.69106683s) [seg_pos={0 0}, upload=443263]: Segment reached optimal threshold of 13 pieces. Dec 26 16:37:32.820176242 (505h8m32.691259277s) [noise=true , address=145.79.227.43:4157 , piece_id=A5H2QYKRN3HHWQUMWWUKGJLILKOQVLGQ5MM4CQRH5ECTOBVROTDA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 26 16:37:32.820182814 (505h8m32.691265849s) [noise=true , address=145.79.227.43:4157 , piece_id=A5H2QYKRN3HHWQUMWWUKGJLILKOQVLGQ5MM4CQRH5ECTOBVROTDA, seg_pos={0 0}, upload=443263]: Long tail cancellation due to optimal threshold Dec 26 16:37:32.820296280 (505h8m32.691379315s) [noise=true , address=45.91.116.14:4191 , piece_id=CRS5MFAUUM4QAMCSA27GIUNK2DYGCPZUQ45OI75BSALOPW32UCKQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 16:37:32.820301560 (505h8m32.691384615s) [noise=true , address=45.91.116.14:4191 , piece_id=CRS5MFAUUM4QAMCSA27GIUNK2DYGCPZUQ45OI75BSALOPW32UCKQ, seg_pos={0 0}, upload=443263]: Long tail cancellation due to optimal threshold Dec 26 16:37:32.820339582 (505h8m32.691422618s) [seg_pos={0 0}, upload=443263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:37:32.820416289 (505h8m32.691499324s) [upload=443263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:37:32.820420507 (505h8m32.691503542s) [upload=443263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:37:33.108070794 (505h8m32.979153829s) [upload=443263]: Done uploading segments... Dec 26 16:37:33.108097985 (505h8m32.97918102s) [upload=443263]: Upload finished. err: <nil> Duration: 30.003198745s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 18.569796043s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:37:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 18.569826682s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [......X.............XXX...X...XX.....XX..........X......XXX.] duration: 4.952580417s trace: https://jaeger.dev.storj.io/trace/5b9c7e160a55f0bd FAIL (2025-12-26 16:37:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.952608681s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X..X...X.......XX..........X.........X..X...............X..] duration: 53.238264228s trace: https://jaeger.dev.storj.io/trace/4bc9becdddcc2037 FAIL (2025-12-26 16:37:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443223]: Uploading piece... Dec 26 16:34:46.586854691 (505h5m46.457937736s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.587812465 (505h5m46.4588955s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.588277791 (505h5m46.459360826s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590149876 (505h5m46.461232911s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590674354 (505h5m46.461757399s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591254680 (505h5m46.462337715s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591632860 (505h5m46.462715895s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591778888 (505h5m46.462861923s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592107854 (505h5m46.463190889s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592306482 (505h5m46.463389517s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592535257 (505h5m46.463618302s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592867591 (505h5m46.463950636s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592905783 (505h5m46.463988828s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593325633 (505h5m46.464408678s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593417989 (505h5m46.464501034s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593721267 (505h5m46.464804302s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.594046005 (505h5m46.46512905s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:47.885134462 (505h5m47.756217497s) [upload=443223]: Next returned nil segment Dec 26 16:34:47.885151225 (505h5m47.75623426s) [upload=443223]: Waiting for error group managing segments... Dec 26 16:34:48.239296449 (505h5m48.110379494s) [noise=true , address=64.49.13.140:4505 , piece_id=FXVVLCPQXGJLYWDGHDOQECN3S4ZZXTEEF7HNVLQW33IUEIPYW6OA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465120747 (505h5m48.336203792s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465174059 (505h5m48.336257094s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.468068951 (505h5m48.339151986s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.481407861 (505h5m48.352490896s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.482107242 (505h5m48.353190277s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483127506 (505h5m48.354210551s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483731115 (505h5m48.35481415s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.639322688 (505h5m48.510405733s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.760495594 (505h5m48.631578639s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.763286658 (505h5m48.634369693s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.771028379 (505h5m48.642111414s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.341293323 (505h5m49.212376368s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446731015 (505h5m49.31781406s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446757516 (505h5m49.317840561s) [seg_pos={0 0}, upload=443223]: Segment reached optimal threshold of 14 pieces. Dec 26 16:34:49.446947537 (505h5m49.318030572s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.446954180 (505h5m49.318037216s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447148209 (505h5m49.318231244s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447154361 (505h5m49.318237396s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447201702 (505h5m49.318284767s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447232551 (505h5m49.318315606s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447285240 (505h5m49.318368275s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447291793 (505h5m49.318374838s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447309938 (505h5m49.318392973s) [seg_pos={0 0}, upload=443223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:34:49.447375853 (505h5m49.318458898s) [upload=443223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:34:49.447380402 (505h5m49.318463437s) [upload=443223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:34:49.756168298 (505h5m49.627251333s) [upload=443223]: Done uploading segments... Dec 26 16:34:49.756192103 (505h5m49.627275149s) [upload=443223]: Upload finished. err: <nil> Duration: 53.238283805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.........X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X..] duration: 6.040046463s trace: https://jaeger.dev.storj.io/trace/64f12cb65d7db0b6 FAIL (2025-12-26 16:36:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: KK322Q3BL4VKH3XQSMWU7VZ2A32UQLPXS3HR3APAOLAFRKIG5NSQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 25T5NRLRGQ7F77Q5VESFSJG66KX6IRSVZW3Z5RIS2GP4SN6YZRIA) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: ZB3STKDNGCGCXXNJTGFABJ3YKXGHTCIJEKC3CUSX6HUTGLTBM7QQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: EJSHXLUUFD4GBSJHRYU2LNTSM6RSD7DLKSBHOACZ63USJBDEWTEA) context deadline exceeded ======LOGS====== (505h6m25.177144957s) [upload=443230]: Uploading segments... Dec 26 16:35:25.306210225 (505h6m25.17729326s) [upload=443230]: Got next segment. Inline: false Dec 26 16:35:25.306276360 (505h6m25.177359395s) [upload=443230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:35:25.306281611 (505h6m25.177364646s) [upload=443230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:35:25.552989044 (505h6m25.424072079s) [seg_pos={0 0}, upload=443230]: Begin upload segment... Dec 26 16:35:25.553040944 (505h6m25.424123989s) [seg_pos={0 0}, upload=443230]: Done begin upload segment. Dec 26 16:35:25.553122709 (505h6m25.424205754s) [noise=true , address=145.79.227.11:4732 , piece_id=DM4ENZHLFTWBAYXC72PUCL6ZZQXT2TWS2B4VV4GU5QBCHD7LFT4A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.553163637 (505h6m25.424246682s) [noise=true , address=145.79.227.31:4799 , piece_id=ZOVTL43UW4REIXOC2RF6UN73ITFZEVQV4GD44R4K5FS6HA4I4I5Q, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.553923965 (505h6m25.42500701s) [noise=true , address=45.91.116.6:4973 , piece_id=MRAAH4XLPLCQC72WIKOJMCG6MKAYR4LFFVOFWX5K6ILHVBS6ZNAQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.554399040 (505h6m25.425482085s) [noise=true , address=145.79.227.50:4375 , piece_id=NU6XHJVGLP3HSULT5T4YRSZGWNVSCTAZJZC53EMYFGIPCF36CKUQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.554883532 (505h6m25.425966587s) [noise=true , address=145.79.227.38:4978 , piece_id=OKFY6FYGXFDUFO64BF275IDMWDAQE46R5O2YFOI2DTOWVTREGHWA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.555373064 (505h6m25.426456109s) [noise=true , address=145.79.227.8:4330 , piece_id=HCB24RIFWDQRDKBFLUL4UJEBN2AEBCW4HUDDLRFJASGWCAE7PPNQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.555865983 (505h6m25.426949028s) [noise=true , address=145.79.227.45:4229 , piece_id=LGDVW53JZQOS2WV7C77VYXVZGLDSQBYJA7FB7HPY65E5CBPBNF6A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.556875355 (505h6m25.4279584s) [noise=true , address=145.79.227.18:4007 , piece_id=VVBTOPZPD6KBKTJT5XY7ECZEKTKGOMW5JQ3WD7CS5WXB2RTLZIIA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.557919453 (505h6m25.429002498s) [noise=true , address=145.79.227.49:4337 , piece_id=XBKBKBUGRE5JZSBO2ZPBYMLHBEGXVG76HDMTJYRPFKUSIGX3TOSQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.559175565 (505h6m25.4302586s) [noise=true , address=45.91.116.14:4192 , piece_id=KK322Q3BL4VKH3XQSMWU7VZ2A32UQLPXS3HR3APAOLAFRKIG5NSQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.560312209 (505h6m25.431395254s) [noise=true , address=77.90.31.10:4518 , piece_id=25T5NRLRGQ7F77Q5VESFSJG66KX6IRSVZW3Z5RIS2GP4SN6YZRIA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.560383074 (505h6m25.431466129s) [noise=true , address=145.79.227.48:4301 , piece_id=ZB3STKDNGCGCXXNJTGFABJ3YKXGHTCIJEKC3CUSX6HUTGLTBM7QQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.561525960 (505h6m25.432608995s) [noise=true , address=145.79.227.6:4259 , piece_id=OOCYZFJWNM2FJ5KMIEY73LUCETOHOWPK22UI6UUD5XNG2WGK5W7A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.562958318 (505h6m25.434041353s) [noise=true , address=145.79.227.2:4113 , piece_id=3UNELYLESMPBLPFFXTODTB3NH5BVD57OWHMX6FN3CTV25CCTVG6A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.563017440 (505h6m25.434100485s) [noise=true , address=145.79.227.34:4907 , piece_id=M6GET3TBB2SETWCOMBTCVA3TKCY3Q3X2RPZRKXFSIMHRPJXHC5ZA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.564357242 (505h6m25.435440277s) [noise=true , address=45.91.116.5:4297 , piece_id=EJSHXLUUFD4GBSJHRYU2LNTSM6RSD7DLKSBHOACZ63USJBDEWTEA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:27.167253516 (505h6m27.038336551s) [upload=443230]: Next returned nil segment Dec 26 16:35:27.167267121 (505h6m27.038350166s) [upload=443230]: Waiting for error group managing segments... Dec 26 16:35:27.552207475 (505h6m27.42329051s) [noise=true , address=145.79.227.11:4732 , piece_id=DM4ENZHLFTWBAYXC72PUCL6ZZQXT2TWS2B4VV4GU5QBCHD7LFT4A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.566801595 (505h6m27.43788463s) [noise=true , address=145.79.227.48:4301 , piece_id=ZB3STKDNGCGCXXNJTGFABJ3YKXGHTCIJEKC3CUSX6HUTGLTBM7QQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.569841033 (505h6m27.440924078s) [noise=true , address=145.79.227.8:4330 , piece_id=HCB24RIFWDQRDKBFLUL4UJEBN2AEBCW4HUDDLRFJASGWCAE7PPNQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.573863191 (505h6m27.444946226s) [noise=true , address=145.79.227.49:4337 , piece_id=XBKBKBUGRE5JZSBO2ZPBYMLHBEGXVG76HDMTJYRPFKUSIGX3TOSQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.588141019 (505h6m27.459224064s) [noise=true , address=145.79.227.38:4978 , piece_id=OKFY6FYGXFDUFO64BF275IDMWDAQE46R5O2YFOI2DTOWVTREGHWA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.588793191 (505h6m27.459876226s) [noise=true , address=145.79.227.31:4799 , piece_id=ZOVTL43UW4REIXOC2RF6UN73ITFZEVQV4GD44R4K5FS6HA4I4I5Q, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.589226415 (505h6m27.46030946s) [noise=true , address=145.79.227.45:4229 , piece_id=LGDVW53JZQOS2WV7C77VYXVZGLDSQBYJA7FB7HPY65E5CBPBNF6A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.592841078 (505h6m27.463924113s) [noise=true , address=145.79.227.18:4007 , piece_id=VVBTOPZPD6KBKTJT5XY7ECZEKTKGOMW5JQ3WD7CS5WXB2RTLZIIA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.594745725 (505h6m27.46582877s) [noise=true , address=145.79.227.50:4375 , piece_id=NU6XHJVGLP3HSULT5T4YRSZGWNVSCTAZJZC53EMYFGIPCF36CKUQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.790554880 (505h6m27.661637915s) [noise=true , address=45.91.116.6:4973 , piece_id=MRAAH4XLPLCQC72WIKOJMCG6MKAYR4LFFVOFWX5K6ILHVBS6ZNAQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.800118801 (505h6m27.671201847s) [noise=true , address=45.91.116.14:4192 , piece_id=KK322Q3BL4VKH3XQSMWU7VZ2A32UQLPXS3HR3APAOLAFRKIG5NSQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.811160206 (505h6m27.682243241s) [noise=true , address=77.90.31.10:4518 , piece_id=25T5NRLRGQ7F77Q5VESFSJG66KX6IRSVZW3Z5RIS2GP4SN6YZRIA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.825394921 (505h6m27.696477966s) [noise=true , address=45.91.116.5:4297 , piece_id=EJSHXLUUFD4GBSJHRYU2LNTSM6RSD7DLKSBHOACZ63USJBDEWTEA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.825416092 (505h6m27.696499127s) [seg_pos={0 0}, upload=443230]: Segment reached optimal threshold of 13 pieces. Dec 26 16:35:27.825610271 (505h6m27.696693306s) [noise=true , address=145.79.227.2:4113 , piece_id=3UNELYLESMPBLPFFXTODTB3NH5BVD57OWHMX6FN3CTV25CCTVG6A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:43570->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:43570->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43570->145.79.227.2:4113: use of closed network connection Dec 26 16:35:27.825617766 (505h6m27.696700801s) [noise=true , address=145.79.227.2:4113 , piece_id=3UNELYLESMPBLPFFXTODTB3NH5BVD57OWHMX6FN3CTV25CCTVG6A, seg_pos={0 0}, upload=443230]: Long tail cancellation due to optimal threshold Dec 26 16:35:27.825930141 (505h6m27.697013176s) [noise=true , address=145.79.227.6:4259 , piece_id=OOCYZFJWNM2FJ5KMIEY73LUCETOHOWPK22UI6UUD5XNG2WGK5W7A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:48232->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48232->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48232->145.79.227.6:4259: use of closed network connection Dec 26 16:35:27.825944909 (505h6m27.697027944s) [noise=true , address=145.79.227.6:4259 , piece_id=OOCYZFJWNM2FJ5KMIEY73LUCETOHOWPK22UI6UUD5XNG2WGK5W7A, seg_pos={0 0}, upload=443230]: Long tail cancellation due to optimal threshold Dec 26 16:35:27.825979014 (505h6m27.697062059s) [noise=true , address=145.79.227.34:4907 , piece_id=M6GET3TBB2SETWCOMBTCVA3TKCY3Q3X2RPZRKXFSIMHRPJXHC5ZA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 16:35:27.825988341 (505h6m27.697071386s) [noise=true , address=145.79.227.34:4907 , piece_id=M6GET3TBB2SETWCOMBTCVA3TKCY3Q3X2RPZRKXFSIMHRPJXHC5ZA, seg_pos={0 0}, upload=443230]: Long tail cancellation due to optimal threshold Dec 26 16:35:27.826004603 (505h6m27.697087638s) [seg_pos={0 0}, upload=443230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:35:27.826078824 (505h6m27.697161869s) [upload=443230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:35:27.826085026 (505h6m27.697168071s) [upload=443230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:35:28.123831196 (505h6m27.994914242s) [upload=443230]: Done uploading segments... Dec 26 16:35:28.123855914 (505h6m27.994938959s) [upload=443230]: Upload finished. err: <nil> Duration: 6.040071159s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.......X.............XXX...X...XX.....XX..........X......XXX] duration: 5.001619456s trace: https://jaeger.dev.storj.io/trace/5b9c7e160a55f0bd FAIL (2025-12-26 16:36:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001640156s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 8.148480479s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:36:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 8.148620075s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..........X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X.] duration: 6.154464535s trace: https://jaeger.dev.storj.io/trace/64f12cb65d7db0b6 FAIL (2025-12-26 16:36:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: KK322Q3BL4VKH3XQSMWU7VZ2A32UQLPXS3HR3APAOLAFRKIG5NSQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 25T5NRLRGQ7F77Q5VESFSJG66KX6IRSVZW3Z5RIS2GP4SN6YZRIA) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: ZB3STKDNGCGCXXNJTGFABJ3YKXGHTCIJEKC3CUSX6HUTGLTBM7QQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: EJSHXLUUFD4GBSJHRYU2LNTSM6RSD7DLKSBHOACZ63USJBDEWTEA) context deadline exceeded ======LOGS====== (505h6m25.177144957s) [upload=443230]: Uploading segments... Dec 26 16:35:25.306210225 (505h6m25.17729326s) [upload=443230]: Got next segment. Inline: false Dec 26 16:35:25.306276360 (505h6m25.177359395s) [upload=443230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:35:25.306281611 (505h6m25.177364646s) [upload=443230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:35:25.552989044 (505h6m25.424072079s) [seg_pos={0 0}, upload=443230]: Begin upload segment... Dec 26 16:35:25.553040944 (505h6m25.424123989s) [seg_pos={0 0}, upload=443230]: Done begin upload segment. Dec 26 16:35:25.553122709 (505h6m25.424205754s) [noise=true , address=145.79.227.11:4732 , piece_id=DM4ENZHLFTWBAYXC72PUCL6ZZQXT2TWS2B4VV4GU5QBCHD7LFT4A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.553163637 (505h6m25.424246682s) [noise=true , address=145.79.227.31:4799 , piece_id=ZOVTL43UW4REIXOC2RF6UN73ITFZEVQV4GD44R4K5FS6HA4I4I5Q, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.553923965 (505h6m25.42500701s) [noise=true , address=45.91.116.6:4973 , piece_id=MRAAH4XLPLCQC72WIKOJMCG6MKAYR4LFFVOFWX5K6ILHVBS6ZNAQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.554399040 (505h6m25.425482085s) [noise=true , address=145.79.227.50:4375 , piece_id=NU6XHJVGLP3HSULT5T4YRSZGWNVSCTAZJZC53EMYFGIPCF36CKUQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.554883532 (505h6m25.425966587s) [noise=true , address=145.79.227.38:4978 , piece_id=OKFY6FYGXFDUFO64BF275IDMWDAQE46R5O2YFOI2DTOWVTREGHWA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.555373064 (505h6m25.426456109s) [noise=true , address=145.79.227.8:4330 , piece_id=HCB24RIFWDQRDKBFLUL4UJEBN2AEBCW4HUDDLRFJASGWCAE7PPNQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.555865983 (505h6m25.426949028s) [noise=true , address=145.79.227.45:4229 , piece_id=LGDVW53JZQOS2WV7C77VYXVZGLDSQBYJA7FB7HPY65E5CBPBNF6A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.556875355 (505h6m25.4279584s) [noise=true , address=145.79.227.18:4007 , piece_id=VVBTOPZPD6KBKTJT5XY7ECZEKTKGOMW5JQ3WD7CS5WXB2RTLZIIA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.557919453 (505h6m25.429002498s) [noise=true , address=145.79.227.49:4337 , piece_id=XBKBKBUGRE5JZSBO2ZPBYMLHBEGXVG76HDMTJYRPFKUSIGX3TOSQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.559175565 (505h6m25.4302586s) [noise=true , address=45.91.116.14:4192 , piece_id=KK322Q3BL4VKH3XQSMWU7VZ2A32UQLPXS3HR3APAOLAFRKIG5NSQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.560312209 (505h6m25.431395254s) [noise=true , address=77.90.31.10:4518 , piece_id=25T5NRLRGQ7F77Q5VESFSJG66KX6IRSVZW3Z5RIS2GP4SN6YZRIA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.560383074 (505h6m25.431466129s) [noise=true , address=145.79.227.48:4301 , piece_id=ZB3STKDNGCGCXXNJTGFABJ3YKXGHTCIJEKC3CUSX6HUTGLTBM7QQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.561525960 (505h6m25.432608995s) [noise=true , address=145.79.227.6:4259 , piece_id=OOCYZFJWNM2FJ5KMIEY73LUCETOHOWPK22UI6UUD5XNG2WGK5W7A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.562958318 (505h6m25.434041353s) [noise=true , address=145.79.227.2:4113 , piece_id=3UNELYLESMPBLPFFXTODTB3NH5BVD57OWHMX6FN3CTV25CCTVG6A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.563017440 (505h6m25.434100485s) [noise=true , address=145.79.227.34:4907 , piece_id=M6GET3TBB2SETWCOMBTCVA3TKCY3Q3X2RPZRKXFSIMHRPJXHC5ZA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.564357242 (505h6m25.435440277s) [noise=true , address=45.91.116.5:4297 , piece_id=EJSHXLUUFD4GBSJHRYU2LNTSM6RSD7DLKSBHOACZ63USJBDEWTEA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:27.167253516 (505h6m27.038336551s) [upload=443230]: Next returned nil segment Dec 26 16:35:27.167267121 (505h6m27.038350166s) [upload=443230]: Waiting for error group managing segments... Dec 26 16:35:27.552207475 (505h6m27.42329051s) [noise=true , address=145.79.227.11:4732 , piece_id=DM4ENZHLFTWBAYXC72PUCL6ZZQXT2TWS2B4VV4GU5QBCHD7LFT4A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.566801595 (505h6m27.43788463s) [noise=true , address=145.79.227.48:4301 , piece_id=ZB3STKDNGCGCXXNJTGFABJ3YKXGHTCIJEKC3CUSX6HUTGLTBM7QQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.569841033 (505h6m27.440924078s) [noise=true , address=145.79.227.8:4330 , piece_id=HCB24RIFWDQRDKBFLUL4UJEBN2AEBCW4HUDDLRFJASGWCAE7PPNQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.573863191 (505h6m27.444946226s) [noise=true , address=145.79.227.49:4337 , piece_id=XBKBKBUGRE5JZSBO2ZPBYMLHBEGXVG76HDMTJYRPFKUSIGX3TOSQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.588141019 (505h6m27.459224064s) [noise=true , address=145.79.227.38:4978 , piece_id=OKFY6FYGXFDUFO64BF275IDMWDAQE46R5O2YFOI2DTOWVTREGHWA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.588793191 (505h6m27.459876226s) [noise=true , address=145.79.227.31:4799 , piece_id=ZOVTL43UW4REIXOC2RF6UN73ITFZEVQV4GD44R4K5FS6HA4I4I5Q, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.589226415 (505h6m27.46030946s) [noise=true , address=145.79.227.45:4229 , piece_id=LGDVW53JZQOS2WV7C77VYXVZGLDSQBYJA7FB7HPY65E5CBPBNF6A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.592841078 (505h6m27.463924113s) [noise=true , address=145.79.227.18:4007 , piece_id=VVBTOPZPD6KBKTJT5XY7ECZEKTKGOMW5JQ3WD7CS5WXB2RTLZIIA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.594745725 (505h6m27.46582877s) [noise=true , address=145.79.227.50:4375 , piece_id=NU6XHJVGLP3HSULT5T4YRSZGWNVSCTAZJZC53EMYFGIPCF36CKUQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.790554880 (505h6m27.661637915s) [noise=true , address=45.91.116.6:4973 , piece_id=MRAAH4XLPLCQC72WIKOJMCG6MKAYR4LFFVOFWX5K6ILHVBS6ZNAQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.800118801 (505h6m27.671201847s) [noise=true , address=45.91.116.14:4192 , piece_id=KK322Q3BL4VKH3XQSMWU7VZ2A32UQLPXS3HR3APAOLAFRKIG5NSQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.811160206 (505h6m27.682243241s) [noise=true , address=77.90.31.10:4518 , piece_id=25T5NRLRGQ7F77Q5VESFSJG66KX6IRSVZW3Z5RIS2GP4SN6YZRIA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.825394921 (505h6m27.696477966s) [noise=true , address=45.91.116.5:4297 , piece_id=EJSHXLUUFD4GBSJHRYU2LNTSM6RSD7DLKSBHOACZ63USJBDEWTEA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.825416092 (505h6m27.696499127s) [seg_pos={0 0}, upload=443230]: Segment reached optimal threshold of 13 pieces. Dec 26 16:35:27.825610271 (505h6m27.696693306s) [noise=true , address=145.79.227.2:4113 , piece_id=3UNELYLESMPBLPFFXTODTB3NH5BVD57OWHMX6FN3CTV25CCTVG6A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:43570->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:43570->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43570->145.79.227.2:4113: use of closed network connection Dec 26 16:35:27.825617766 (505h6m27.696700801s) [noise=true , address=145.79.227.2:4113 , piece_id=3UNELYLESMPBLPFFXTODTB3NH5BVD57OWHMX6FN3CTV25CCTVG6A, seg_pos={0 0}, upload=443230]: Long tail cancellation due to optimal threshold Dec 26 16:35:27.825930141 (505h6m27.697013176s) [noise=true , address=145.79.227.6:4259 , piece_id=OOCYZFJWNM2FJ5KMIEY73LUCETOHOWPK22UI6UUD5XNG2WGK5W7A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:48232->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48232->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48232->145.79.227.6:4259: use of closed network connection Dec 26 16:35:27.825944909 (505h6m27.697027944s) [noise=true , address=145.79.227.6:4259 , piece_id=OOCYZFJWNM2FJ5KMIEY73LUCETOHOWPK22UI6UUD5XNG2WGK5W7A, seg_pos={0 0}, upload=443230]: Long tail cancellation due to optimal threshold Dec 26 16:35:27.825979014 (505h6m27.697062059s) [noise=true , address=145.79.227.34:4907 , piece_id=M6GET3TBB2SETWCOMBTCVA3TKCY3Q3X2RPZRKXFSIMHRPJXHC5ZA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 16:35:27.825988341 (505h6m27.697071386s) [noise=true , address=145.79.227.34:4907 , piece_id=M6GET3TBB2SETWCOMBTCVA3TKCY3Q3X2RPZRKXFSIMHRPJXHC5ZA, seg_pos={0 0}, upload=443230]: Long tail cancellation due to optimal threshold Dec 26 16:35:27.826004603 (505h6m27.697087638s) [seg_pos={0 0}, upload=443230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:35:27.826078824 (505h6m27.697161869s) [upload=443230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:35:27.826085026 (505h6m27.697168071s) [upload=443230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:35:28.123831196 (505h6m27.994914242s) [upload=443230]: Done uploading segments... Dec 26 16:35:28.123855914 (505h6m27.994938959s) [upload=443230]: Upload finished. err: <nil> Duration: 6.154487849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X..X...X.......XX..........X.........X..X...............X.] duration: 55.618224808s trace: https://jaeger.dev.storj.io/trace/4bc9becdddcc2037 FAIL (2025-12-26 16:36:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443223]: Uploading piece... Dec 26 16:34:46.586854691 (505h5m46.457937736s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.587812465 (505h5m46.4588955s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.588277791 (505h5m46.459360826s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590149876 (505h5m46.461232911s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590674354 (505h5m46.461757399s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591254680 (505h5m46.462337715s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591632860 (505h5m46.462715895s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591778888 (505h5m46.462861923s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592107854 (505h5m46.463190889s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592306482 (505h5m46.463389517s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592535257 (505h5m46.463618302s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592867591 (505h5m46.463950636s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592905783 (505h5m46.463988828s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593325633 (505h5m46.464408678s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593417989 (505h5m46.464501034s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593721267 (505h5m46.464804302s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.594046005 (505h5m46.46512905s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:47.885134462 (505h5m47.756217497s) [upload=443223]: Next returned nil segment Dec 26 16:34:47.885151225 (505h5m47.75623426s) [upload=443223]: Waiting for error group managing segments... Dec 26 16:34:48.239296449 (505h5m48.110379494s) [noise=true , address=64.49.13.140:4505 , piece_id=FXVVLCPQXGJLYWDGHDOQECN3S4ZZXTEEF7HNVLQW33IUEIPYW6OA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465120747 (505h5m48.336203792s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465174059 (505h5m48.336257094s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.468068951 (505h5m48.339151986s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.481407861 (505h5m48.352490896s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.482107242 (505h5m48.353190277s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483127506 (505h5m48.354210551s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483731115 (505h5m48.35481415s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.639322688 (505h5m48.510405733s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.760495594 (505h5m48.631578639s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.763286658 (505h5m48.634369693s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.771028379 (505h5m48.642111414s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.341293323 (505h5m49.212376368s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446731015 (505h5m49.31781406s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446757516 (505h5m49.317840561s) [seg_pos={0 0}, upload=443223]: Segment reached optimal threshold of 14 pieces. Dec 26 16:34:49.446947537 (505h5m49.318030572s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.446954180 (505h5m49.318037216s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447148209 (505h5m49.318231244s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447154361 (505h5m49.318237396s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447201702 (505h5m49.318284767s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447232551 (505h5m49.318315606s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447285240 (505h5m49.318368275s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447291793 (505h5m49.318374838s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447309938 (505h5m49.318392973s) [seg_pos={0 0}, upload=443223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:34:49.447375853 (505h5m49.318458898s) [upload=443223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:34:49.447380402 (505h5m49.318463437s) [upload=443223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:34:49.756168298 (505h5m49.627251333s) [upload=443223]: Done uploading segments... Dec 26 16:34:49.756192103 (505h5m49.627275149s) [upload=443223]: Upload finished. err: <nil> Duration: 55.618254635s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...........X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X] duration: 30.002824593s trace: https://jaeger.dev.storj.io/trace/64f12cb65d7db0b6 FAIL (2025-12-26 16:35:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: KK322Q3BL4VKH3XQSMWU7VZ2A32UQLPXS3HR3APAOLAFRKIG5NSQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 25T5NRLRGQ7F77Q5VESFSJG66KX6IRSVZW3Z5RIS2GP4SN6YZRIA) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: ZB3STKDNGCGCXXNJTGFABJ3YKXGHTCIJEKC3CUSX6HUTGLTBM7QQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: EJSHXLUUFD4GBSJHRYU2LNTSM6RSD7DLKSBHOACZ63USJBDEWTEA) context deadline exceeded ======LOGS====== (505h6m25.177144957s) [upload=443230]: Uploading segments... Dec 26 16:35:25.306210225 (505h6m25.17729326s) [upload=443230]: Got next segment. Inline: false Dec 26 16:35:25.306276360 (505h6m25.177359395s) [upload=443230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:35:25.306281611 (505h6m25.177364646s) [upload=443230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:35:25.552989044 (505h6m25.424072079s) [seg_pos={0 0}, upload=443230]: Begin upload segment... Dec 26 16:35:25.553040944 (505h6m25.424123989s) [seg_pos={0 0}, upload=443230]: Done begin upload segment. Dec 26 16:35:25.553122709 (505h6m25.424205754s) [noise=true , address=145.79.227.11:4732 , piece_id=DM4ENZHLFTWBAYXC72PUCL6ZZQXT2TWS2B4VV4GU5QBCHD7LFT4A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.553163637 (505h6m25.424246682s) [noise=true , address=145.79.227.31:4799 , piece_id=ZOVTL43UW4REIXOC2RF6UN73ITFZEVQV4GD44R4K5FS6HA4I4I5Q, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.553923965 (505h6m25.42500701s) [noise=true , address=45.91.116.6:4973 , piece_id=MRAAH4XLPLCQC72WIKOJMCG6MKAYR4LFFVOFWX5K6ILHVBS6ZNAQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.554399040 (505h6m25.425482085s) [noise=true , address=145.79.227.50:4375 , piece_id=NU6XHJVGLP3HSULT5T4YRSZGWNVSCTAZJZC53EMYFGIPCF36CKUQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.554883532 (505h6m25.425966587s) [noise=true , address=145.79.227.38:4978 , piece_id=OKFY6FYGXFDUFO64BF275IDMWDAQE46R5O2YFOI2DTOWVTREGHWA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.555373064 (505h6m25.426456109s) [noise=true , address=145.79.227.8:4330 , piece_id=HCB24RIFWDQRDKBFLUL4UJEBN2AEBCW4HUDDLRFJASGWCAE7PPNQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.555865983 (505h6m25.426949028s) [noise=true , address=145.79.227.45:4229 , piece_id=LGDVW53JZQOS2WV7C77VYXVZGLDSQBYJA7FB7HPY65E5CBPBNF6A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.556875355 (505h6m25.4279584s) [noise=true , address=145.79.227.18:4007 , piece_id=VVBTOPZPD6KBKTJT5XY7ECZEKTKGOMW5JQ3WD7CS5WXB2RTLZIIA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.557919453 (505h6m25.429002498s) [noise=true , address=145.79.227.49:4337 , piece_id=XBKBKBUGRE5JZSBO2ZPBYMLHBEGXVG76HDMTJYRPFKUSIGX3TOSQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.559175565 (505h6m25.4302586s) [noise=true , address=45.91.116.14:4192 , piece_id=KK322Q3BL4VKH3XQSMWU7VZ2A32UQLPXS3HR3APAOLAFRKIG5NSQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.560312209 (505h6m25.431395254s) [noise=true , address=77.90.31.10:4518 , piece_id=25T5NRLRGQ7F77Q5VESFSJG66KX6IRSVZW3Z5RIS2GP4SN6YZRIA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.560383074 (505h6m25.431466129s) [noise=true , address=145.79.227.48:4301 , piece_id=ZB3STKDNGCGCXXNJTGFABJ3YKXGHTCIJEKC3CUSX6HUTGLTBM7QQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.561525960 (505h6m25.432608995s) [noise=true , address=145.79.227.6:4259 , piece_id=OOCYZFJWNM2FJ5KMIEY73LUCETOHOWPK22UI6UUD5XNG2WGK5W7A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.562958318 (505h6m25.434041353s) [noise=true , address=145.79.227.2:4113 , piece_id=3UNELYLESMPBLPFFXTODTB3NH5BVD57OWHMX6FN3CTV25CCTVG6A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.563017440 (505h6m25.434100485s) [noise=true , address=145.79.227.34:4907 , piece_id=M6GET3TBB2SETWCOMBTCVA3TKCY3Q3X2RPZRKXFSIMHRPJXHC5ZA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.564357242 (505h6m25.435440277s) [noise=true , address=45.91.116.5:4297 , piece_id=EJSHXLUUFD4GBSJHRYU2LNTSM6RSD7DLKSBHOACZ63USJBDEWTEA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:27.167253516 (505h6m27.038336551s) [upload=443230]: Next returned nil segment Dec 26 16:35:27.167267121 (505h6m27.038350166s) [upload=443230]: Waiting for error group managing segments... Dec 26 16:35:27.552207475 (505h6m27.42329051s) [noise=true , address=145.79.227.11:4732 , piece_id=DM4ENZHLFTWBAYXC72PUCL6ZZQXT2TWS2B4VV4GU5QBCHD7LFT4A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.566801595 (505h6m27.43788463s) [noise=true , address=145.79.227.48:4301 , piece_id=ZB3STKDNGCGCXXNJTGFABJ3YKXGHTCIJEKC3CUSX6HUTGLTBM7QQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.569841033 (505h6m27.440924078s) [noise=true , address=145.79.227.8:4330 , piece_id=HCB24RIFWDQRDKBFLUL4UJEBN2AEBCW4HUDDLRFJASGWCAE7PPNQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.573863191 (505h6m27.444946226s) [noise=true , address=145.79.227.49:4337 , piece_id=XBKBKBUGRE5JZSBO2ZPBYMLHBEGXVG76HDMTJYRPFKUSIGX3TOSQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.588141019 (505h6m27.459224064s) [noise=true , address=145.79.227.38:4978 , piece_id=OKFY6FYGXFDUFO64BF275IDMWDAQE46R5O2YFOI2DTOWVTREGHWA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.588793191 (505h6m27.459876226s) [noise=true , address=145.79.227.31:4799 , piece_id=ZOVTL43UW4REIXOC2RF6UN73ITFZEVQV4GD44R4K5FS6HA4I4I5Q, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.589226415 (505h6m27.46030946s) [noise=true , address=145.79.227.45:4229 , piece_id=LGDVW53JZQOS2WV7C77VYXVZGLDSQBYJA7FB7HPY65E5CBPBNF6A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.592841078 (505h6m27.463924113s) [noise=true , address=145.79.227.18:4007 , piece_id=VVBTOPZPD6KBKTJT5XY7ECZEKTKGOMW5JQ3WD7CS5WXB2RTLZIIA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.594745725 (505h6m27.46582877s) [noise=true , address=145.79.227.50:4375 , piece_id=NU6XHJVGLP3HSULT5T4YRSZGWNVSCTAZJZC53EMYFGIPCF36CKUQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.790554880 (505h6m27.661637915s) [noise=true , address=45.91.116.6:4973 , piece_id=MRAAH4XLPLCQC72WIKOJMCG6MKAYR4LFFVOFWX5K6ILHVBS6ZNAQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.800118801 (505h6m27.671201847s) [noise=true , address=45.91.116.14:4192 , piece_id=KK322Q3BL4VKH3XQSMWU7VZ2A32UQLPXS3HR3APAOLAFRKIG5NSQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.811160206 (505h6m27.682243241s) [noise=true , address=77.90.31.10:4518 , piece_id=25T5NRLRGQ7F77Q5VESFSJG66KX6IRSVZW3Z5RIS2GP4SN6YZRIA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.825394921 (505h6m27.696477966s) [noise=true , address=45.91.116.5:4297 , piece_id=EJSHXLUUFD4GBSJHRYU2LNTSM6RSD7DLKSBHOACZ63USJBDEWTEA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.825416092 (505h6m27.696499127s) [seg_pos={0 0}, upload=443230]: Segment reached optimal threshold of 13 pieces. Dec 26 16:35:27.825610271 (505h6m27.696693306s) [noise=true , address=145.79.227.2:4113 , piece_id=3UNELYLESMPBLPFFXTODTB3NH5BVD57OWHMX6FN3CTV25CCTVG6A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:43570->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:43570->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43570->145.79.227.2:4113: use of closed network connection Dec 26 16:35:27.825617766 (505h6m27.696700801s) [noise=true , address=145.79.227.2:4113 , piece_id=3UNELYLESMPBLPFFXTODTB3NH5BVD57OWHMX6FN3CTV25CCTVG6A, seg_pos={0 0}, upload=443230]: Long tail cancellation due to optimal threshold Dec 26 16:35:27.825930141 (505h6m27.697013176s) [noise=true , address=145.79.227.6:4259 , piece_id=OOCYZFJWNM2FJ5KMIEY73LUCETOHOWPK22UI6UUD5XNG2WGK5W7A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:48232->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48232->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48232->145.79.227.6:4259: use of closed network connection Dec 26 16:35:27.825944909 (505h6m27.697027944s) [noise=true , address=145.79.227.6:4259 , piece_id=OOCYZFJWNM2FJ5KMIEY73LUCETOHOWPK22UI6UUD5XNG2WGK5W7A, seg_pos={0 0}, upload=443230]: Long tail cancellation due to optimal threshold Dec 26 16:35:27.825979014 (505h6m27.697062059s) [noise=true , address=145.79.227.34:4907 , piece_id=M6GET3TBB2SETWCOMBTCVA3TKCY3Q3X2RPZRKXFSIMHRPJXHC5ZA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 16:35:27.825988341 (505h6m27.697071386s) [noise=true , address=145.79.227.34:4907 , piece_id=M6GET3TBB2SETWCOMBTCVA3TKCY3Q3X2RPZRKXFSIMHRPJXHC5ZA, seg_pos={0 0}, upload=443230]: Long tail cancellation due to optimal threshold Dec 26 16:35:27.826004603 (505h6m27.697087638s) [seg_pos={0 0}, upload=443230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:35:27.826078824 (505h6m27.697161869s) [upload=443230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:35:27.826085026 (505h6m27.697168071s) [upload=443230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:35:28.123831196 (505h6m27.994914242s) [upload=443230]: Done uploading segments... Dec 26 16:35:28.123855914 (505h6m27.994938959s) [upload=443230]: Upload finished. err: <nil> Duration: 30.002846635s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [........X.............XXX...X...XX.....XX..........X......XX] duration: 5.001165401s trace: https://jaeger.dev.storj.io/trace/6ca0cd450308b69c FAIL (2025-12-26 16:35:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001193705s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.868932967s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:35:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 6.868956411s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.........X.............XXX...X...XX.....XX..........X......X] duration: 5.001197733s trace: https://jaeger.dev.storj.io/trace/78ab13ad2eb4355 FAIL (2025-12-26 16:35:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001220948s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X..X...X.......XX..........X.........X..X...............X] duration: 22.190234319s trace: https://jaeger.dev.storj.io/trace/4bc9becdddcc2037 FAIL (2025-12-26 16:35:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443223]: Uploading piece... Dec 26 16:34:46.586854691 (505h5m46.457937736s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.587812465 (505h5m46.4588955s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.588277791 (505h5m46.459360826s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590149876 (505h5m46.461232911s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590674354 (505h5m46.461757399s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591254680 (505h5m46.462337715s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591632860 (505h5m46.462715895s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591778888 (505h5m46.462861923s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592107854 (505h5m46.463190889s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592306482 (505h5m46.463389517s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592535257 (505h5m46.463618302s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592867591 (505h5m46.463950636s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592905783 (505h5m46.463988828s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593325633 (505h5m46.464408678s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593417989 (505h5m46.464501034s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593721267 (505h5m46.464804302s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.594046005 (505h5m46.46512905s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:47.885134462 (505h5m47.756217497s) [upload=443223]: Next returned nil segment Dec 26 16:34:47.885151225 (505h5m47.75623426s) [upload=443223]: Waiting for error group managing segments... Dec 26 16:34:48.239296449 (505h5m48.110379494s) [noise=true , address=64.49.13.140:4505 , piece_id=FXVVLCPQXGJLYWDGHDOQECN3S4ZZXTEEF7HNVLQW33IUEIPYW6OA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465120747 (505h5m48.336203792s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465174059 (505h5m48.336257094s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.468068951 (505h5m48.339151986s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.481407861 (505h5m48.352490896s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.482107242 (505h5m48.353190277s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483127506 (505h5m48.354210551s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483731115 (505h5m48.35481415s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.639322688 (505h5m48.510405733s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.760495594 (505h5m48.631578639s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.763286658 (505h5m48.634369693s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.771028379 (505h5m48.642111414s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.341293323 (505h5m49.212376368s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446731015 (505h5m49.31781406s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446757516 (505h5m49.317840561s) [seg_pos={0 0}, upload=443223]: Segment reached optimal threshold of 14 pieces. Dec 26 16:34:49.446947537 (505h5m49.318030572s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.446954180 (505h5m49.318037216s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447148209 (505h5m49.318231244s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447154361 (505h5m49.318237396s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447201702 (505h5m49.318284767s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447232551 (505h5m49.318315606s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447285240 (505h5m49.318368275s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447291793 (505h5m49.318374838s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447309938 (505h5m49.318392973s) [seg_pos={0 0}, upload=443223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:34:49.447375853 (505h5m49.318458898s) [upload=443223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:34:49.447380402 (505h5m49.318463437s) [upload=443223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:34:49.756168298 (505h5m49.627251333s) [upload=443223]: Done uploading segments... Dec 26 16:34:49.756192103 (505h5m49.627275149s) [upload=443223]: Upload finished. err: <nil> Duration: 22.19027659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 10.366884807s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:34:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 10.366910808s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [............X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..] duration: 6.143446955s trace: https://jaeger.dev.storj.io/trace/1000e58d11c5593 FAIL (2025-12-26 16:34:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: QVBEA2BDFXXF5IQ5KPGVLCL3YW37WGD6KGROWSV26F3BG33WC46A) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: GBGWGNZNF6ZXRGPXZC6NNVB6OUJW4BJ74CRTQ2UFJN4RX2462AWQ) context deadline exceeded ======LOGS====== BatchRequestItem_ObjectBegin Dec 26 16:32:55.748472856 (505h3m55.619555891s) [upload=443196]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:32:55.990233411 (505h3m55.861316446s) [seg_pos={0 0}, upload=443196]: Begin upload segment... Dec 26 16:32:55.990279388 (505h3m55.861362423s) [seg_pos={0 0}, upload=443196]: Done begin upload segment. Dec 26 16:32:55.990318382 (505h3m55.861401417s) [noise=true , address=145.79.227.41:4087 , piece_id=PW35BAT23OD7M3FDPMPCXNNP3B5SKPPKRSXM2QJ7ULP7WKSZIM4Q, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.990326097 (505h3m55.861409132s) [noise=true , address=145.79.227.47:4266 , piece_id=5RYXQP3FJZID3RBGL4WFI3IBRJUNCSKGYJJNDV2X74XD5AK7UJYQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.990739674 (505h3m55.861822729s) [noise=true , address=145.79.227.42:4121 , piece_id=UCZXAO5YFCNBGXRT2XEDF2R5HUPOQ7RSZEQXLQ47XGQQNX7GN5ZQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.991159463 (505h3m55.862242509s) [noise=true , address=145.79.227.6:4259 , piece_id=34OGAWRQRAR4DAUNSMXKSBR6HIYI47ITCUXE2QNXNV5ONASH3W4A, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.991535961 (505h3m55.862619006s) [noise=true , address=145.79.227.31:4799 , piece_id=HSOHCLYHJMJYQQQDAT6JSPEJHRQBJ6MKQ44SFJ5Z6R2JEOV6DWUA, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.991938788 (505h3m55.863021823s) [noise=true , address=45.91.116.8:4045 , piece_id=HR5TEJZZR6DPQS2UXT6P3PSYBGAYSZKN6GHXA2PZL6BQZEQSAX5Q, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.992325534 (505h3m55.863408579s) [noise=true , address=145.79.227.51:4411 , piece_id=2NILIF67FCLWX66GJ2K2TBEDK5WMZRRSFWTOJJ7E3OQRLOMGJFPQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.992732580 (505h3m55.863815625s) [noise=true , address=145.79.227.49:4337 , piece_id=ZTTGLNTXF3MTD7KGCINBZJXY5XXIGOLFKLVQ2DT5K4UNRO5ERSAQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.994057483 (505h3m55.865140538s) [noise=true , address=145.79.227.50:4374 , piece_id=XBZNWUVPVIAFHRVTKCKI5DKRTWSTHCJXLKSMSJPOSJYMTJID34DA, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.994280787 (505h3m55.865363832s) [noise=true , address=145.79.227.36:4517 , piece_id=TFUWTSQJLIMMJYQKPJTXPLHLTCEQVPDRSQFU2CY5LXZJRAGOLPPQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.995192643 (505h3m55.866275679s) [noise=true , address=77.90.31.10:4518 , piece_id=NNDCPNYD6E7XGCGQQUJL2JVUWWFDZ5LDJ423EKRPUXQHEFYEKSJQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.996239617 (505h3m55.867322652s) [noise=true , address=77.90.31.15:4700 , piece_id=EGD5XSSWVOVCATQJHCNOQXRLD5K6FDDSIYUER6J7MC5H4LYRZ4GA, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.998518126 (505h3m55.869601171s) [noise=true , address=77.90.31.17:4773 , piece_id=GBGWGNZNF6ZXRGPXZC6NNVB6OUJW4BJ74CRTQ2UFJN4RX2462AWQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:56.000187415 (505h3m55.87127047s) [noise=true , address=77.90.31.13:4629 , piece_id=QVBEA2BDFXXF5IQ5KPGVLCL3YW37WGD6KGROWSV26F3BG33WC46A, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:56.001338356 (505h3m55.872421402s) [noise=true , address=145.79.227.38:4977 , piece_id=QSYMNKN5QE2HDV32SEHN457Y2ZNFDB3MY3WPDPUZYJ2SAZN3BDTQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:56.002218242 (505h3m55.873301297s) [noise=true , address=145.79.227.22:4511 , piece_id=OXZWIZUYMK3SZTWFUMRRKTSBAEGAC27QSOGNQD7MNTKCGYHDDO6Q, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:57.708956739 (505h3m57.580039774s) [upload=443196]: Next returned nil segment Dec 26 16:32:57.708968663 (505h3m57.580051698s) [upload=443196]: Waiting for error group managing segments... Dec 26 16:32:57.996082417 (505h3m57.867165452s) [noise=true , address=145.79.227.36:4517 , piece_id=TFUWTSQJLIMMJYQKPJTXPLHLTCEQVPDRSQFU2CY5LXZJRAGOLPPQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:57.996364233 (505h3m57.867447269s) [noise=true , address=145.79.227.31:4799 , piece_id=HSOHCLYHJMJYQQQDAT6JSPEJHRQBJ6MKQ44SFJ5Z6R2JEOV6DWUA, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:57.997930696 (505h3m57.869013731s) [noise=true , address=145.79.227.42:4121 , piece_id=UCZXAO5YFCNBGXRT2XEDF2R5HUPOQ7RSZEQXLQ47XGQQNX7GN5ZQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:57.998955959 (505h3m57.870039004s) [noise=true , address=145.79.227.50:4374 , piece_id=XBZNWUVPVIAFHRVTKCKI5DKRTWSTHCJXLKSMSJPOSJYMTJID34DA, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.002102520 (505h3m57.873185565s) [noise=true , address=145.79.227.49:4337 , piece_id=ZTTGLNTXF3MTD7KGCINBZJXY5XXIGOLFKLVQ2DT5K4UNRO5ERSAQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.002326246 (505h3m57.873409301s) [noise=true , address=145.79.227.38:4977 , piece_id=QSYMNKN5QE2HDV32SEHN457Y2ZNFDB3MY3WPDPUZYJ2SAZN3BDTQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.008062539 (505h3m57.879145584s) [noise=true , address=145.79.227.22:4511 , piece_id=OXZWIZUYMK3SZTWFUMRRKTSBAEGAC27QSOGNQD7MNTKCGYHDDO6Q, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.013517547 (505h3m57.884600582s) [noise=true , address=145.79.227.51:4411 , piece_id=2NILIF67FCLWX66GJ2K2TBEDK5WMZRRSFWTOJJ7E3OQRLOMGJFPQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.028055369 (505h3m57.899138405s) [noise=true , address=145.79.227.47:4266 , piece_id=5RYXQP3FJZID3RBGL4WFI3IBRJUNCSKGYJJNDV2X74XD5AK7UJYQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.046156557 (505h3m57.917239592s) [noise=true , address=145.79.227.6:4259 , piece_id=34OGAWRQRAR4DAUNSMXKSBR6HIYI47ITCUXE2QNXNV5ONASH3W4A, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.202158942 (505h3m58.073241977s) [noise=true , address=77.90.31.13:4629 , piece_id=QVBEA2BDFXXF5IQ5KPGVLCL3YW37WGD6KGROWSV26F3BG33WC46A, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.277974568 (505h3m58.149057613s) [noise=true , address=77.90.31.17:4773 , piece_id=GBGWGNZNF6ZXRGPXZC6NNVB6OUJW4BJ74CRTQ2UFJN4RX2462AWQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.376870194 (505h3m58.247953229s) [noise=true , address=77.90.31.15:4700 , piece_id=EGD5XSSWVOVCATQJHCNOQXRLD5K6FDDSIYUER6J7MC5H4LYRZ4GA, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.376896934 (505h3m58.247979979s) [seg_pos={0 0}, upload=443196]: Segment reached optimal threshold of 13 pieces. Dec 26 16:32:58.377175755 (505h3m58.2482588s) [noise=true , address=77.90.31.10:4518 , piece_id=NNDCPNYD6E7XGCGQQUJL2JVUWWFDZ5LDJ423EKRPUXQHEFYEKSJQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:49172->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:49172->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49172->77.90.31.10:4518: use of closed network connection Dec 26 16:32:58.377183550 (505h3m58.248266586s) [noise=true , address=77.90.31.10:4518 , piece_id=NNDCPNYD6E7XGCGQQUJL2JVUWWFDZ5LDJ423EKRPUXQHEFYEKSJQ, seg_pos={0 0}, upload=443196]: Long tail cancellation due to optimal threshold Dec 26 16:32:58.377471599 (505h3m58.248554644s) [noise=true , address=45.91.116.8:4045 , piece_id=HR5TEJZZR6DPQS2UXT6P3PSYBGAYSZKN6GHXA2PZL6BQZEQSAX5Q, seg_pos={0 0}, upload=443196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:47566->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:47566->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47566->45.91.116.8:4045: use of closed network connection Dec 26 16:32:58.377480505 (505h3m58.24856354s) [noise=true , address=45.91.116.8:4045 , piece_id=HR5TEJZZR6DPQS2UXT6P3PSYBGAYSZKN6GHXA2PZL6BQZEQSAX5Q, seg_pos={0 0}, upload=443196]: Long tail cancellation due to optimal threshold Dec 26 16:32:58.377585515 (505h3m58.24866856s) [noise=true , address=145.79.227.41:4087 , piece_id=PW35BAT23OD7M3FDPMPCXNNP3B5SKPPKRSXM2QJ7ULP7WKSZIM4Q, seg_pos={0 0}, upload=443196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:50806->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:50806->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50806->145.79.227.41:4087: use of closed network connection Dec 26 16:32:58.377594232 (505h3m58.248677267s) [noise=true , address=145.79.227.41:4087 , piece_id=PW35BAT23OD7M3FDPMPCXNNP3B5SKPPKRSXM2QJ7ULP7WKSZIM4Q, seg_pos={0 0}, upload=443196]: Long tail cancellation due to optimal threshold Dec 26 16:32:58.377612497 (505h3m58.248695542s) [seg_pos={0 0}, upload=443196]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:32:58.377694032 (505h3m58.248777067s) [upload=443196]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:32:58.377699973 (505h3m58.248783018s) [upload=443196]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:32:58.669645940 (505h3m58.540728975s) [upload=443196]: Done uploading segments... Dec 26 16:32:58.669665266 (505h3m58.540748301s) [upload=443196]: Upload finished. err: <nil> Duration:* 6.143471442s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X..X...X.......XX..........X.........X..X...............] duration: 47.362563948s trace: https://jaeger.dev.storj.io/trace/3e3dfe0984f19d4e FAIL (2025-12-26 16:34:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.466998847 (504h51m15.338081892s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467028614 (504h51m15.338111659s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467359594 (504h51m15.338442639s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467908379 (504h51m15.338991424s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.468520054 (504h51m15.339603089s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469055293 (504h51m15.340138328s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469497605 (504h51m15.34058064s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469973642 (504h51m15.341056687s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.470454397 (504h51m15.341537443s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471043630 (504h51m15.342126695s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471470552 (504h51m15.342553587s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471676735 (504h51m15.34275977s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472062900 (504h51m15.343145935s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472369224 (504h51m15.343452259s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472823849 (504h51m15.343906884s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:16.743410939 (504h51m16.614493984s) [upload=443005]: Next returned nil segment Dec 26 16:20:16.743428862 (504h51m16.614511907s) [upload=443005]: Waiting for error group managing segments... Dec 26 16:20:17.091086984 (504h51m16.962170029s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.389229539 (504h51m17.260312594s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.464846438 (504h51m17.335929483s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.542075705 (504h51m17.41315875s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.623422163 (504h51m17.494505208s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.665409184 (504h51m17.536492219s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.301995626 (504h51m18.173078671s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.311693532 (504h51m18.182776567s) [noise=true , address=145.79.227.34:4915 , piece_id=7ZBWNPUF54BEGIXTDRDI3J7HNEZ63MEXKUBMRZW6PFWWNCEAH3AA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.340201235 (504h51m18.21128428s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.379076200 (504h51m18.250159245s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.474362322 (504h51m18.345445367s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.555133063 (504h51m18.426216108s) [noise=true , address=77.90.31.17:4791 , piece_id=QFHBBE4B5DWZU7HTAARXW655MKX4YPKPOXVM77CH5DCPSLZ7UGNA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.707673447 (504h51m18.578756492s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734654664 (504h51m18.605737709s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734678970 (504h51m18.605762005s) [seg_pos={0 0}, upload=443005]: Segment reached optimal threshold of 14 pieces. Dec 26 16:20:18.734942853 (504h51m18.606025888s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.734952842 (504h51m18.606035887s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735555389 (504h51m18.606638434s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection Dec 26 16:20:18.735566701 (504h51m18.606649746s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735681349 (504h51m18.606764394s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735699113 (504h51m18.606782148s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735826605 (504h51m18.60690968s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735837737 (504h51m18.606920782s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735908392 (504h51m18.606991447s) [seg_pos={0 0}, upload=443005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:20:18.736024182 (504h51m18.607107227s) [upload=443005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:20:18.736031275 (504h51m18.60711431s) [upload=443005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:20:19.031573339 (504h51m18.902656384s) [upload=443005]: Done uploading segments... Dec 26 16:20:19.031602895 (504h51m18.90268594s) [upload=443005]: Upload finished. err: <nil> Duration: 47.36258039s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 9.851905597s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:34:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 9.851929984s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [X.........X.............XXX...X...XX.....XX..........X......] duration: 5.001232108s trace: https://jaeger.dev.storj.io/trace/4d8eedf07808cbf FAIL (2025-12-26 16:34:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001256956s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.............X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX.] duration: 6.485521496s trace: https://jaeger.dev.storj.io/trace/1000e58d11c5593 FAIL (2025-12-26 16:34:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: QVBEA2BDFXXF5IQ5KPGVLCL3YW37WGD6KGROWSV26F3BG33WC46A) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: GBGWGNZNF6ZXRGPXZC6NNVB6OUJW4BJ74CRTQ2UFJN4RX2462AWQ) context deadline exceeded ======LOGS====== BatchRequestItem_ObjectBegin Dec 26 16:32:55.748472856 (505h3m55.619555891s) [upload=443196]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:32:55.990233411 (505h3m55.861316446s) [seg_pos={0 0}, upload=443196]: Begin upload segment... Dec 26 16:32:55.990279388 (505h3m55.861362423s) [seg_pos={0 0}, upload=443196]: Done begin upload segment. Dec 26 16:32:55.990318382 (505h3m55.861401417s) [noise=true , address=145.79.227.41:4087 , piece_id=PW35BAT23OD7M3FDPMPCXNNP3B5SKPPKRSXM2QJ7ULP7WKSZIM4Q, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.990326097 (505h3m55.861409132s) [noise=true , address=145.79.227.47:4266 , piece_id=5RYXQP3FJZID3RBGL4WFI3IBRJUNCSKGYJJNDV2X74XD5AK7UJYQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.990739674 (505h3m55.861822729s) [noise=true , address=145.79.227.42:4121 , piece_id=UCZXAO5YFCNBGXRT2XEDF2R5HUPOQ7RSZEQXLQ47XGQQNX7GN5ZQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.991159463 (505h3m55.862242509s) [noise=true , address=145.79.227.6:4259 , piece_id=34OGAWRQRAR4DAUNSMXKSBR6HIYI47ITCUXE2QNXNV5ONASH3W4A, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.991535961 (505h3m55.862619006s) [noise=true , address=145.79.227.31:4799 , piece_id=HSOHCLYHJMJYQQQDAT6JSPEJHRQBJ6MKQ44SFJ5Z6R2JEOV6DWUA, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.991938788 (505h3m55.863021823s) [noise=true , address=45.91.116.8:4045 , piece_id=HR5TEJZZR6DPQS2UXT6P3PSYBGAYSZKN6GHXA2PZL6BQZEQSAX5Q, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.992325534 (505h3m55.863408579s) [noise=true , address=145.79.227.51:4411 , piece_id=2NILIF67FCLWX66GJ2K2TBEDK5WMZRRSFWTOJJ7E3OQRLOMGJFPQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.992732580 (505h3m55.863815625s) [noise=true , address=145.79.227.49:4337 , piece_id=ZTTGLNTXF3MTD7KGCINBZJXY5XXIGOLFKLVQ2DT5K4UNRO5ERSAQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.994057483 (505h3m55.865140538s) [noise=true , address=145.79.227.50:4374 , piece_id=XBZNWUVPVIAFHRVTKCKI5DKRTWSTHCJXLKSMSJPOSJYMTJID34DA, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.994280787 (505h3m55.865363832s) [noise=true , address=145.79.227.36:4517 , piece_id=TFUWTSQJLIMMJYQKPJTXPLHLTCEQVPDRSQFU2CY5LXZJRAGOLPPQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.995192643 (505h3m55.866275679s) [noise=true , address=77.90.31.10:4518 , piece_id=NNDCPNYD6E7XGCGQQUJL2JVUWWFDZ5LDJ423EKRPUXQHEFYEKSJQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.996239617 (505h3m55.867322652s) [noise=true , address=77.90.31.15:4700 , piece_id=EGD5XSSWVOVCATQJHCNOQXRLD5K6FDDSIYUER6J7MC5H4LYRZ4GA, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.998518126 (505h3m55.869601171s) [noise=true , address=77.90.31.17:4773 , piece_id=GBGWGNZNF6ZXRGPXZC6NNVB6OUJW4BJ74CRTQ2UFJN4RX2462AWQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:56.000187415 (505h3m55.87127047s) [noise=true , address=77.90.31.13:4629 , piece_id=QVBEA2BDFXXF5IQ5KPGVLCL3YW37WGD6KGROWSV26F3BG33WC46A, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:56.001338356 (505h3m55.872421402s) [noise=true , address=145.79.227.38:4977 , piece_id=QSYMNKN5QE2HDV32SEHN457Y2ZNFDB3MY3WPDPUZYJ2SAZN3BDTQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:56.002218242 (505h3m55.873301297s) [noise=true , address=145.79.227.22:4511 , piece_id=OXZWIZUYMK3SZTWFUMRRKTSBAEGAC27QSOGNQD7MNTKCGYHDDO6Q, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:57.708956739 (505h3m57.580039774s) [upload=443196]: Next returned nil segment Dec 26 16:32:57.708968663 (505h3m57.580051698s) [upload=443196]: Waiting for error group managing segments... Dec 26 16:32:57.996082417 (505h3m57.867165452s) [noise=true , address=145.79.227.36:4517 , piece_id=TFUWTSQJLIMMJYQKPJTXPLHLTCEQVPDRSQFU2CY5LXZJRAGOLPPQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:57.996364233 (505h3m57.867447269s) [noise=true , address=145.79.227.31:4799 , piece_id=HSOHCLYHJMJYQQQDAT6JSPEJHRQBJ6MKQ44SFJ5Z6R2JEOV6DWUA, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:57.997930696 (505h3m57.869013731s) [noise=true , address=145.79.227.42:4121 , piece_id=UCZXAO5YFCNBGXRT2XEDF2R5HUPOQ7RSZEQXLQ47XGQQNX7GN5ZQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:57.998955959 (505h3m57.870039004s) [noise=true , address=145.79.227.50:4374 , piece_id=XBZNWUVPVIAFHRVTKCKI5DKRTWSTHCJXLKSMSJPOSJYMTJID34DA, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.002102520 (505h3m57.873185565s) [noise=true , address=145.79.227.49:4337 , piece_id=ZTTGLNTXF3MTD7KGCINBZJXY5XXIGOLFKLVQ2DT5K4UNRO5ERSAQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.002326246 (505h3m57.873409301s) [noise=true , address=145.79.227.38:4977 , piece_id=QSYMNKN5QE2HDV32SEHN457Y2ZNFDB3MY3WPDPUZYJ2SAZN3BDTQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.008062539 (505h3m57.879145584s) [noise=true , address=145.79.227.22:4511 , piece_id=OXZWIZUYMK3SZTWFUMRRKTSBAEGAC27QSOGNQD7MNTKCGYHDDO6Q, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.013517547 (505h3m57.884600582s) [noise=true , address=145.79.227.51:4411 , piece_id=2NILIF67FCLWX66GJ2K2TBEDK5WMZRRSFWTOJJ7E3OQRLOMGJFPQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.028055369 (505h3m57.899138405s) [noise=true , address=145.79.227.47:4266 , piece_id=5RYXQP3FJZID3RBGL4WFI3IBRJUNCSKGYJJNDV2X74XD5AK7UJYQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.046156557 (505h3m57.917239592s) [noise=true , address=145.79.227.6:4259 , piece_id=34OGAWRQRAR4DAUNSMXKSBR6HIYI47ITCUXE2QNXNV5ONASH3W4A, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.202158942 (505h3m58.073241977s) [noise=true , address=77.90.31.13:4629 , piece_id=QVBEA2BDFXXF5IQ5KPGVLCL3YW37WGD6KGROWSV26F3BG33WC46A, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.277974568 (505h3m58.149057613s) [noise=true , address=77.90.31.17:4773 , piece_id=GBGWGNZNF6ZXRGPXZC6NNVB6OUJW4BJ74CRTQ2UFJN4RX2462AWQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.376870194 (505h3m58.247953229s) [noise=true , address=77.90.31.15:4700 , piece_id=EGD5XSSWVOVCATQJHCNOQXRLD5K6FDDSIYUER6J7MC5H4LYRZ4GA, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.376896934 (505h3m58.247979979s) [seg_pos={0 0}, upload=443196]: Segment reached optimal threshold of 13 pieces. Dec 26 16:32:58.377175755 (505h3m58.2482588s) [noise=true , address=77.90.31.10:4518 , piece_id=NNDCPNYD6E7XGCGQQUJL2JVUWWFDZ5LDJ423EKRPUXQHEFYEKSJQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:49172->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:49172->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49172->77.90.31.10:4518: use of closed network connection Dec 26 16:32:58.377183550 (505h3m58.248266586s) [noise=true , address=77.90.31.10:4518 , piece_id=NNDCPNYD6E7XGCGQQUJL2JVUWWFDZ5LDJ423EKRPUXQHEFYEKSJQ, seg_pos={0 0}, upload=443196]: Long tail cancellation due to optimal threshold Dec 26 16:32:58.377471599 (505h3m58.248554644s) [noise=true , address=45.91.116.8:4045 , piece_id=HR5TEJZZR6DPQS2UXT6P3PSYBGAYSZKN6GHXA2PZL6BQZEQSAX5Q, seg_pos={0 0}, upload=443196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:47566->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:47566->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47566->45.91.116.8:4045: use of closed network connection Dec 26 16:32:58.377480505 (505h3m58.24856354s) [noise=true , address=45.91.116.8:4045 , piece_id=HR5TEJZZR6DPQS2UXT6P3PSYBGAYSZKN6GHXA2PZL6BQZEQSAX5Q, seg_pos={0 0}, upload=443196]: Long tail cancellation due to optimal threshold Dec 26 16:32:58.377585515 (505h3m58.24866856s) [noise=true , address=145.79.227.41:4087 , piece_id=PW35BAT23OD7M3FDPMPCXNNP3B5SKPPKRSXM2QJ7ULP7WKSZIM4Q, seg_pos={0 0}, upload=443196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:50806->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:50806->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50806->145.79.227.41:4087: use of closed network connection Dec 26 16:32:58.377594232 (505h3m58.248677267s) [noise=true , address=145.79.227.41:4087 , piece_id=PW35BAT23OD7M3FDPMPCXNNP3B5SKPPKRSXM2QJ7ULP7WKSZIM4Q, seg_pos={0 0}, upload=443196]: Long tail cancellation due to optimal threshold Dec 26 16:32:58.377612497 (505h3m58.248695542s) [seg_pos={0 0}, upload=443196]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:32:58.377694032 (505h3m58.248777067s) [upload=443196]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:32:58.377699973 (505h3m58.248783018s) [upload=443196]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:32:58.669645940 (505h3m58.540728975s) [upload=443196]: Done uploading segments... Dec 26 16:32:58.669665266 (505h3m58.540748301s) [upload=443196]: Upload finished. err: <nil> Duration:* 6.485548256s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [X....X..X...X.......XX..........X.........X..X..............] duration: 43.660160035s trace: https://jaeger.dev.storj.io/trace/3e3dfe0984f19d4e FAIL (2025-12-26 16:33:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.466998847 (504h51m15.338081892s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467028614 (504h51m15.338111659s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467359594 (504h51m15.338442639s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467908379 (504h51m15.338991424s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.468520054 (504h51m15.339603089s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469055293 (504h51m15.340138328s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469497605 (504h51m15.34058064s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469973642 (504h51m15.341056687s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.470454397 (504h51m15.341537443s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471043630 (504h51m15.342126695s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471470552 (504h51m15.342553587s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471676735 (504h51m15.34275977s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472062900 (504h51m15.343145935s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472369224 (504h51m15.343452259s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472823849 (504h51m15.343906884s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:16.743410939 (504h51m16.614493984s) [upload=443005]: Next returned nil segment Dec 26 16:20:16.743428862 (504h51m16.614511907s) [upload=443005]: Waiting for error group managing segments... Dec 26 16:20:17.091086984 (504h51m16.962170029s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.389229539 (504h51m17.260312594s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.464846438 (504h51m17.335929483s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.542075705 (504h51m17.41315875s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.623422163 (504h51m17.494505208s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.665409184 (504h51m17.536492219s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.301995626 (504h51m18.173078671s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.311693532 (504h51m18.182776567s) [noise=true , address=145.79.227.34:4915 , piece_id=7ZBWNPUF54BEGIXTDRDI3J7HNEZ63MEXKUBMRZW6PFWWNCEAH3AA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.340201235 (504h51m18.21128428s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.379076200 (504h51m18.250159245s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.474362322 (504h51m18.345445367s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.555133063 (504h51m18.426216108s) [noise=true , address=77.90.31.17:4791 , piece_id=QFHBBE4B5DWZU7HTAARXW655MKX4YPKPOXVM77CH5DCPSLZ7UGNA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.707673447 (504h51m18.578756492s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734654664 (504h51m18.605737709s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734678970 (504h51m18.605762005s) [seg_pos={0 0}, upload=443005]: Segment reached optimal threshold of 14 pieces. Dec 26 16:20:18.734942853 (504h51m18.606025888s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.734952842 (504h51m18.606035887s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735555389 (504h51m18.606638434s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection Dec 26 16:20:18.735566701 (504h51m18.606649746s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735681349 (504h51m18.606764394s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735699113 (504h51m18.606782148s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735826605 (504h51m18.60690968s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735837737 (504h51m18.606920782s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735908392 (504h51m18.606991447s) [seg_pos={0 0}, upload=443005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:20:18.736024182 (504h51m18.607107227s) [upload=443005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:20:18.736031275 (504h51m18.60711431s) [upload=443005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:20:19.031573339 (504h51m18.902656384s) [upload=443005]: Done uploading segments... Dec 26 16:20:19.031602895 (504h51m18.90268594s) [upload=443005]: Upload finished. err: <nil> Duration: 43.660186286s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 8.374052829s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:33:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 8.374086502s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..............X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX] duration: 30.002745323s trace: https://jaeger.dev.storj.io/trace/1000e58d11c5593 FAIL (2025-12-26 16:33:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: QVBEA2BDFXXF5IQ5KPGVLCL3YW37WGD6KGROWSV26F3BG33WC46A) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: GBGWGNZNF6ZXRGPXZC6NNVB6OUJW4BJ74CRTQ2UFJN4RX2462AWQ) context deadline exceeded ======LOGS====== BatchRequestItem_ObjectBegin Dec 26 16:32:55.748472856 (505h3m55.619555891s) [upload=443196]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:32:55.990233411 (505h3m55.861316446s) [seg_pos={0 0}, upload=443196]: Begin upload segment... Dec 26 16:32:55.990279388 (505h3m55.861362423s) [seg_pos={0 0}, upload=443196]: Done begin upload segment. Dec 26 16:32:55.990318382 (505h3m55.861401417s) [noise=true , address=145.79.227.41:4087 , piece_id=PW35BAT23OD7M3FDPMPCXNNP3B5SKPPKRSXM2QJ7ULP7WKSZIM4Q, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.990326097 (505h3m55.861409132s) [noise=true , address=145.79.227.47:4266 , piece_id=5RYXQP3FJZID3RBGL4WFI3IBRJUNCSKGYJJNDV2X74XD5AK7UJYQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.990739674 (505h3m55.861822729s) [noise=true , address=145.79.227.42:4121 , piece_id=UCZXAO5YFCNBGXRT2XEDF2R5HUPOQ7RSZEQXLQ47XGQQNX7GN5ZQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.991159463 (505h3m55.862242509s) [noise=true , address=145.79.227.6:4259 , piece_id=34OGAWRQRAR4DAUNSMXKSBR6HIYI47ITCUXE2QNXNV5ONASH3W4A, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.991535961 (505h3m55.862619006s) [noise=true , address=145.79.227.31:4799 , piece_id=HSOHCLYHJMJYQQQDAT6JSPEJHRQBJ6MKQ44SFJ5Z6R2JEOV6DWUA, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.991938788 (505h3m55.863021823s) [noise=true , address=45.91.116.8:4045 , piece_id=HR5TEJZZR6DPQS2UXT6P3PSYBGAYSZKN6GHXA2PZL6BQZEQSAX5Q, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.992325534 (505h3m55.863408579s) [noise=true , address=145.79.227.51:4411 , piece_id=2NILIF67FCLWX66GJ2K2TBEDK5WMZRRSFWTOJJ7E3OQRLOMGJFPQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.992732580 (505h3m55.863815625s) [noise=true , address=145.79.227.49:4337 , piece_id=ZTTGLNTXF3MTD7KGCINBZJXY5XXIGOLFKLVQ2DT5K4UNRO5ERSAQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.994057483 (505h3m55.865140538s) [noise=true , address=145.79.227.50:4374 , piece_id=XBZNWUVPVIAFHRVTKCKI5DKRTWSTHCJXLKSMSJPOSJYMTJID34DA, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.994280787 (505h3m55.865363832s) [noise=true , address=145.79.227.36:4517 , piece_id=TFUWTSQJLIMMJYQKPJTXPLHLTCEQVPDRSQFU2CY5LXZJRAGOLPPQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.995192643 (505h3m55.866275679s) [noise=true , address=77.90.31.10:4518 , piece_id=NNDCPNYD6E7XGCGQQUJL2JVUWWFDZ5LDJ423EKRPUXQHEFYEKSJQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.996239617 (505h3m55.867322652s) [noise=true , address=77.90.31.15:4700 , piece_id=EGD5XSSWVOVCATQJHCNOQXRLD5K6FDDSIYUER6J7MC5H4LYRZ4GA, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.998518126 (505h3m55.869601171s) [noise=true , address=77.90.31.17:4773 , piece_id=GBGWGNZNF6ZXRGPXZC6NNVB6OUJW4BJ74CRTQ2UFJN4RX2462AWQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:56.000187415 (505h3m55.87127047s) [noise=true , address=77.90.31.13:4629 , piece_id=QVBEA2BDFXXF5IQ5KPGVLCL3YW37WGD6KGROWSV26F3BG33WC46A, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:56.001338356 (505h3m55.872421402s) [noise=true , address=145.79.227.38:4977 , piece_id=QSYMNKN5QE2HDV32SEHN457Y2ZNFDB3MY3WPDPUZYJ2SAZN3BDTQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:56.002218242 (505h3m55.873301297s) [noise=true , address=145.79.227.22:4511 , piece_id=OXZWIZUYMK3SZTWFUMRRKTSBAEGAC27QSOGNQD7MNTKCGYHDDO6Q, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:57.708956739 (505h3m57.580039774s) [upload=443196]: Next returned nil segment Dec 26 16:32:57.708968663 (505h3m57.580051698s) [upload=443196]: Waiting for error group managing segments... Dec 26 16:32:57.996082417 (505h3m57.867165452s) [noise=true , address=145.79.227.36:4517 , piece_id=TFUWTSQJLIMMJYQKPJTXPLHLTCEQVPDRSQFU2CY5LXZJRAGOLPPQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:57.996364233 (505h3m57.867447269s) [noise=true , address=145.79.227.31:4799 , piece_id=HSOHCLYHJMJYQQQDAT6JSPEJHRQBJ6MKQ44SFJ5Z6R2JEOV6DWUA, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:57.997930696 (505h3m57.869013731s) [noise=true , address=145.79.227.42:4121 , piece_id=UCZXAO5YFCNBGXRT2XEDF2R5HUPOQ7RSZEQXLQ47XGQQNX7GN5ZQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:57.998955959 (505h3m57.870039004s) [noise=true , address=145.79.227.50:4374 , piece_id=XBZNWUVPVIAFHRVTKCKI5DKRTWSTHCJXLKSMSJPOSJYMTJID34DA, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.002102520 (505h3m57.873185565s) [noise=true , address=145.79.227.49:4337 , piece_id=ZTTGLNTXF3MTD7KGCINBZJXY5XXIGOLFKLVQ2DT5K4UNRO5ERSAQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.002326246 (505h3m57.873409301s) [noise=true , address=145.79.227.38:4977 , piece_id=QSYMNKN5QE2HDV32SEHN457Y2ZNFDB3MY3WPDPUZYJ2SAZN3BDTQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.008062539 (505h3m57.879145584s) [noise=true , address=145.79.227.22:4511 , piece_id=OXZWIZUYMK3SZTWFUMRRKTSBAEGAC27QSOGNQD7MNTKCGYHDDO6Q, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.013517547 (505h3m57.884600582s) [noise=true , address=145.79.227.51:4411 , piece_id=2NILIF67FCLWX66GJ2K2TBEDK5WMZRRSFWTOJJ7E3OQRLOMGJFPQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.028055369 (505h3m57.899138405s) [noise=true , address=145.79.227.47:4266 , piece_id=5RYXQP3FJZID3RBGL4WFI3IBRJUNCSKGYJJNDV2X74XD5AK7UJYQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.046156557 (505h3m57.917239592s) [noise=true , address=145.79.227.6:4259 , piece_id=34OGAWRQRAR4DAUNSMXKSBR6HIYI47ITCUXE2QNXNV5ONASH3W4A, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.202158942 (505h3m58.073241977s) [noise=true , address=77.90.31.13:4629 , piece_id=QVBEA2BDFXXF5IQ5KPGVLCL3YW37WGD6KGROWSV26F3BG33WC46A, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.277974568 (505h3m58.149057613s) [noise=true , address=77.90.31.17:4773 , piece_id=GBGWGNZNF6ZXRGPXZC6NNVB6OUJW4BJ74CRTQ2UFJN4RX2462AWQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.376870194 (505h3m58.247953229s) [noise=true , address=77.90.31.15:4700 , piece_id=EGD5XSSWVOVCATQJHCNOQXRLD5K6FDDSIYUER6J7MC5H4LYRZ4GA, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.376896934 (505h3m58.247979979s) [seg_pos={0 0}, upload=443196]: Segment reached optimal threshold of 13 pieces. Dec 26 16:32:58.377175755 (505h3m58.2482588s) [noise=true , address=77.90.31.10:4518 , piece_id=NNDCPNYD6E7XGCGQQUJL2JVUWWFDZ5LDJ423EKRPUXQHEFYEKSJQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:49172->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:49172->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49172->77.90.31.10:4518: use of closed network connection Dec 26 16:32:58.377183550 (505h3m58.248266586s) [noise=true , address=77.90.31.10:4518 , piece_id=NNDCPNYD6E7XGCGQQUJL2JVUWWFDZ5LDJ423EKRPUXQHEFYEKSJQ, seg_pos={0 0}, upload=443196]: Long tail cancellation due to optimal threshold Dec 26 16:32:58.377471599 (505h3m58.248554644s) [noise=true , address=45.91.116.8:4045 , piece_id=HR5TEJZZR6DPQS2UXT6P3PSYBGAYSZKN6GHXA2PZL6BQZEQSAX5Q, seg_pos={0 0}, upload=443196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:47566->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:47566->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47566->45.91.116.8:4045: use of closed network connection Dec 26 16:32:58.377480505 (505h3m58.24856354s) [noise=true , address=45.91.116.8:4045 , piece_id=HR5TEJZZR6DPQS2UXT6P3PSYBGAYSZKN6GHXA2PZL6BQZEQSAX5Q, seg_pos={0 0}, upload=443196]: Long tail cancellation due to optimal threshold Dec 26 16:32:58.377585515 (505h3m58.24866856s) [noise=true , address=145.79.227.41:4087 , piece_id=PW35BAT23OD7M3FDPMPCXNNP3B5SKPPKRSXM2QJ7ULP7WKSZIM4Q, seg_pos={0 0}, upload=443196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:50806->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:50806->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50806->145.79.227.41:4087: use of closed network connection Dec 26 16:32:58.377594232 (505h3m58.248677267s) [noise=true , address=145.79.227.41:4087 , piece_id=PW35BAT23OD7M3FDPMPCXNNP3B5SKPPKRSXM2QJ7ULP7WKSZIM4Q, seg_pos={0 0}, upload=443196]: Long tail cancellation due to optimal threshold Dec 26 16:32:58.377612497 (505h3m58.248695542s) [seg_pos={0 0}, upload=443196]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:32:58.377694032 (505h3m58.248777067s) [upload=443196]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:32:58.377699973 (505h3m58.248783018s) [upload=443196]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:32:58.669645940 (505h3m58.540728975s) [upload=443196]: Done uploading segments... Dec 26 16:32:58.669665266 (505h3m58.540748301s) [upload=443196]: Upload finished. err: <nil> Duration:* 30.002768657s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [XX.........X.............XXX...X...XX.....XX..........X.....] duration: 5.000911488s trace: https://jaeger.dev.storj.io/trace/4d8eedf07808cbf FAIL (2025-12-26 16:33:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000969399s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X....X..X...X.......XX..........X.........X..X.............] duration: 51.899741497s trace: https://jaeger.dev.storj.io/trace/3e3dfe0984f19d4e FAIL (2025-12-26 16:33:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.466998847 (504h51m15.338081892s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467028614 (504h51m15.338111659s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467359594 (504h51m15.338442639s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467908379 (504h51m15.338991424s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.468520054 (504h51m15.339603089s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469055293 (504h51m15.340138328s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469497605 (504h51m15.34058064s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469973642 (504h51m15.341056687s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.470454397 (504h51m15.341537443s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471043630 (504h51m15.342126695s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471470552 (504h51m15.342553587s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471676735 (504h51m15.34275977s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472062900 (504h51m15.343145935s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472369224 (504h51m15.343452259s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472823849 (504h51m15.343906884s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:16.743410939 (504h51m16.614493984s) [upload=443005]: Next returned nil segment Dec 26 16:20:16.743428862 (504h51m16.614511907s) [upload=443005]: Waiting for error group managing segments... Dec 26 16:20:17.091086984 (504h51m16.962170029s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.389229539 (504h51m17.260312594s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.464846438 (504h51m17.335929483s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.542075705 (504h51m17.41315875s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.623422163 (504h51m17.494505208s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.665409184 (504h51m17.536492219s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.301995626 (504h51m18.173078671s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.311693532 (504h51m18.182776567s) [noise=true , address=145.79.227.34:4915 , piece_id=7ZBWNPUF54BEGIXTDRDI3J7HNEZ63MEXKUBMRZW6PFWWNCEAH3AA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.340201235 (504h51m18.21128428s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.379076200 (504h51m18.250159245s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.474362322 (504h51m18.345445367s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.555133063 (504h51m18.426216108s) [noise=true , address=77.90.31.17:4791 , piece_id=QFHBBE4B5DWZU7HTAARXW655MKX4YPKPOXVM77CH5DCPSLZ7UGNA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.707673447 (504h51m18.578756492s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734654664 (504h51m18.605737709s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734678970 (504h51m18.605762005s) [seg_pos={0 0}, upload=443005]: Segment reached optimal threshold of 14 pieces. Dec 26 16:20:18.734942853 (504h51m18.606025888s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.734952842 (504h51m18.606035887s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735555389 (504h51m18.606638434s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection Dec 26 16:20:18.735566701 (504h51m18.606649746s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735681349 (504h51m18.606764394s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735699113 (504h51m18.606782148s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735826605 (504h51m18.60690968s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735837737 (504h51m18.606920782s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735908392 (504h51m18.606991447s) [seg_pos={0 0}, upload=443005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:20:18.736024182 (504h51m18.607107227s) [upload=443005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:20:18.736031275 (504h51m18.60711431s) [upload=443005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:20:19.031573339 (504h51m18.902656384s) [upload=443005]: Done uploading segments... Dec 26 16:20:19.031602895 (504h51m18.90268594s) [upload=443005]: Upload finished. err: <nil> Duration: 51.899766143s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.XX.........X.............XXX...X...XX.....XX..........X....] duration: 5.001128362s trace: https://jaeger.dev.storj.io/trace/4d8eedf07808cbf FAIL (2025-12-26 16:32:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.0011491s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)