Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 7.69404768s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:38:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 7.694078459s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.....X.............XXX...X...XX.....XX..........X......XXX..] duration: 5.000690287s trace: https://jaeger.dev.storj.io/trace/5b9c7e160a55f0bd FAIL (2025-12-26 16:38:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000719061s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [X..X...X.......XX..........X.........X..X...............X...] duration: 54.569939901s trace: https://jaeger.dev.storj.io/trace/4bc9becdddcc2037 FAIL (2025-12-26 16:38:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443223]: Uploading piece... Dec 26 16:34:46.586854691 (505h5m46.457937736s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.587812465 (505h5m46.4588955s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.588277791 (505h5m46.459360826s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590149876 (505h5m46.461232911s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590674354 (505h5m46.461757399s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591254680 (505h5m46.462337715s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591632860 (505h5m46.462715895s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591778888 (505h5m46.462861923s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592107854 (505h5m46.463190889s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592306482 (505h5m46.463389517s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592535257 (505h5m46.463618302s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592867591 (505h5m46.463950636s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592905783 (505h5m46.463988828s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593325633 (505h5m46.464408678s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593417989 (505h5m46.464501034s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593721267 (505h5m46.464804302s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.594046005 (505h5m46.46512905s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:47.885134462 (505h5m47.756217497s) [upload=443223]: Next returned nil segment Dec 26 16:34:47.885151225 (505h5m47.75623426s) [upload=443223]: Waiting for error group managing segments... Dec 26 16:34:48.239296449 (505h5m48.110379494s) [noise=true , address=64.49.13.140:4505 , piece_id=FXVVLCPQXGJLYWDGHDOQECN3S4ZZXTEEF7HNVLQW33IUEIPYW6OA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465120747 (505h5m48.336203792s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465174059 (505h5m48.336257094s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.468068951 (505h5m48.339151986s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.481407861 (505h5m48.352490896s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.482107242 (505h5m48.353190277s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483127506 (505h5m48.354210551s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483731115 (505h5m48.35481415s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.639322688 (505h5m48.510405733s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.760495594 (505h5m48.631578639s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.763286658 (505h5m48.634369693s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.771028379 (505h5m48.642111414s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.341293323 (505h5m49.212376368s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446731015 (505h5m49.31781406s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446757516 (505h5m49.317840561s) [seg_pos={0 0}, upload=443223]: Segment reached optimal threshold of 14 pieces. Dec 26 16:34:49.446947537 (505h5m49.318030572s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.446954180 (505h5m49.318037216s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447148209 (505h5m49.318231244s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447154361 (505h5m49.318237396s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447201702 (505h5m49.318284767s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447232551 (505h5m49.318315606s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447285240 (505h5m49.318368275s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447291793 (505h5m49.318374838s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447309938 (505h5m49.318392973s) [seg_pos={0 0}, upload=443223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:34:49.447375853 (505h5m49.318458898s) [upload=443223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:34:49.447380402 (505h5m49.318463437s) [upload=443223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:34:49.756168298 (505h5m49.627251333s) [upload=443223]: Done uploading segments... Dec 26 16:34:49.756192103 (505h5m49.627275149s) [upload=443223]: Upload finished. err: <nil> Duration: 54.569958016s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [........X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X..X] duration: 30.00317459s trace: https://jaeger.dev.storj.io/trace/3080d1fa188efb2e FAIL (2025-12-26 16:37:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: DIKBZ34FZCFLGFBHJJDX5BBZYNL2FTHNG3V6TSDCBPN5TXHVZMRQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: XINQ4KY3TRQ675HEJRVUQXMHMCP4EJWKDMIIKXDOXSMQUQXAPJYA) context deadline exceeded ======LOGS====== Dec 26 16:37:29.297178880 (505h8m29.168261925s) [upload=443263]: Starting upload Dec 26 16:37:29.297263381 (505h8m29.168346426s) [upload=443263]: Done starting upload Dec 26 16:37:29.297386335 (505h8m29.16846937s) [upload=443263]: Uploading segments... Dec 26 16:37:29.297473902 (505h8m29.168556947s) [upload=443263]: Got next segment. Inline: false Dec 26 16:37:29.297515431 (505h8m29.168598467s) [upload=443263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:37:29.297521964 (505h8m29.168605009s) [upload=443263]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:37:29.544301414 (505h8m29.415384449s) [seg_pos={0 0}, upload=443263]: Begin upload segment... Dec 26 16:37:29.544359325 (505h8m29.41544236s) [seg_pos={0 0}, upload=443263]: Done begin upload segment. Dec 26 16:37:29.544388360 (505h8m29.415471395s) [noise=true , address=45.91.116.14:4191 , piece_id=CRS5MFAUUM4QAMCSA27GIUNK2DYGCPZUQ45OI75BSALOPW32UCKQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.544452883 (505h8m29.415535918s) [noise=true , address=145.79.227.51:4409 , piece_id=TP5KPGXQT46CH5O2QUWAPAQQFHQTXVVE465C7NMTWDHYO2Q2RA3A, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.544975058 (505h8m29.416058093s) [noise=true , address=145.79.227.37:4566 , piece_id=MDTJ7G6ZAGFR42S5API3IBD2FGOZXI7NLE3I2OIVCF3P4T4NQ3LA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.545259659 (505h8m29.416342704s) [noise=true , address=145.79.227.13:4804 , piece_id=UOL2F3ULLGMJYUM3DDMLS37CYGAAVYAAXEJXULC4ZM7QIMLKOCSA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.545535645 (505h8m29.41661868s) [noise=true , address=45.91.116.4:4940 , piece_id=RZ4UPCTN4INRLKKB2Z7NAPMFGJROLNPBCGR463MO7OOQ6QDKVGNQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.545944193 (505h8m29.417027228s) [noise=true , address=45.91.116.5:4300 , piece_id=VN3C7NU2X2Q4CEOABPEDYTGLSACEAIUKOAPPAY7576JVUH6JI6DA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.546222914 (505h8m29.417305959s) [noise=true , address=77.90.31.10:4520 , piece_id=XINQ4KY3TRQ675HEJRVUQXMHMCP4EJWKDMIIKXDOXSMQUQXAPJYA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.546501424 (505h8m29.417584459s) [noise=true , address=145.79.227.22:4509 , piece_id=JTVULVGIRWP6YWVB2LRWKTDPW3LTVAWE4KN3LDPOPCAETV3UQMLA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.546824910 (505h8m29.417907955s) [noise=true , address=77.90.31.19:4841 , piece_id=AFOELLKOI6X6SCWVF23QRDYWFSUP2AM5DCKNL5NHDRUUKODIRUHA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.548078477 (505h8m29.419161512s) [noise=true , address=145.79.227.43:4157 , piece_id=A5H2QYKRN3HHWQUMWWUKGJLILKOQVLGQ5MM4CQRH5ECTOBVROTDA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.548184078 (505h8m29.419267113s) [noise=true , address=145.79.227.6:4257 , piece_id=XLHDMVMFGODI5L7LBBMD7MZSREBO7HLMR4LIL4IPMVK2HMTBJWUQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.549250759 (505h8m29.420333794s) [noise=true , address=145.79.227.39:4014 , piece_id=ONUVY2Q26V64DMVZ36UFD7WESAKWRPCKXO2LCB22EVOA3QOTTV4Q, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.549380546 (505h8m29.420463591s) [noise=true , address=45.91.116.7:4009 , piece_id=GAMH75PUG45WLH7K2FYMBYPVOTLDDBCJSP6Z7MB4NXYLEZPRP2VQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.550718123 (505h8m29.421801168s) [noise=true , address=145.79.227.45:4229 , piece_id=ZQYVHRSFNR5LSRQDKHN7JFZWFPZCLKK3D4DP7F3FRS2NSUXJ2THA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.551967743 (505h8m29.423050788s) [noise=true , address=145.79.227.46:4601 , piece_id=MWAIQ23QBDSQ5H7SE7OTI55T5C3KO7OJG6AGGKNT56KIOX2PDWPA, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:29.553114406 (505h8m29.424197452s) [noise=true , address=77.90.31.14:4663 , piece_id=DIKBZ34FZCFLGFBHJJDX5BBZYNL2FTHNG3V6TSDCBPN5TXHVZMRQ, seg_pos={0 0}, upload=443263]: Uploading piece... Dec 26 16:37:31.267104923 (505h8m31.138187968s) [upload=443263]: Next returned nil segment Dec 26 16:37:31.267129039 (505h8m31.138212084s) [upload=443263]: Waiting for error group managing segments... Dec 26 16:37:31.373605980 (505h8m31.244689025s) [noise=true , address=77.90.31.19:4841 , piece_id=AFOELLKOI6X6SCWVF23QRDYWFSUP2AM5DCKNL5NHDRUUKODIRUHA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 26 16:37:31.547383522 (505h8m31.418466567s) [noise=true , address=145.79.227.13:4804 , piece_id=UOL2F3ULLGMJYUM3DDMLS37CYGAAVYAAXEJXULC4ZM7QIMLKOCSA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.556110039 (505h8m31.427193074s) [noise=true , address=145.79.227.6:4257 , piece_id=XLHDMVMFGODI5L7LBBMD7MZSREBO7HLMR4LIL4IPMVK2HMTBJWUQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.556833557 (505h8m31.427916602s) [noise=true , address=145.79.227.39:4014 , piece_id=ONUVY2Q26V64DMVZ36UFD7WESAKWRPCKXO2LCB22EVOA3QOTTV4Q, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.558425728 (505h8m31.429508773s) [noise=true , address=145.79.227.46:4601 , piece_id=MWAIQ23QBDSQ5H7SE7OTI55T5C3KO7OJG6AGGKNT56KIOX2PDWPA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.559746073 (505h8m31.430829108s) [noise=true , address=145.79.227.22:4509 , piece_id=JTVULVGIRWP6YWVB2LRWKTDPW3LTVAWE4KN3LDPOPCAETV3UQMLA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.560533582 (505h8m31.431616628s) [noise=true , address=145.79.227.45:4229 , piece_id=ZQYVHRSFNR5LSRQDKHN7JFZWFPZCLKK3D4DP7F3FRS2NSUXJ2THA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.560664382 (505h8m31.431747417s) [noise=true , address=145.79.227.51:4409 , piece_id=TP5KPGXQT46CH5O2QUWAPAQQFHQTXVVE465C7NMTWDHYO2Q2RA3A, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.571627928 (505h8m31.442710973s) [noise=true , address=145.79.227.37:4566 , piece_id=MDTJ7G6ZAGFR42S5API3IBD2FGOZXI7NLE3I2OIVCF3P4T4NQ3LA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.843361235 (505h8m31.714444281s) [noise=true , address=77.90.31.14:4663 , piece_id=DIKBZ34FZCFLGFBHJJDX5BBZYNL2FTHNG3V6TSDCBPN5TXHVZMRQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.864291030 (505h8m31.735374065s) [noise=true , address=77.90.31.10:4520 , piece_id=XINQ4KY3TRQ675HEJRVUQXMHMCP4EJWKDMIIKXDOXSMQUQXAPJYA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.878425975 (505h8m31.749509011s) [noise=true , address=45.91.116.7:4009 , piece_id=GAMH75PUG45WLH7K2FYMBYPVOTLDDBCJSP6Z7MB4NXYLEZPRP2VQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:31.908628876 (505h8m31.779711911s) [noise=true , address=45.91.116.4:4940 , piece_id=RZ4UPCTN4INRLKKB2Z7NAPMFGJROLNPBCGR463MO7OOQ6QDKVGNQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:32.819947576 (505h8m32.691030611s) [noise=true , address=45.91.116.5:4300 , piece_id=VN3C7NU2X2Q4CEOABPEDYTGLSACEAIUKOAPPAY7576JVUH6JI6DA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: <nil> Dec 26 16:37:32.819983785 (505h8m32.69106683s) [seg_pos={0 0}, upload=443263]: Segment reached optimal threshold of 13 pieces. Dec 26 16:37:32.820176242 (505h8m32.691259277s) [noise=true , address=145.79.227.43:4157 , piece_id=A5H2QYKRN3HHWQUMWWUKGJLILKOQVLGQ5MM4CQRH5ECTOBVROTDA, seg_pos={0 0}, upload=443263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 26 16:37:32.820182814 (505h8m32.691265849s) [noise=true , address=145.79.227.43:4157 , piece_id=A5H2QYKRN3HHWQUMWWUKGJLILKOQVLGQ5MM4CQRH5ECTOBVROTDA, seg_pos={0 0}, upload=443263]: Long tail cancellation due to optimal threshold Dec 26 16:37:32.820296280 (505h8m32.691379315s) [noise=true , address=45.91.116.14:4191 , piece_id=CRS5MFAUUM4QAMCSA27GIUNK2DYGCPZUQ45OI75BSALOPW32UCKQ, seg_pos={0 0}, upload=443263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 16:37:32.820301560 (505h8m32.691384615s) [noise=true , address=45.91.116.14:4191 , piece_id=CRS5MFAUUM4QAMCSA27GIUNK2DYGCPZUQ45OI75BSALOPW32UCKQ, seg_pos={0 0}, upload=443263]: Long tail cancellation due to optimal threshold Dec 26 16:37:32.820339582 (505h8m32.691422618s) [seg_pos={0 0}, upload=443263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:37:32.820416289 (505h8m32.691499324s) [upload=443263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:37:32.820420507 (505h8m32.691503542s) [upload=443263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:37:33.108070794 (505h8m32.979153829s) [upload=443263]: Done uploading segments... Dec 26 16:37:33.108097985 (505h8m32.97918102s) [upload=443263]: Upload finished. err: <nil> Duration: 30.003198745s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 18.569796043s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:37:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 18.569826682s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [......X.............XXX...X...XX.....XX..........X......XXX.] duration: 4.952580417s trace: https://jaeger.dev.storj.io/trace/5b9c7e160a55f0bd FAIL (2025-12-26 16:37:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.952608681s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X..X...X.......XX..........X.........X..X...............X..] duration: 53.238264228s trace: https://jaeger.dev.storj.io/trace/4bc9becdddcc2037 FAIL (2025-12-26 16:37:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443223]: Uploading piece... Dec 26 16:34:46.586854691 (505h5m46.457937736s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.587812465 (505h5m46.4588955s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.588277791 (505h5m46.459360826s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590149876 (505h5m46.461232911s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590674354 (505h5m46.461757399s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591254680 (505h5m46.462337715s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591632860 (505h5m46.462715895s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591778888 (505h5m46.462861923s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592107854 (505h5m46.463190889s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592306482 (505h5m46.463389517s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592535257 (505h5m46.463618302s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592867591 (505h5m46.463950636s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592905783 (505h5m46.463988828s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593325633 (505h5m46.464408678s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593417989 (505h5m46.464501034s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593721267 (505h5m46.464804302s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.594046005 (505h5m46.46512905s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:47.885134462 (505h5m47.756217497s) [upload=443223]: Next returned nil segment Dec 26 16:34:47.885151225 (505h5m47.75623426s) [upload=443223]: Waiting for error group managing segments... Dec 26 16:34:48.239296449 (505h5m48.110379494s) [noise=true , address=64.49.13.140:4505 , piece_id=FXVVLCPQXGJLYWDGHDOQECN3S4ZZXTEEF7HNVLQW33IUEIPYW6OA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465120747 (505h5m48.336203792s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465174059 (505h5m48.336257094s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.468068951 (505h5m48.339151986s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.481407861 (505h5m48.352490896s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.482107242 (505h5m48.353190277s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483127506 (505h5m48.354210551s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483731115 (505h5m48.35481415s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.639322688 (505h5m48.510405733s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.760495594 (505h5m48.631578639s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.763286658 (505h5m48.634369693s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.771028379 (505h5m48.642111414s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.341293323 (505h5m49.212376368s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446731015 (505h5m49.31781406s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446757516 (505h5m49.317840561s) [seg_pos={0 0}, upload=443223]: Segment reached optimal threshold of 14 pieces. Dec 26 16:34:49.446947537 (505h5m49.318030572s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.446954180 (505h5m49.318037216s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447148209 (505h5m49.318231244s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447154361 (505h5m49.318237396s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447201702 (505h5m49.318284767s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447232551 (505h5m49.318315606s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447285240 (505h5m49.318368275s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447291793 (505h5m49.318374838s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447309938 (505h5m49.318392973s) [seg_pos={0 0}, upload=443223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:34:49.447375853 (505h5m49.318458898s) [upload=443223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:34:49.447380402 (505h5m49.318463437s) [upload=443223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:34:49.756168298 (505h5m49.627251333s) [upload=443223]: Done uploading segments... Dec 26 16:34:49.756192103 (505h5m49.627275149s) [upload=443223]: Upload finished. err: <nil> Duration: 53.238283805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.........X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X..] duration: 6.040046463s trace: https://jaeger.dev.storj.io/trace/64f12cb65d7db0b6 FAIL (2025-12-26 16:36:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: KK322Q3BL4VKH3XQSMWU7VZ2A32UQLPXS3HR3APAOLAFRKIG5NSQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 25T5NRLRGQ7F77Q5VESFSJG66KX6IRSVZW3Z5RIS2GP4SN6YZRIA) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: ZB3STKDNGCGCXXNJTGFABJ3YKXGHTCIJEKC3CUSX6HUTGLTBM7QQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: EJSHXLUUFD4GBSJHRYU2LNTSM6RSD7DLKSBHOACZ63USJBDEWTEA) context deadline exceeded ======LOGS====== (505h6m25.177144957s) [upload=443230]: Uploading segments... Dec 26 16:35:25.306210225 (505h6m25.17729326s) [upload=443230]: Got next segment. Inline: false Dec 26 16:35:25.306276360 (505h6m25.177359395s) [upload=443230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:35:25.306281611 (505h6m25.177364646s) [upload=443230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:35:25.552989044 (505h6m25.424072079s) [seg_pos={0 0}, upload=443230]: Begin upload segment... Dec 26 16:35:25.553040944 (505h6m25.424123989s) [seg_pos={0 0}, upload=443230]: Done begin upload segment. Dec 26 16:35:25.553122709 (505h6m25.424205754s) [noise=true , address=145.79.227.11:4732 , piece_id=DM4ENZHLFTWBAYXC72PUCL6ZZQXT2TWS2B4VV4GU5QBCHD7LFT4A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.553163637 (505h6m25.424246682s) [noise=true , address=145.79.227.31:4799 , piece_id=ZOVTL43UW4REIXOC2RF6UN73ITFZEVQV4GD44R4K5FS6HA4I4I5Q, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.553923965 (505h6m25.42500701s) [noise=true , address=45.91.116.6:4973 , piece_id=MRAAH4XLPLCQC72WIKOJMCG6MKAYR4LFFVOFWX5K6ILHVBS6ZNAQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.554399040 (505h6m25.425482085s) [noise=true , address=145.79.227.50:4375 , piece_id=NU6XHJVGLP3HSULT5T4YRSZGWNVSCTAZJZC53EMYFGIPCF36CKUQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.554883532 (505h6m25.425966587s) [noise=true , address=145.79.227.38:4978 , piece_id=OKFY6FYGXFDUFO64BF275IDMWDAQE46R5O2YFOI2DTOWVTREGHWA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.555373064 (505h6m25.426456109s) [noise=true , address=145.79.227.8:4330 , piece_id=HCB24RIFWDQRDKBFLUL4UJEBN2AEBCW4HUDDLRFJASGWCAE7PPNQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.555865983 (505h6m25.426949028s) [noise=true , address=145.79.227.45:4229 , piece_id=LGDVW53JZQOS2WV7C77VYXVZGLDSQBYJA7FB7HPY65E5CBPBNF6A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.556875355 (505h6m25.4279584s) [noise=true , address=145.79.227.18:4007 , piece_id=VVBTOPZPD6KBKTJT5XY7ECZEKTKGOMW5JQ3WD7CS5WXB2RTLZIIA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.557919453 (505h6m25.429002498s) [noise=true , address=145.79.227.49:4337 , piece_id=XBKBKBUGRE5JZSBO2ZPBYMLHBEGXVG76HDMTJYRPFKUSIGX3TOSQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.559175565 (505h6m25.4302586s) [noise=true , address=45.91.116.14:4192 , piece_id=KK322Q3BL4VKH3XQSMWU7VZ2A32UQLPXS3HR3APAOLAFRKIG5NSQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.560312209 (505h6m25.431395254s) [noise=true , address=77.90.31.10:4518 , piece_id=25T5NRLRGQ7F77Q5VESFSJG66KX6IRSVZW3Z5RIS2GP4SN6YZRIA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.560383074 (505h6m25.431466129s) [noise=true , address=145.79.227.48:4301 , piece_id=ZB3STKDNGCGCXXNJTGFABJ3YKXGHTCIJEKC3CUSX6HUTGLTBM7QQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.561525960 (505h6m25.432608995s) [noise=true , address=145.79.227.6:4259 , piece_id=OOCYZFJWNM2FJ5KMIEY73LUCETOHOWPK22UI6UUD5XNG2WGK5W7A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.562958318 (505h6m25.434041353s) [noise=true , address=145.79.227.2:4113 , piece_id=3UNELYLESMPBLPFFXTODTB3NH5BVD57OWHMX6FN3CTV25CCTVG6A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.563017440 (505h6m25.434100485s) [noise=true , address=145.79.227.34:4907 , piece_id=M6GET3TBB2SETWCOMBTCVA3TKCY3Q3X2RPZRKXFSIMHRPJXHC5ZA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.564357242 (505h6m25.435440277s) [noise=true , address=45.91.116.5:4297 , piece_id=EJSHXLUUFD4GBSJHRYU2LNTSM6RSD7DLKSBHOACZ63USJBDEWTEA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:27.167253516 (505h6m27.038336551s) [upload=443230]: Next returned nil segment Dec 26 16:35:27.167267121 (505h6m27.038350166s) [upload=443230]: Waiting for error group managing segments... Dec 26 16:35:27.552207475 (505h6m27.42329051s) [noise=true , address=145.79.227.11:4732 , piece_id=DM4ENZHLFTWBAYXC72PUCL6ZZQXT2TWS2B4VV4GU5QBCHD7LFT4A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.566801595 (505h6m27.43788463s) [noise=true , address=145.79.227.48:4301 , piece_id=ZB3STKDNGCGCXXNJTGFABJ3YKXGHTCIJEKC3CUSX6HUTGLTBM7QQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.569841033 (505h6m27.440924078s) [noise=true , address=145.79.227.8:4330 , piece_id=HCB24RIFWDQRDKBFLUL4UJEBN2AEBCW4HUDDLRFJASGWCAE7PPNQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.573863191 (505h6m27.444946226s) [noise=true , address=145.79.227.49:4337 , piece_id=XBKBKBUGRE5JZSBO2ZPBYMLHBEGXVG76HDMTJYRPFKUSIGX3TOSQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.588141019 (505h6m27.459224064s) [noise=true , address=145.79.227.38:4978 , piece_id=OKFY6FYGXFDUFO64BF275IDMWDAQE46R5O2YFOI2DTOWVTREGHWA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.588793191 (505h6m27.459876226s) [noise=true , address=145.79.227.31:4799 , piece_id=ZOVTL43UW4REIXOC2RF6UN73ITFZEVQV4GD44R4K5FS6HA4I4I5Q, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.589226415 (505h6m27.46030946s) [noise=true , address=145.79.227.45:4229 , piece_id=LGDVW53JZQOS2WV7C77VYXVZGLDSQBYJA7FB7HPY65E5CBPBNF6A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.592841078 (505h6m27.463924113s) [noise=true , address=145.79.227.18:4007 , piece_id=VVBTOPZPD6KBKTJT5XY7ECZEKTKGOMW5JQ3WD7CS5WXB2RTLZIIA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.594745725 (505h6m27.46582877s) [noise=true , address=145.79.227.50:4375 , piece_id=NU6XHJVGLP3HSULT5T4YRSZGWNVSCTAZJZC53EMYFGIPCF36CKUQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.790554880 (505h6m27.661637915s) [noise=true , address=45.91.116.6:4973 , piece_id=MRAAH4XLPLCQC72WIKOJMCG6MKAYR4LFFVOFWX5K6ILHVBS6ZNAQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.800118801 (505h6m27.671201847s) [noise=true , address=45.91.116.14:4192 , piece_id=KK322Q3BL4VKH3XQSMWU7VZ2A32UQLPXS3HR3APAOLAFRKIG5NSQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.811160206 (505h6m27.682243241s) [noise=true , address=77.90.31.10:4518 , piece_id=25T5NRLRGQ7F77Q5VESFSJG66KX6IRSVZW3Z5RIS2GP4SN6YZRIA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.825394921 (505h6m27.696477966s) [noise=true , address=45.91.116.5:4297 , piece_id=EJSHXLUUFD4GBSJHRYU2LNTSM6RSD7DLKSBHOACZ63USJBDEWTEA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.825416092 (505h6m27.696499127s) [seg_pos={0 0}, upload=443230]: Segment reached optimal threshold of 13 pieces. Dec 26 16:35:27.825610271 (505h6m27.696693306s) [noise=true , address=145.79.227.2:4113 , piece_id=3UNELYLESMPBLPFFXTODTB3NH5BVD57OWHMX6FN3CTV25CCTVG6A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:43570->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:43570->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43570->145.79.227.2:4113: use of closed network connection Dec 26 16:35:27.825617766 (505h6m27.696700801s) [noise=true , address=145.79.227.2:4113 , piece_id=3UNELYLESMPBLPFFXTODTB3NH5BVD57OWHMX6FN3CTV25CCTVG6A, seg_pos={0 0}, upload=443230]: Long tail cancellation due to optimal threshold Dec 26 16:35:27.825930141 (505h6m27.697013176s) [noise=true , address=145.79.227.6:4259 , piece_id=OOCYZFJWNM2FJ5KMIEY73LUCETOHOWPK22UI6UUD5XNG2WGK5W7A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:48232->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48232->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48232->145.79.227.6:4259: use of closed network connection Dec 26 16:35:27.825944909 (505h6m27.697027944s) [noise=true , address=145.79.227.6:4259 , piece_id=OOCYZFJWNM2FJ5KMIEY73LUCETOHOWPK22UI6UUD5XNG2WGK5W7A, seg_pos={0 0}, upload=443230]: Long tail cancellation due to optimal threshold Dec 26 16:35:27.825979014 (505h6m27.697062059s) [noise=true , address=145.79.227.34:4907 , piece_id=M6GET3TBB2SETWCOMBTCVA3TKCY3Q3X2RPZRKXFSIMHRPJXHC5ZA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 16:35:27.825988341 (505h6m27.697071386s) [noise=true , address=145.79.227.34:4907 , piece_id=M6GET3TBB2SETWCOMBTCVA3TKCY3Q3X2RPZRKXFSIMHRPJXHC5ZA, seg_pos={0 0}, upload=443230]: Long tail cancellation due to optimal threshold Dec 26 16:35:27.826004603 (505h6m27.697087638s) [seg_pos={0 0}, upload=443230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:35:27.826078824 (505h6m27.697161869s) [upload=443230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:35:27.826085026 (505h6m27.697168071s) [upload=443230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:35:28.123831196 (505h6m27.994914242s) [upload=443230]: Done uploading segments... Dec 26 16:35:28.123855914 (505h6m27.994938959s) [upload=443230]: Upload finished. err: <nil> Duration: 6.040071159s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.......X.............XXX...X...XX.....XX..........X......XXX] duration: 5.001619456s trace: https://jaeger.dev.storj.io/trace/5b9c7e160a55f0bd FAIL (2025-12-26 16:36:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001640156s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 8.148480479s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:36:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 8.148620075s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..........X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X.] duration: 6.154464535s trace: https://jaeger.dev.storj.io/trace/64f12cb65d7db0b6 FAIL (2025-12-26 16:36:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: KK322Q3BL4VKH3XQSMWU7VZ2A32UQLPXS3HR3APAOLAFRKIG5NSQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 25T5NRLRGQ7F77Q5VESFSJG66KX6IRSVZW3Z5RIS2GP4SN6YZRIA) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: ZB3STKDNGCGCXXNJTGFABJ3YKXGHTCIJEKC3CUSX6HUTGLTBM7QQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: EJSHXLUUFD4GBSJHRYU2LNTSM6RSD7DLKSBHOACZ63USJBDEWTEA) context deadline exceeded ======LOGS====== (505h6m25.177144957s) [upload=443230]: Uploading segments... Dec 26 16:35:25.306210225 (505h6m25.17729326s) [upload=443230]: Got next segment. Inline: false Dec 26 16:35:25.306276360 (505h6m25.177359395s) [upload=443230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:35:25.306281611 (505h6m25.177364646s) [upload=443230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:35:25.552989044 (505h6m25.424072079s) [seg_pos={0 0}, upload=443230]: Begin upload segment... Dec 26 16:35:25.553040944 (505h6m25.424123989s) [seg_pos={0 0}, upload=443230]: Done begin upload segment. Dec 26 16:35:25.553122709 (505h6m25.424205754s) [noise=true , address=145.79.227.11:4732 , piece_id=DM4ENZHLFTWBAYXC72PUCL6ZZQXT2TWS2B4VV4GU5QBCHD7LFT4A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.553163637 (505h6m25.424246682s) [noise=true , address=145.79.227.31:4799 , piece_id=ZOVTL43UW4REIXOC2RF6UN73ITFZEVQV4GD44R4K5FS6HA4I4I5Q, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.553923965 (505h6m25.42500701s) [noise=true , address=45.91.116.6:4973 , piece_id=MRAAH4XLPLCQC72WIKOJMCG6MKAYR4LFFVOFWX5K6ILHVBS6ZNAQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.554399040 (505h6m25.425482085s) [noise=true , address=145.79.227.50:4375 , piece_id=NU6XHJVGLP3HSULT5T4YRSZGWNVSCTAZJZC53EMYFGIPCF36CKUQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.554883532 (505h6m25.425966587s) [noise=true , address=145.79.227.38:4978 , piece_id=OKFY6FYGXFDUFO64BF275IDMWDAQE46R5O2YFOI2DTOWVTREGHWA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.555373064 (505h6m25.426456109s) [noise=true , address=145.79.227.8:4330 , piece_id=HCB24RIFWDQRDKBFLUL4UJEBN2AEBCW4HUDDLRFJASGWCAE7PPNQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.555865983 (505h6m25.426949028s) [noise=true , address=145.79.227.45:4229 , piece_id=LGDVW53JZQOS2WV7C77VYXVZGLDSQBYJA7FB7HPY65E5CBPBNF6A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.556875355 (505h6m25.4279584s) [noise=true , address=145.79.227.18:4007 , piece_id=VVBTOPZPD6KBKTJT5XY7ECZEKTKGOMW5JQ3WD7CS5WXB2RTLZIIA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.557919453 (505h6m25.429002498s) [noise=true , address=145.79.227.49:4337 , piece_id=XBKBKBUGRE5JZSBO2ZPBYMLHBEGXVG76HDMTJYRPFKUSIGX3TOSQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.559175565 (505h6m25.4302586s) [noise=true , address=45.91.116.14:4192 , piece_id=KK322Q3BL4VKH3XQSMWU7VZ2A32UQLPXS3HR3APAOLAFRKIG5NSQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.560312209 (505h6m25.431395254s) [noise=true , address=77.90.31.10:4518 , piece_id=25T5NRLRGQ7F77Q5VESFSJG66KX6IRSVZW3Z5RIS2GP4SN6YZRIA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.560383074 (505h6m25.431466129s) [noise=true , address=145.79.227.48:4301 , piece_id=ZB3STKDNGCGCXXNJTGFABJ3YKXGHTCIJEKC3CUSX6HUTGLTBM7QQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.561525960 (505h6m25.432608995s) [noise=true , address=145.79.227.6:4259 , piece_id=OOCYZFJWNM2FJ5KMIEY73LUCETOHOWPK22UI6UUD5XNG2WGK5W7A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.562958318 (505h6m25.434041353s) [noise=true , address=145.79.227.2:4113 , piece_id=3UNELYLESMPBLPFFXTODTB3NH5BVD57OWHMX6FN3CTV25CCTVG6A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.563017440 (505h6m25.434100485s) [noise=true , address=145.79.227.34:4907 , piece_id=M6GET3TBB2SETWCOMBTCVA3TKCY3Q3X2RPZRKXFSIMHRPJXHC5ZA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.564357242 (505h6m25.435440277s) [noise=true , address=45.91.116.5:4297 , piece_id=EJSHXLUUFD4GBSJHRYU2LNTSM6RSD7DLKSBHOACZ63USJBDEWTEA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:27.167253516 (505h6m27.038336551s) [upload=443230]: Next returned nil segment Dec 26 16:35:27.167267121 (505h6m27.038350166s) [upload=443230]: Waiting for error group managing segments... Dec 26 16:35:27.552207475 (505h6m27.42329051s) [noise=true , address=145.79.227.11:4732 , piece_id=DM4ENZHLFTWBAYXC72PUCL6ZZQXT2TWS2B4VV4GU5QBCHD7LFT4A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.566801595 (505h6m27.43788463s) [noise=true , address=145.79.227.48:4301 , piece_id=ZB3STKDNGCGCXXNJTGFABJ3YKXGHTCIJEKC3CUSX6HUTGLTBM7QQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.569841033 (505h6m27.440924078s) [noise=true , address=145.79.227.8:4330 , piece_id=HCB24RIFWDQRDKBFLUL4UJEBN2AEBCW4HUDDLRFJASGWCAE7PPNQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.573863191 (505h6m27.444946226s) [noise=true , address=145.79.227.49:4337 , piece_id=XBKBKBUGRE5JZSBO2ZPBYMLHBEGXVG76HDMTJYRPFKUSIGX3TOSQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.588141019 (505h6m27.459224064s) [noise=true , address=145.79.227.38:4978 , piece_id=OKFY6FYGXFDUFO64BF275IDMWDAQE46R5O2YFOI2DTOWVTREGHWA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.588793191 (505h6m27.459876226s) [noise=true , address=145.79.227.31:4799 , piece_id=ZOVTL43UW4REIXOC2RF6UN73ITFZEVQV4GD44R4K5FS6HA4I4I5Q, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.589226415 (505h6m27.46030946s) [noise=true , address=145.79.227.45:4229 , piece_id=LGDVW53JZQOS2WV7C77VYXVZGLDSQBYJA7FB7HPY65E5CBPBNF6A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.592841078 (505h6m27.463924113s) [noise=true , address=145.79.227.18:4007 , piece_id=VVBTOPZPD6KBKTJT5XY7ECZEKTKGOMW5JQ3WD7CS5WXB2RTLZIIA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.594745725 (505h6m27.46582877s) [noise=true , address=145.79.227.50:4375 , piece_id=NU6XHJVGLP3HSULT5T4YRSZGWNVSCTAZJZC53EMYFGIPCF36CKUQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.790554880 (505h6m27.661637915s) [noise=true , address=45.91.116.6:4973 , piece_id=MRAAH4XLPLCQC72WIKOJMCG6MKAYR4LFFVOFWX5K6ILHVBS6ZNAQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.800118801 (505h6m27.671201847s) [noise=true , address=45.91.116.14:4192 , piece_id=KK322Q3BL4VKH3XQSMWU7VZ2A32UQLPXS3HR3APAOLAFRKIG5NSQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.811160206 (505h6m27.682243241s) [noise=true , address=77.90.31.10:4518 , piece_id=25T5NRLRGQ7F77Q5VESFSJG66KX6IRSVZW3Z5RIS2GP4SN6YZRIA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.825394921 (505h6m27.696477966s) [noise=true , address=45.91.116.5:4297 , piece_id=EJSHXLUUFD4GBSJHRYU2LNTSM6RSD7DLKSBHOACZ63USJBDEWTEA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.825416092 (505h6m27.696499127s) [seg_pos={0 0}, upload=443230]: Segment reached optimal threshold of 13 pieces. Dec 26 16:35:27.825610271 (505h6m27.696693306s) [noise=true , address=145.79.227.2:4113 , piece_id=3UNELYLESMPBLPFFXTODTB3NH5BVD57OWHMX6FN3CTV25CCTVG6A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:43570->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:43570->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43570->145.79.227.2:4113: use of closed network connection Dec 26 16:35:27.825617766 (505h6m27.696700801s) [noise=true , address=145.79.227.2:4113 , piece_id=3UNELYLESMPBLPFFXTODTB3NH5BVD57OWHMX6FN3CTV25CCTVG6A, seg_pos={0 0}, upload=443230]: Long tail cancellation due to optimal threshold Dec 26 16:35:27.825930141 (505h6m27.697013176s) [noise=true , address=145.79.227.6:4259 , piece_id=OOCYZFJWNM2FJ5KMIEY73LUCETOHOWPK22UI6UUD5XNG2WGK5W7A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:48232->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48232->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48232->145.79.227.6:4259: use of closed network connection Dec 26 16:35:27.825944909 (505h6m27.697027944s) [noise=true , address=145.79.227.6:4259 , piece_id=OOCYZFJWNM2FJ5KMIEY73LUCETOHOWPK22UI6UUD5XNG2WGK5W7A, seg_pos={0 0}, upload=443230]: Long tail cancellation due to optimal threshold Dec 26 16:35:27.825979014 (505h6m27.697062059s) [noise=true , address=145.79.227.34:4907 , piece_id=M6GET3TBB2SETWCOMBTCVA3TKCY3Q3X2RPZRKXFSIMHRPJXHC5ZA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 16:35:27.825988341 (505h6m27.697071386s) [noise=true , address=145.79.227.34:4907 , piece_id=M6GET3TBB2SETWCOMBTCVA3TKCY3Q3X2RPZRKXFSIMHRPJXHC5ZA, seg_pos={0 0}, upload=443230]: Long tail cancellation due to optimal threshold Dec 26 16:35:27.826004603 (505h6m27.697087638s) [seg_pos={0 0}, upload=443230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:35:27.826078824 (505h6m27.697161869s) [upload=443230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:35:27.826085026 (505h6m27.697168071s) [upload=443230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:35:28.123831196 (505h6m27.994914242s) [upload=443230]: Done uploading segments... Dec 26 16:35:28.123855914 (505h6m27.994938959s) [upload=443230]: Upload finished. err: <nil> Duration: 6.154487849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X..X...X.......XX..........X.........X..X...............X.] duration: 55.618224808s trace: https://jaeger.dev.storj.io/trace/4bc9becdddcc2037 FAIL (2025-12-26 16:36:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443223]: Uploading piece... Dec 26 16:34:46.586854691 (505h5m46.457937736s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.587812465 (505h5m46.4588955s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.588277791 (505h5m46.459360826s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590149876 (505h5m46.461232911s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590674354 (505h5m46.461757399s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591254680 (505h5m46.462337715s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591632860 (505h5m46.462715895s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591778888 (505h5m46.462861923s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592107854 (505h5m46.463190889s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592306482 (505h5m46.463389517s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592535257 (505h5m46.463618302s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592867591 (505h5m46.463950636s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592905783 (505h5m46.463988828s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593325633 (505h5m46.464408678s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593417989 (505h5m46.464501034s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593721267 (505h5m46.464804302s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.594046005 (505h5m46.46512905s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:47.885134462 (505h5m47.756217497s) [upload=443223]: Next returned nil segment Dec 26 16:34:47.885151225 (505h5m47.75623426s) [upload=443223]: Waiting for error group managing segments... Dec 26 16:34:48.239296449 (505h5m48.110379494s) [noise=true , address=64.49.13.140:4505 , piece_id=FXVVLCPQXGJLYWDGHDOQECN3S4ZZXTEEF7HNVLQW33IUEIPYW6OA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465120747 (505h5m48.336203792s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465174059 (505h5m48.336257094s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.468068951 (505h5m48.339151986s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.481407861 (505h5m48.352490896s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.482107242 (505h5m48.353190277s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483127506 (505h5m48.354210551s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483731115 (505h5m48.35481415s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.639322688 (505h5m48.510405733s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.760495594 (505h5m48.631578639s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.763286658 (505h5m48.634369693s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.771028379 (505h5m48.642111414s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.341293323 (505h5m49.212376368s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446731015 (505h5m49.31781406s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446757516 (505h5m49.317840561s) [seg_pos={0 0}, upload=443223]: Segment reached optimal threshold of 14 pieces. Dec 26 16:34:49.446947537 (505h5m49.318030572s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.446954180 (505h5m49.318037216s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447148209 (505h5m49.318231244s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447154361 (505h5m49.318237396s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447201702 (505h5m49.318284767s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447232551 (505h5m49.318315606s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447285240 (505h5m49.318368275s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447291793 (505h5m49.318374838s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447309938 (505h5m49.318392973s) [seg_pos={0 0}, upload=443223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:34:49.447375853 (505h5m49.318458898s) [upload=443223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:34:49.447380402 (505h5m49.318463437s) [upload=443223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:34:49.756168298 (505h5m49.627251333s) [upload=443223]: Done uploading segments... Dec 26 16:34:49.756192103 (505h5m49.627275149s) [upload=443223]: Upload finished. err: <nil> Duration: 55.618254635s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...........X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X] duration: 30.002824593s trace: https://jaeger.dev.storj.io/trace/64f12cb65d7db0b6 FAIL (2025-12-26 16:35:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: KK322Q3BL4VKH3XQSMWU7VZ2A32UQLPXS3HR3APAOLAFRKIG5NSQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 25T5NRLRGQ7F77Q5VESFSJG66KX6IRSVZW3Z5RIS2GP4SN6YZRIA) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: ZB3STKDNGCGCXXNJTGFABJ3YKXGHTCIJEKC3CUSX6HUTGLTBM7QQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: EJSHXLUUFD4GBSJHRYU2LNTSM6RSD7DLKSBHOACZ63USJBDEWTEA) context deadline exceeded ======LOGS====== (505h6m25.177144957s) [upload=443230]: Uploading segments... Dec 26 16:35:25.306210225 (505h6m25.17729326s) [upload=443230]: Got next segment. Inline: false Dec 26 16:35:25.306276360 (505h6m25.177359395s) [upload=443230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:35:25.306281611 (505h6m25.177364646s) [upload=443230]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:35:25.552989044 (505h6m25.424072079s) [seg_pos={0 0}, upload=443230]: Begin upload segment... Dec 26 16:35:25.553040944 (505h6m25.424123989s) [seg_pos={0 0}, upload=443230]: Done begin upload segment. Dec 26 16:35:25.553122709 (505h6m25.424205754s) [noise=true , address=145.79.227.11:4732 , piece_id=DM4ENZHLFTWBAYXC72PUCL6ZZQXT2TWS2B4VV4GU5QBCHD7LFT4A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.553163637 (505h6m25.424246682s) [noise=true , address=145.79.227.31:4799 , piece_id=ZOVTL43UW4REIXOC2RF6UN73ITFZEVQV4GD44R4K5FS6HA4I4I5Q, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.553923965 (505h6m25.42500701s) [noise=true , address=45.91.116.6:4973 , piece_id=MRAAH4XLPLCQC72WIKOJMCG6MKAYR4LFFVOFWX5K6ILHVBS6ZNAQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.554399040 (505h6m25.425482085s) [noise=true , address=145.79.227.50:4375 , piece_id=NU6XHJVGLP3HSULT5T4YRSZGWNVSCTAZJZC53EMYFGIPCF36CKUQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.554883532 (505h6m25.425966587s) [noise=true , address=145.79.227.38:4978 , piece_id=OKFY6FYGXFDUFO64BF275IDMWDAQE46R5O2YFOI2DTOWVTREGHWA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.555373064 (505h6m25.426456109s) [noise=true , address=145.79.227.8:4330 , piece_id=HCB24RIFWDQRDKBFLUL4UJEBN2AEBCW4HUDDLRFJASGWCAE7PPNQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.555865983 (505h6m25.426949028s) [noise=true , address=145.79.227.45:4229 , piece_id=LGDVW53JZQOS2WV7C77VYXVZGLDSQBYJA7FB7HPY65E5CBPBNF6A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.556875355 (505h6m25.4279584s) [noise=true , address=145.79.227.18:4007 , piece_id=VVBTOPZPD6KBKTJT5XY7ECZEKTKGOMW5JQ3WD7CS5WXB2RTLZIIA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.557919453 (505h6m25.429002498s) [noise=true , address=145.79.227.49:4337 , piece_id=XBKBKBUGRE5JZSBO2ZPBYMLHBEGXVG76HDMTJYRPFKUSIGX3TOSQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.559175565 (505h6m25.4302586s) [noise=true , address=45.91.116.14:4192 , piece_id=KK322Q3BL4VKH3XQSMWU7VZ2A32UQLPXS3HR3APAOLAFRKIG5NSQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.560312209 (505h6m25.431395254s) [noise=true , address=77.90.31.10:4518 , piece_id=25T5NRLRGQ7F77Q5VESFSJG66KX6IRSVZW3Z5RIS2GP4SN6YZRIA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.560383074 (505h6m25.431466129s) [noise=true , address=145.79.227.48:4301 , piece_id=ZB3STKDNGCGCXXNJTGFABJ3YKXGHTCIJEKC3CUSX6HUTGLTBM7QQ, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.561525960 (505h6m25.432608995s) [noise=true , address=145.79.227.6:4259 , piece_id=OOCYZFJWNM2FJ5KMIEY73LUCETOHOWPK22UI6UUD5XNG2WGK5W7A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.562958318 (505h6m25.434041353s) [noise=true , address=145.79.227.2:4113 , piece_id=3UNELYLESMPBLPFFXTODTB3NH5BVD57OWHMX6FN3CTV25CCTVG6A, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.563017440 (505h6m25.434100485s) [noise=true , address=145.79.227.34:4907 , piece_id=M6GET3TBB2SETWCOMBTCVA3TKCY3Q3X2RPZRKXFSIMHRPJXHC5ZA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:25.564357242 (505h6m25.435440277s) [noise=true , address=45.91.116.5:4297 , piece_id=EJSHXLUUFD4GBSJHRYU2LNTSM6RSD7DLKSBHOACZ63USJBDEWTEA, seg_pos={0 0}, upload=443230]: Uploading piece... Dec 26 16:35:27.167253516 (505h6m27.038336551s) [upload=443230]: Next returned nil segment Dec 26 16:35:27.167267121 (505h6m27.038350166s) [upload=443230]: Waiting for error group managing segments... Dec 26 16:35:27.552207475 (505h6m27.42329051s) [noise=true , address=145.79.227.11:4732 , piece_id=DM4ENZHLFTWBAYXC72PUCL6ZZQXT2TWS2B4VV4GU5QBCHD7LFT4A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.566801595 (505h6m27.43788463s) [noise=true , address=145.79.227.48:4301 , piece_id=ZB3STKDNGCGCXXNJTGFABJ3YKXGHTCIJEKC3CUSX6HUTGLTBM7QQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.569841033 (505h6m27.440924078s) [noise=true , address=145.79.227.8:4330 , piece_id=HCB24RIFWDQRDKBFLUL4UJEBN2AEBCW4HUDDLRFJASGWCAE7PPNQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.573863191 (505h6m27.444946226s) [noise=true , address=145.79.227.49:4337 , piece_id=XBKBKBUGRE5JZSBO2ZPBYMLHBEGXVG76HDMTJYRPFKUSIGX3TOSQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.588141019 (505h6m27.459224064s) [noise=true , address=145.79.227.38:4978 , piece_id=OKFY6FYGXFDUFO64BF275IDMWDAQE46R5O2YFOI2DTOWVTREGHWA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.588793191 (505h6m27.459876226s) [noise=true , address=145.79.227.31:4799 , piece_id=ZOVTL43UW4REIXOC2RF6UN73ITFZEVQV4GD44R4K5FS6HA4I4I5Q, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.589226415 (505h6m27.46030946s) [noise=true , address=145.79.227.45:4229 , piece_id=LGDVW53JZQOS2WV7C77VYXVZGLDSQBYJA7FB7HPY65E5CBPBNF6A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.592841078 (505h6m27.463924113s) [noise=true , address=145.79.227.18:4007 , piece_id=VVBTOPZPD6KBKTJT5XY7ECZEKTKGOMW5JQ3WD7CS5WXB2RTLZIIA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.594745725 (505h6m27.46582877s) [noise=true , address=145.79.227.50:4375 , piece_id=NU6XHJVGLP3HSULT5T4YRSZGWNVSCTAZJZC53EMYFGIPCF36CKUQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.790554880 (505h6m27.661637915s) [noise=true , address=45.91.116.6:4973 , piece_id=MRAAH4XLPLCQC72WIKOJMCG6MKAYR4LFFVOFWX5K6ILHVBS6ZNAQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.800118801 (505h6m27.671201847s) [noise=true , address=45.91.116.14:4192 , piece_id=KK322Q3BL4VKH3XQSMWU7VZ2A32UQLPXS3HR3APAOLAFRKIG5NSQ, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.811160206 (505h6m27.682243241s) [noise=true , address=77.90.31.10:4518 , piece_id=25T5NRLRGQ7F77Q5VESFSJG66KX6IRSVZW3Z5RIS2GP4SN6YZRIA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.825394921 (505h6m27.696477966s) [noise=true , address=45.91.116.5:4297 , piece_id=EJSHXLUUFD4GBSJHRYU2LNTSM6RSD7DLKSBHOACZ63USJBDEWTEA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: <nil> Dec 26 16:35:27.825416092 (505h6m27.696499127s) [seg_pos={0 0}, upload=443230]: Segment reached optimal threshold of 13 pieces. Dec 26 16:35:27.825610271 (505h6m27.696693306s) [noise=true , address=145.79.227.2:4113 , piece_id=3UNELYLESMPBLPFFXTODTB3NH5BVD57OWHMX6FN3CTV25CCTVG6A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:43570->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:43570->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43570->145.79.227.2:4113: use of closed network connection Dec 26 16:35:27.825617766 (505h6m27.696700801s) [noise=true , address=145.79.227.2:4113 , piece_id=3UNELYLESMPBLPFFXTODTB3NH5BVD57OWHMX6FN3CTV25CCTVG6A, seg_pos={0 0}, upload=443230]: Long tail cancellation due to optimal threshold Dec 26 16:35:27.825930141 (505h6m27.697013176s) [noise=true , address=145.79.227.6:4259 , piece_id=OOCYZFJWNM2FJ5KMIEY73LUCETOHOWPK22UI6UUD5XNG2WGK5W7A, seg_pos={0 0}, upload=443230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:48232->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48232->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48232->145.79.227.6:4259: use of closed network connection Dec 26 16:35:27.825944909 (505h6m27.697027944s) [noise=true , address=145.79.227.6:4259 , piece_id=OOCYZFJWNM2FJ5KMIEY73LUCETOHOWPK22UI6UUD5XNG2WGK5W7A, seg_pos={0 0}, upload=443230]: Long tail cancellation due to optimal threshold Dec 26 16:35:27.825979014 (505h6m27.697062059s) [noise=true , address=145.79.227.34:4907 , piece_id=M6GET3TBB2SETWCOMBTCVA3TKCY3Q3X2RPZRKXFSIMHRPJXHC5ZA, seg_pos={0 0}, upload=443230]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 16:35:27.825988341 (505h6m27.697071386s) [noise=true , address=145.79.227.34:4907 , piece_id=M6GET3TBB2SETWCOMBTCVA3TKCY3Q3X2RPZRKXFSIMHRPJXHC5ZA, seg_pos={0 0}, upload=443230]: Long tail cancellation due to optimal threshold Dec 26 16:35:27.826004603 (505h6m27.697087638s) [seg_pos={0 0}, upload=443230]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:35:27.826078824 (505h6m27.697161869s) [upload=443230]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:35:27.826085026 (505h6m27.697168071s) [upload=443230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:35:28.123831196 (505h6m27.994914242s) [upload=443230]: Done uploading segments... Dec 26 16:35:28.123855914 (505h6m27.994938959s) [upload=443230]: Upload finished. err: <nil> Duration: 30.002846635s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [........X.............XXX...X...XX.....XX..........X......XX] duration: 5.001165401s trace: https://jaeger.dev.storj.io/trace/6ca0cd450308b69c FAIL (2025-12-26 16:35:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001193705s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.868932967s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:35:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 6.868956411s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.........X.............XXX...X...XX.....XX..........X......X] duration: 5.001197733s trace: https://jaeger.dev.storj.io/trace/78ab13ad2eb4355 FAIL (2025-12-26 16:35:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001220948s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X..X...X.......XX..........X.........X..X...............X] duration: 22.190234319s trace: https://jaeger.dev.storj.io/trace/4bc9becdddcc2037 FAIL (2025-12-26 16:35:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443223]: Uploading piece... Dec 26 16:34:46.586854691 (505h5m46.457937736s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.587812465 (505h5m46.4588955s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.588277791 (505h5m46.459360826s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590149876 (505h5m46.461232911s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.590674354 (505h5m46.461757399s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591254680 (505h5m46.462337715s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591632860 (505h5m46.462715895s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.591778888 (505h5m46.462861923s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592107854 (505h5m46.463190889s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592306482 (505h5m46.463389517s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592535257 (505h5m46.463618302s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592867591 (505h5m46.463950636s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.592905783 (505h5m46.463988828s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593325633 (505h5m46.464408678s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593417989 (505h5m46.464501034s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.593721267 (505h5m46.464804302s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:46.594046005 (505h5m46.46512905s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Uploading piece... Dec 26 16:34:47.885134462 (505h5m47.756217497s) [upload=443223]: Next returned nil segment Dec 26 16:34:47.885151225 (505h5m47.75623426s) [upload=443223]: Waiting for error group managing segments... Dec 26 16:34:48.239296449 (505h5m48.110379494s) [noise=true , address=64.49.13.140:4505 , piece_id=FXVVLCPQXGJLYWDGHDOQECN3S4ZZXTEEF7HNVLQW33IUEIPYW6OA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465120747 (505h5m48.336203792s) [noise=true , address=145.79.227.40:4072 , piece_id=NEHGYSH3LXEM6DA3DSQFRGFE3ZNLGSYD6KIB3I3G3IGLFY6MURWA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.465174059 (505h5m48.336257094s) [noise=true , address=145.79.227.11:4748 , piece_id=6IHQJWWHYWVU7NXAS2Y5A6KKK67JHCI7IKPTYVJQWDHVFJDZVHJA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.468068951 (505h5m48.339151986s) [noise=true , address=145.79.227.40:4074 , piece_id=K7EI2EA7SUNRG2QPX6GPO6VODCAQVIYUMUMILQTXOITXAL5JKATA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.481407861 (505h5m48.352490896s) [noise=true , address=145.79.227.13:4828 , piece_id=UCZM6NACVZRQHOYKWNL35FNM7AIUE576J4MXW6UYLT4O6YPXI73Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.482107242 (505h5m48.353190277s) [noise=true , address=145.79.227.38:4008 , piece_id=QGPDNM6A4VLDWS4PC6AHRDIEZJX7KVV4UYHBJWQXXAYWZU4DPG6Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483127506 (505h5m48.354210551s) [noise=true , address=145.79.227.42:4133 , piece_id=D7FIKOZ3OSZQWTUR5Y4FWDPMJYXMPYKUYHU7LSRSM2AAWRVMD5QA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.483731115 (505h5m48.35481415s) [noise=true , address=145.79.227.31:4824 , piece_id=TTCSWOOX2CVI4VUPGDXNWGHLIOO5X42HGYDKHWXDMDSCYECLQYTA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.639322688 (505h5m48.510405733s) [noise=true , address=5.231.73.4:4008 , piece_id=42IJDM3F2V7KTQONAZMZZ5FAQBRXO7AQPNUKIQ56RBZCLWXGUBZQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.760495594 (505h5m48.631578639s) [noise=true , address=45.91.116.5:4308 , piece_id=FVDUENXSDVT5VT5P4T6IMR3DVU5RCJAOQDET2YPYU233ZGOKIXQA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.763286658 (505h5m48.634369693s) [noise=true , address=77.90.31.16:4745 , piece_id=QE4XKAMAAHPVC3SFYA5CAYJQA7ESUVZLEYGYN7UMTDQBFGKCNFJQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:48.771028379 (505h5m48.642111414s) [noise=true , address=77.90.31.21:4906 , piece_id=JPKFQ35B4HLHUG645SWCMRVPTBQJENVJHMTOZHQP5XLKW7W4AX4Q, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.341293323 (505h5m49.212376368s) [noise=true , address=145.79.227.30:4779 , piece_id=QQJ2IJEARUHLTKERKJ2OHUYT2QL4JKQRGF6Y4SWHDG6NNFJNPZMQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446731015 (505h5m49.31781406s) [noise=true , address=145.79.227.43:4192 , piece_id=CRQTQ2UC36WRYWWWLNXP5PTZPFURZVNSMLZYHKJXLAXZVH7R5HVQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: <nil> Dec 26 16:34:49.446757516 (505h5m49.317840561s) [seg_pos={0 0}, upload=443223]: Segment reached optimal threshold of 14 pieces. Dec 26 16:34:49.446947537 (505h5m49.318030572s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.446954180 (505h5m49.318037216s) [noise=true , address=77.90.31.18:4831 , piece_id=IPAPCZLYGRYPHKJQQ3MK4VLTQJTSEJ32G6DASMYQSC3VIT4JXNEA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447148209 (505h5m49.318231244s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447154361 (505h5m49.318237396s) [noise=true , address=45.91.116.8:4064 , piece_id=Q5BSB3XOSSAPAPQTH4UXHPS2UXFHONWFQWLJZ5XM4JTZTEYGMIDA, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447201702 (505h5m49.318284767s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447232551 (505h5m49.318315606s) [noise=true , address=77.90.31.12:4612 , piece_id=KXMEUNAQJDRILWPGSD75R5RZM2FQZVBE2JGB3SLIKKYP6DRZSSEQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447285240 (505h5m49.318368275s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG): protocol: expected piece hash; context canceled; EOF Dec 26 16:34:49.447291793 (505h5m49.318374838s) [noise=true , address=77.90.31.14:4685 , piece_id=S2T2PVMUKEMNS3O6MDKSPW6Y7VFHNWDYFHQJHSDOBMWOI4SWTQXQ, seg_pos={0 0}, upload=443223]: Long tail cancellation due to optimal threshold Dec 26 16:34:49.447309938 (505h5m49.318392973s) [seg_pos={0 0}, upload=443223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:34:49.447375853 (505h5m49.318458898s) [upload=443223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:34:49.447380402 (505h5m49.318463437s) [upload=443223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:34:49.756168298 (505h5m49.627251333s) [upload=443223]: Done uploading segments... Dec 26 16:34:49.756192103 (505h5m49.627275149s) [upload=443223]: Upload finished. err: <nil> Duration: 22.19027659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 10.366884807s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:34:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 10.366910808s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [............X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..] duration: 6.143446955s trace: https://jaeger.dev.storj.io/trace/1000e58d11c5593 FAIL (2025-12-26 16:34:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: QVBEA2BDFXXF5IQ5KPGVLCL3YW37WGD6KGROWSV26F3BG33WC46A) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: GBGWGNZNF6ZXRGPXZC6NNVB6OUJW4BJ74CRTQ2UFJN4RX2462AWQ) context deadline exceeded ======LOGS====== BatchRequestItem_ObjectBegin Dec 26 16:32:55.748472856 (505h3m55.619555891s) [upload=443196]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:32:55.990233411 (505h3m55.861316446s) [seg_pos={0 0}, upload=443196]: Begin upload segment... Dec 26 16:32:55.990279388 (505h3m55.861362423s) [seg_pos={0 0}, upload=443196]: Done begin upload segment. Dec 26 16:32:55.990318382 (505h3m55.861401417s) [noise=true , address=145.79.227.41:4087 , piece_id=PW35BAT23OD7M3FDPMPCXNNP3B5SKPPKRSXM2QJ7ULP7WKSZIM4Q, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.990326097 (505h3m55.861409132s) [noise=true , address=145.79.227.47:4266 , piece_id=5RYXQP3FJZID3RBGL4WFI3IBRJUNCSKGYJJNDV2X74XD5AK7UJYQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.990739674 (505h3m55.861822729s) [noise=true , address=145.79.227.42:4121 , piece_id=UCZXAO5YFCNBGXRT2XEDF2R5HUPOQ7RSZEQXLQ47XGQQNX7GN5ZQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.991159463 (505h3m55.862242509s) [noise=true , address=145.79.227.6:4259 , piece_id=34OGAWRQRAR4DAUNSMXKSBR6HIYI47ITCUXE2QNXNV5ONASH3W4A, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.991535961 (505h3m55.862619006s) [noise=true , address=145.79.227.31:4799 , piece_id=HSOHCLYHJMJYQQQDAT6JSPEJHRQBJ6MKQ44SFJ5Z6R2JEOV6DWUA, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.991938788 (505h3m55.863021823s) [noise=true , address=45.91.116.8:4045 , piece_id=HR5TEJZZR6DPQS2UXT6P3PSYBGAYSZKN6GHXA2PZL6BQZEQSAX5Q, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.992325534 (505h3m55.863408579s) [noise=true , address=145.79.227.51:4411 , piece_id=2NILIF67FCLWX66GJ2K2TBEDK5WMZRRSFWTOJJ7E3OQRLOMGJFPQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.992732580 (505h3m55.863815625s) [noise=true , address=145.79.227.49:4337 , piece_id=ZTTGLNTXF3MTD7KGCINBZJXY5XXIGOLFKLVQ2DT5K4UNRO5ERSAQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.994057483 (505h3m55.865140538s) [noise=true , address=145.79.227.50:4374 , piece_id=XBZNWUVPVIAFHRVTKCKI5DKRTWSTHCJXLKSMSJPOSJYMTJID34DA, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.994280787 (505h3m55.865363832s) [noise=true , address=145.79.227.36:4517 , piece_id=TFUWTSQJLIMMJYQKPJTXPLHLTCEQVPDRSQFU2CY5LXZJRAGOLPPQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.995192643 (505h3m55.866275679s) [noise=true , address=77.90.31.10:4518 , piece_id=NNDCPNYD6E7XGCGQQUJL2JVUWWFDZ5LDJ423EKRPUXQHEFYEKSJQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.996239617 (505h3m55.867322652s) [noise=true , address=77.90.31.15:4700 , piece_id=EGD5XSSWVOVCATQJHCNOQXRLD5K6FDDSIYUER6J7MC5H4LYRZ4GA, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.998518126 (505h3m55.869601171s) [noise=true , address=77.90.31.17:4773 , piece_id=GBGWGNZNF6ZXRGPXZC6NNVB6OUJW4BJ74CRTQ2UFJN4RX2462AWQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:56.000187415 (505h3m55.87127047s) [noise=true , address=77.90.31.13:4629 , piece_id=QVBEA2BDFXXF5IQ5KPGVLCL3YW37WGD6KGROWSV26F3BG33WC46A, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:56.001338356 (505h3m55.872421402s) [noise=true , address=145.79.227.38:4977 , piece_id=QSYMNKN5QE2HDV32SEHN457Y2ZNFDB3MY3WPDPUZYJ2SAZN3BDTQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:56.002218242 (505h3m55.873301297s) [noise=true , address=145.79.227.22:4511 , piece_id=OXZWIZUYMK3SZTWFUMRRKTSBAEGAC27QSOGNQD7MNTKCGYHDDO6Q, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:57.708956739 (505h3m57.580039774s) [upload=443196]: Next returned nil segment Dec 26 16:32:57.708968663 (505h3m57.580051698s) [upload=443196]: Waiting for error group managing segments... Dec 26 16:32:57.996082417 (505h3m57.867165452s) [noise=true , address=145.79.227.36:4517 , piece_id=TFUWTSQJLIMMJYQKPJTXPLHLTCEQVPDRSQFU2CY5LXZJRAGOLPPQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:57.996364233 (505h3m57.867447269s) [noise=true , address=145.79.227.31:4799 , piece_id=HSOHCLYHJMJYQQQDAT6JSPEJHRQBJ6MKQ44SFJ5Z6R2JEOV6DWUA, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:57.997930696 (505h3m57.869013731s) [noise=true , address=145.79.227.42:4121 , piece_id=UCZXAO5YFCNBGXRT2XEDF2R5HUPOQ7RSZEQXLQ47XGQQNX7GN5ZQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:57.998955959 (505h3m57.870039004s) [noise=true , address=145.79.227.50:4374 , piece_id=XBZNWUVPVIAFHRVTKCKI5DKRTWSTHCJXLKSMSJPOSJYMTJID34DA, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.002102520 (505h3m57.873185565s) [noise=true , address=145.79.227.49:4337 , piece_id=ZTTGLNTXF3MTD7KGCINBZJXY5XXIGOLFKLVQ2DT5K4UNRO5ERSAQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.002326246 (505h3m57.873409301s) [noise=true , address=145.79.227.38:4977 , piece_id=QSYMNKN5QE2HDV32SEHN457Y2ZNFDB3MY3WPDPUZYJ2SAZN3BDTQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.008062539 (505h3m57.879145584s) [noise=true , address=145.79.227.22:4511 , piece_id=OXZWIZUYMK3SZTWFUMRRKTSBAEGAC27QSOGNQD7MNTKCGYHDDO6Q, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.013517547 (505h3m57.884600582s) [noise=true , address=145.79.227.51:4411 , piece_id=2NILIF67FCLWX66GJ2K2TBEDK5WMZRRSFWTOJJ7E3OQRLOMGJFPQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.028055369 (505h3m57.899138405s) [noise=true , address=145.79.227.47:4266 , piece_id=5RYXQP3FJZID3RBGL4WFI3IBRJUNCSKGYJJNDV2X74XD5AK7UJYQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.046156557 (505h3m57.917239592s) [noise=true , address=145.79.227.6:4259 , piece_id=34OGAWRQRAR4DAUNSMXKSBR6HIYI47ITCUXE2QNXNV5ONASH3W4A, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.202158942 (505h3m58.073241977s) [noise=true , address=77.90.31.13:4629 , piece_id=QVBEA2BDFXXF5IQ5KPGVLCL3YW37WGD6KGROWSV26F3BG33WC46A, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.277974568 (505h3m58.149057613s) [noise=true , address=77.90.31.17:4773 , piece_id=GBGWGNZNF6ZXRGPXZC6NNVB6OUJW4BJ74CRTQ2UFJN4RX2462AWQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.376870194 (505h3m58.247953229s) [noise=true , address=77.90.31.15:4700 , piece_id=EGD5XSSWVOVCATQJHCNOQXRLD5K6FDDSIYUER6J7MC5H4LYRZ4GA, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.376896934 (505h3m58.247979979s) [seg_pos={0 0}, upload=443196]: Segment reached optimal threshold of 13 pieces. Dec 26 16:32:58.377175755 (505h3m58.2482588s) [noise=true , address=77.90.31.10:4518 , piece_id=NNDCPNYD6E7XGCGQQUJL2JVUWWFDZ5LDJ423EKRPUXQHEFYEKSJQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:49172->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:49172->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49172->77.90.31.10:4518: use of closed network connection Dec 26 16:32:58.377183550 (505h3m58.248266586s) [noise=true , address=77.90.31.10:4518 , piece_id=NNDCPNYD6E7XGCGQQUJL2JVUWWFDZ5LDJ423EKRPUXQHEFYEKSJQ, seg_pos={0 0}, upload=443196]: Long tail cancellation due to optimal threshold Dec 26 16:32:58.377471599 (505h3m58.248554644s) [noise=true , address=45.91.116.8:4045 , piece_id=HR5TEJZZR6DPQS2UXT6P3PSYBGAYSZKN6GHXA2PZL6BQZEQSAX5Q, seg_pos={0 0}, upload=443196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:47566->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:47566->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47566->45.91.116.8:4045: use of closed network connection Dec 26 16:32:58.377480505 (505h3m58.24856354s) [noise=true , address=45.91.116.8:4045 , piece_id=HR5TEJZZR6DPQS2UXT6P3PSYBGAYSZKN6GHXA2PZL6BQZEQSAX5Q, seg_pos={0 0}, upload=443196]: Long tail cancellation due to optimal threshold Dec 26 16:32:58.377585515 (505h3m58.24866856s) [noise=true , address=145.79.227.41:4087 , piece_id=PW35BAT23OD7M3FDPMPCXNNP3B5SKPPKRSXM2QJ7ULP7WKSZIM4Q, seg_pos={0 0}, upload=443196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:50806->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:50806->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50806->145.79.227.41:4087: use of closed network connection Dec 26 16:32:58.377594232 (505h3m58.248677267s) [noise=true , address=145.79.227.41:4087 , piece_id=PW35BAT23OD7M3FDPMPCXNNP3B5SKPPKRSXM2QJ7ULP7WKSZIM4Q, seg_pos={0 0}, upload=443196]: Long tail cancellation due to optimal threshold Dec 26 16:32:58.377612497 (505h3m58.248695542s) [seg_pos={0 0}, upload=443196]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:32:58.377694032 (505h3m58.248777067s) [upload=443196]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:32:58.377699973 (505h3m58.248783018s) [upload=443196]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:32:58.669645940 (505h3m58.540728975s) [upload=443196]: Done uploading segments... Dec 26 16:32:58.669665266 (505h3m58.540748301s) [upload=443196]: Upload finished. err: <nil> Duration:* 6.143471442s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X..X...X.......XX..........X.........X..X...............] duration: 47.362563948s trace: https://jaeger.dev.storj.io/trace/3e3dfe0984f19d4e FAIL (2025-12-26 16:34:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.466998847 (504h51m15.338081892s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467028614 (504h51m15.338111659s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467359594 (504h51m15.338442639s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467908379 (504h51m15.338991424s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.468520054 (504h51m15.339603089s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469055293 (504h51m15.340138328s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469497605 (504h51m15.34058064s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469973642 (504h51m15.341056687s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.470454397 (504h51m15.341537443s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471043630 (504h51m15.342126695s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471470552 (504h51m15.342553587s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471676735 (504h51m15.34275977s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472062900 (504h51m15.343145935s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472369224 (504h51m15.343452259s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472823849 (504h51m15.343906884s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:16.743410939 (504h51m16.614493984s) [upload=443005]: Next returned nil segment Dec 26 16:20:16.743428862 (504h51m16.614511907s) [upload=443005]: Waiting for error group managing segments... Dec 26 16:20:17.091086984 (504h51m16.962170029s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.389229539 (504h51m17.260312594s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.464846438 (504h51m17.335929483s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.542075705 (504h51m17.41315875s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.623422163 (504h51m17.494505208s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.665409184 (504h51m17.536492219s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.301995626 (504h51m18.173078671s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.311693532 (504h51m18.182776567s) [noise=true , address=145.79.227.34:4915 , piece_id=7ZBWNPUF54BEGIXTDRDI3J7HNEZ63MEXKUBMRZW6PFWWNCEAH3AA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.340201235 (504h51m18.21128428s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.379076200 (504h51m18.250159245s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.474362322 (504h51m18.345445367s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.555133063 (504h51m18.426216108s) [noise=true , address=77.90.31.17:4791 , piece_id=QFHBBE4B5DWZU7HTAARXW655MKX4YPKPOXVM77CH5DCPSLZ7UGNA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.707673447 (504h51m18.578756492s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734654664 (504h51m18.605737709s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734678970 (504h51m18.605762005s) [seg_pos={0 0}, upload=443005]: Segment reached optimal threshold of 14 pieces. Dec 26 16:20:18.734942853 (504h51m18.606025888s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.734952842 (504h51m18.606035887s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735555389 (504h51m18.606638434s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection Dec 26 16:20:18.735566701 (504h51m18.606649746s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735681349 (504h51m18.606764394s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735699113 (504h51m18.606782148s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735826605 (504h51m18.60690968s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735837737 (504h51m18.606920782s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735908392 (504h51m18.606991447s) [seg_pos={0 0}, upload=443005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:20:18.736024182 (504h51m18.607107227s) [upload=443005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:20:18.736031275 (504h51m18.60711431s) [upload=443005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:20:19.031573339 (504h51m18.902656384s) [upload=443005]: Done uploading segments... Dec 26 16:20:19.031602895 (504h51m18.90268594s) [upload=443005]: Upload finished. err: <nil> Duration: 47.36258039s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 9.851905597s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:34:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 9.851929984s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [X.........X.............XXX...X...XX.....XX..........X......] duration: 5.001232108s trace: https://jaeger.dev.storj.io/trace/4d8eedf07808cbf FAIL (2025-12-26 16:34:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001256956s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.............X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX.] duration: 6.485521496s trace: https://jaeger.dev.storj.io/trace/1000e58d11c5593 FAIL (2025-12-26 16:34:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: QVBEA2BDFXXF5IQ5KPGVLCL3YW37WGD6KGROWSV26F3BG33WC46A) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: GBGWGNZNF6ZXRGPXZC6NNVB6OUJW4BJ74CRTQ2UFJN4RX2462AWQ) context deadline exceeded ======LOGS====== BatchRequestItem_ObjectBegin Dec 26 16:32:55.748472856 (505h3m55.619555891s) [upload=443196]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:32:55.990233411 (505h3m55.861316446s) [seg_pos={0 0}, upload=443196]: Begin upload segment... Dec 26 16:32:55.990279388 (505h3m55.861362423s) [seg_pos={0 0}, upload=443196]: Done begin upload segment. Dec 26 16:32:55.990318382 (505h3m55.861401417s) [noise=true , address=145.79.227.41:4087 , piece_id=PW35BAT23OD7M3FDPMPCXNNP3B5SKPPKRSXM2QJ7ULP7WKSZIM4Q, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.990326097 (505h3m55.861409132s) [noise=true , address=145.79.227.47:4266 , piece_id=5RYXQP3FJZID3RBGL4WFI3IBRJUNCSKGYJJNDV2X74XD5AK7UJYQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.990739674 (505h3m55.861822729s) [noise=true , address=145.79.227.42:4121 , piece_id=UCZXAO5YFCNBGXRT2XEDF2R5HUPOQ7RSZEQXLQ47XGQQNX7GN5ZQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.991159463 (505h3m55.862242509s) [noise=true , address=145.79.227.6:4259 , piece_id=34OGAWRQRAR4DAUNSMXKSBR6HIYI47ITCUXE2QNXNV5ONASH3W4A, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.991535961 (505h3m55.862619006s) [noise=true , address=145.79.227.31:4799 , piece_id=HSOHCLYHJMJYQQQDAT6JSPEJHRQBJ6MKQ44SFJ5Z6R2JEOV6DWUA, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.991938788 (505h3m55.863021823s) [noise=true , address=45.91.116.8:4045 , piece_id=HR5TEJZZR6DPQS2UXT6P3PSYBGAYSZKN6GHXA2PZL6BQZEQSAX5Q, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.992325534 (505h3m55.863408579s) [noise=true , address=145.79.227.51:4411 , piece_id=2NILIF67FCLWX66GJ2K2TBEDK5WMZRRSFWTOJJ7E3OQRLOMGJFPQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.992732580 (505h3m55.863815625s) [noise=true , address=145.79.227.49:4337 , piece_id=ZTTGLNTXF3MTD7KGCINBZJXY5XXIGOLFKLVQ2DT5K4UNRO5ERSAQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.994057483 (505h3m55.865140538s) [noise=true , address=145.79.227.50:4374 , piece_id=XBZNWUVPVIAFHRVTKCKI5DKRTWSTHCJXLKSMSJPOSJYMTJID34DA, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.994280787 (505h3m55.865363832s) [noise=true , address=145.79.227.36:4517 , piece_id=TFUWTSQJLIMMJYQKPJTXPLHLTCEQVPDRSQFU2CY5LXZJRAGOLPPQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.995192643 (505h3m55.866275679s) [noise=true , address=77.90.31.10:4518 , piece_id=NNDCPNYD6E7XGCGQQUJL2JVUWWFDZ5LDJ423EKRPUXQHEFYEKSJQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.996239617 (505h3m55.867322652s) [noise=true , address=77.90.31.15:4700 , piece_id=EGD5XSSWVOVCATQJHCNOQXRLD5K6FDDSIYUER6J7MC5H4LYRZ4GA, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.998518126 (505h3m55.869601171s) [noise=true , address=77.90.31.17:4773 , piece_id=GBGWGNZNF6ZXRGPXZC6NNVB6OUJW4BJ74CRTQ2UFJN4RX2462AWQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:56.000187415 (505h3m55.87127047s) [noise=true , address=77.90.31.13:4629 , piece_id=QVBEA2BDFXXF5IQ5KPGVLCL3YW37WGD6KGROWSV26F3BG33WC46A, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:56.001338356 (505h3m55.872421402s) [noise=true , address=145.79.227.38:4977 , piece_id=QSYMNKN5QE2HDV32SEHN457Y2ZNFDB3MY3WPDPUZYJ2SAZN3BDTQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:56.002218242 (505h3m55.873301297s) [noise=true , address=145.79.227.22:4511 , piece_id=OXZWIZUYMK3SZTWFUMRRKTSBAEGAC27QSOGNQD7MNTKCGYHDDO6Q, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:57.708956739 (505h3m57.580039774s) [upload=443196]: Next returned nil segment Dec 26 16:32:57.708968663 (505h3m57.580051698s) [upload=443196]: Waiting for error group managing segments... Dec 26 16:32:57.996082417 (505h3m57.867165452s) [noise=true , address=145.79.227.36:4517 , piece_id=TFUWTSQJLIMMJYQKPJTXPLHLTCEQVPDRSQFU2CY5LXZJRAGOLPPQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:57.996364233 (505h3m57.867447269s) [noise=true , address=145.79.227.31:4799 , piece_id=HSOHCLYHJMJYQQQDAT6JSPEJHRQBJ6MKQ44SFJ5Z6R2JEOV6DWUA, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:57.997930696 (505h3m57.869013731s) [noise=true , address=145.79.227.42:4121 , piece_id=UCZXAO5YFCNBGXRT2XEDF2R5HUPOQ7RSZEQXLQ47XGQQNX7GN5ZQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:57.998955959 (505h3m57.870039004s) [noise=true , address=145.79.227.50:4374 , piece_id=XBZNWUVPVIAFHRVTKCKI5DKRTWSTHCJXLKSMSJPOSJYMTJID34DA, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.002102520 (505h3m57.873185565s) [noise=true , address=145.79.227.49:4337 , piece_id=ZTTGLNTXF3MTD7KGCINBZJXY5XXIGOLFKLVQ2DT5K4UNRO5ERSAQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.002326246 (505h3m57.873409301s) [noise=true , address=145.79.227.38:4977 , piece_id=QSYMNKN5QE2HDV32SEHN457Y2ZNFDB3MY3WPDPUZYJ2SAZN3BDTQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.008062539 (505h3m57.879145584s) [noise=true , address=145.79.227.22:4511 , piece_id=OXZWIZUYMK3SZTWFUMRRKTSBAEGAC27QSOGNQD7MNTKCGYHDDO6Q, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.013517547 (505h3m57.884600582s) [noise=true , address=145.79.227.51:4411 , piece_id=2NILIF67FCLWX66GJ2K2TBEDK5WMZRRSFWTOJJ7E3OQRLOMGJFPQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.028055369 (505h3m57.899138405s) [noise=true , address=145.79.227.47:4266 , piece_id=5RYXQP3FJZID3RBGL4WFI3IBRJUNCSKGYJJNDV2X74XD5AK7UJYQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.046156557 (505h3m57.917239592s) [noise=true , address=145.79.227.6:4259 , piece_id=34OGAWRQRAR4DAUNSMXKSBR6HIYI47ITCUXE2QNXNV5ONASH3W4A, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.202158942 (505h3m58.073241977s) [noise=true , address=77.90.31.13:4629 , piece_id=QVBEA2BDFXXF5IQ5KPGVLCL3YW37WGD6KGROWSV26F3BG33WC46A, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.277974568 (505h3m58.149057613s) [noise=true , address=77.90.31.17:4773 , piece_id=GBGWGNZNF6ZXRGPXZC6NNVB6OUJW4BJ74CRTQ2UFJN4RX2462AWQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.376870194 (505h3m58.247953229s) [noise=true , address=77.90.31.15:4700 , piece_id=EGD5XSSWVOVCATQJHCNOQXRLD5K6FDDSIYUER6J7MC5H4LYRZ4GA, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.376896934 (505h3m58.247979979s) [seg_pos={0 0}, upload=443196]: Segment reached optimal threshold of 13 pieces. Dec 26 16:32:58.377175755 (505h3m58.2482588s) [noise=true , address=77.90.31.10:4518 , piece_id=NNDCPNYD6E7XGCGQQUJL2JVUWWFDZ5LDJ423EKRPUXQHEFYEKSJQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:49172->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:49172->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49172->77.90.31.10:4518: use of closed network connection Dec 26 16:32:58.377183550 (505h3m58.248266586s) [noise=true , address=77.90.31.10:4518 , piece_id=NNDCPNYD6E7XGCGQQUJL2JVUWWFDZ5LDJ423EKRPUXQHEFYEKSJQ, seg_pos={0 0}, upload=443196]: Long tail cancellation due to optimal threshold Dec 26 16:32:58.377471599 (505h3m58.248554644s) [noise=true , address=45.91.116.8:4045 , piece_id=HR5TEJZZR6DPQS2UXT6P3PSYBGAYSZKN6GHXA2PZL6BQZEQSAX5Q, seg_pos={0 0}, upload=443196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:47566->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:47566->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47566->45.91.116.8:4045: use of closed network connection Dec 26 16:32:58.377480505 (505h3m58.24856354s) [noise=true , address=45.91.116.8:4045 , piece_id=HR5TEJZZR6DPQS2UXT6P3PSYBGAYSZKN6GHXA2PZL6BQZEQSAX5Q, seg_pos={0 0}, upload=443196]: Long tail cancellation due to optimal threshold Dec 26 16:32:58.377585515 (505h3m58.24866856s) [noise=true , address=145.79.227.41:4087 , piece_id=PW35BAT23OD7M3FDPMPCXNNP3B5SKPPKRSXM2QJ7ULP7WKSZIM4Q, seg_pos={0 0}, upload=443196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:50806->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:50806->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50806->145.79.227.41:4087: use of closed network connection Dec 26 16:32:58.377594232 (505h3m58.248677267s) [noise=true , address=145.79.227.41:4087 , piece_id=PW35BAT23OD7M3FDPMPCXNNP3B5SKPPKRSXM2QJ7ULP7WKSZIM4Q, seg_pos={0 0}, upload=443196]: Long tail cancellation due to optimal threshold Dec 26 16:32:58.377612497 (505h3m58.248695542s) [seg_pos={0 0}, upload=443196]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:32:58.377694032 (505h3m58.248777067s) [upload=443196]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:32:58.377699973 (505h3m58.248783018s) [upload=443196]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:32:58.669645940 (505h3m58.540728975s) [upload=443196]: Done uploading segments... Dec 26 16:32:58.669665266 (505h3m58.540748301s) [upload=443196]: Upload finished. err: <nil> Duration:* 6.485548256s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [X....X..X...X.......XX..........X.........X..X..............] duration: 43.660160035s trace: https://jaeger.dev.storj.io/trace/3e3dfe0984f19d4e FAIL (2025-12-26 16:33:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.466998847 (504h51m15.338081892s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467028614 (504h51m15.338111659s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467359594 (504h51m15.338442639s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467908379 (504h51m15.338991424s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.468520054 (504h51m15.339603089s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469055293 (504h51m15.340138328s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469497605 (504h51m15.34058064s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469973642 (504h51m15.341056687s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.470454397 (504h51m15.341537443s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471043630 (504h51m15.342126695s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471470552 (504h51m15.342553587s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471676735 (504h51m15.34275977s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472062900 (504h51m15.343145935s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472369224 (504h51m15.343452259s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472823849 (504h51m15.343906884s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:16.743410939 (504h51m16.614493984s) [upload=443005]: Next returned nil segment Dec 26 16:20:16.743428862 (504h51m16.614511907s) [upload=443005]: Waiting for error group managing segments... Dec 26 16:20:17.091086984 (504h51m16.962170029s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.389229539 (504h51m17.260312594s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.464846438 (504h51m17.335929483s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.542075705 (504h51m17.41315875s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.623422163 (504h51m17.494505208s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.665409184 (504h51m17.536492219s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.301995626 (504h51m18.173078671s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.311693532 (504h51m18.182776567s) [noise=true , address=145.79.227.34:4915 , piece_id=7ZBWNPUF54BEGIXTDRDI3J7HNEZ63MEXKUBMRZW6PFWWNCEAH3AA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.340201235 (504h51m18.21128428s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.379076200 (504h51m18.250159245s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.474362322 (504h51m18.345445367s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.555133063 (504h51m18.426216108s) [noise=true , address=77.90.31.17:4791 , piece_id=QFHBBE4B5DWZU7HTAARXW655MKX4YPKPOXVM77CH5DCPSLZ7UGNA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.707673447 (504h51m18.578756492s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734654664 (504h51m18.605737709s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734678970 (504h51m18.605762005s) [seg_pos={0 0}, upload=443005]: Segment reached optimal threshold of 14 pieces. Dec 26 16:20:18.734942853 (504h51m18.606025888s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.734952842 (504h51m18.606035887s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735555389 (504h51m18.606638434s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection Dec 26 16:20:18.735566701 (504h51m18.606649746s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735681349 (504h51m18.606764394s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735699113 (504h51m18.606782148s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735826605 (504h51m18.60690968s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735837737 (504h51m18.606920782s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735908392 (504h51m18.606991447s) [seg_pos={0 0}, upload=443005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:20:18.736024182 (504h51m18.607107227s) [upload=443005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:20:18.736031275 (504h51m18.60711431s) [upload=443005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:20:19.031573339 (504h51m18.902656384s) [upload=443005]: Done uploading segments... Dec 26 16:20:19.031602895 (504h51m18.90268594s) [upload=443005]: Upload finished. err: <nil> Duration: 43.660186286s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 8.374052829s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:33:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 8.374086502s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..............X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX] duration: 30.002745323s trace: https://jaeger.dev.storj.io/trace/1000e58d11c5593 FAIL (2025-12-26 16:33:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: QVBEA2BDFXXF5IQ5KPGVLCL3YW37WGD6KGROWSV26F3BG33WC46A) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: GBGWGNZNF6ZXRGPXZC6NNVB6OUJW4BJ74CRTQ2UFJN4RX2462AWQ) context deadline exceeded ======LOGS====== BatchRequestItem_ObjectBegin Dec 26 16:32:55.748472856 (505h3m55.619555891s) [upload=443196]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:32:55.990233411 (505h3m55.861316446s) [seg_pos={0 0}, upload=443196]: Begin upload segment... Dec 26 16:32:55.990279388 (505h3m55.861362423s) [seg_pos={0 0}, upload=443196]: Done begin upload segment. Dec 26 16:32:55.990318382 (505h3m55.861401417s) [noise=true , address=145.79.227.41:4087 , piece_id=PW35BAT23OD7M3FDPMPCXNNP3B5SKPPKRSXM2QJ7ULP7WKSZIM4Q, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.990326097 (505h3m55.861409132s) [noise=true , address=145.79.227.47:4266 , piece_id=5RYXQP3FJZID3RBGL4WFI3IBRJUNCSKGYJJNDV2X74XD5AK7UJYQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.990739674 (505h3m55.861822729s) [noise=true , address=145.79.227.42:4121 , piece_id=UCZXAO5YFCNBGXRT2XEDF2R5HUPOQ7RSZEQXLQ47XGQQNX7GN5ZQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.991159463 (505h3m55.862242509s) [noise=true , address=145.79.227.6:4259 , piece_id=34OGAWRQRAR4DAUNSMXKSBR6HIYI47ITCUXE2QNXNV5ONASH3W4A, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.991535961 (505h3m55.862619006s) [noise=true , address=145.79.227.31:4799 , piece_id=HSOHCLYHJMJYQQQDAT6JSPEJHRQBJ6MKQ44SFJ5Z6R2JEOV6DWUA, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.991938788 (505h3m55.863021823s) [noise=true , address=45.91.116.8:4045 , piece_id=HR5TEJZZR6DPQS2UXT6P3PSYBGAYSZKN6GHXA2PZL6BQZEQSAX5Q, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.992325534 (505h3m55.863408579s) [noise=true , address=145.79.227.51:4411 , piece_id=2NILIF67FCLWX66GJ2K2TBEDK5WMZRRSFWTOJJ7E3OQRLOMGJFPQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.992732580 (505h3m55.863815625s) [noise=true , address=145.79.227.49:4337 , piece_id=ZTTGLNTXF3MTD7KGCINBZJXY5XXIGOLFKLVQ2DT5K4UNRO5ERSAQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.994057483 (505h3m55.865140538s) [noise=true , address=145.79.227.50:4374 , piece_id=XBZNWUVPVIAFHRVTKCKI5DKRTWSTHCJXLKSMSJPOSJYMTJID34DA, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.994280787 (505h3m55.865363832s) [noise=true , address=145.79.227.36:4517 , piece_id=TFUWTSQJLIMMJYQKPJTXPLHLTCEQVPDRSQFU2CY5LXZJRAGOLPPQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.995192643 (505h3m55.866275679s) [noise=true , address=77.90.31.10:4518 , piece_id=NNDCPNYD6E7XGCGQQUJL2JVUWWFDZ5LDJ423EKRPUXQHEFYEKSJQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.996239617 (505h3m55.867322652s) [noise=true , address=77.90.31.15:4700 , piece_id=EGD5XSSWVOVCATQJHCNOQXRLD5K6FDDSIYUER6J7MC5H4LYRZ4GA, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:55.998518126 (505h3m55.869601171s) [noise=true , address=77.90.31.17:4773 , piece_id=GBGWGNZNF6ZXRGPXZC6NNVB6OUJW4BJ74CRTQ2UFJN4RX2462AWQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:56.000187415 (505h3m55.87127047s) [noise=true , address=77.90.31.13:4629 , piece_id=QVBEA2BDFXXF5IQ5KPGVLCL3YW37WGD6KGROWSV26F3BG33WC46A, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:56.001338356 (505h3m55.872421402s) [noise=true , address=145.79.227.38:4977 , piece_id=QSYMNKN5QE2HDV32SEHN457Y2ZNFDB3MY3WPDPUZYJ2SAZN3BDTQ, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:56.002218242 (505h3m55.873301297s) [noise=true , address=145.79.227.22:4511 , piece_id=OXZWIZUYMK3SZTWFUMRRKTSBAEGAC27QSOGNQD7MNTKCGYHDDO6Q, seg_pos={0 0}, upload=443196]: Uploading piece... Dec 26 16:32:57.708956739 (505h3m57.580039774s) [upload=443196]: Next returned nil segment Dec 26 16:32:57.708968663 (505h3m57.580051698s) [upload=443196]: Waiting for error group managing segments... Dec 26 16:32:57.996082417 (505h3m57.867165452s) [noise=true , address=145.79.227.36:4517 , piece_id=TFUWTSQJLIMMJYQKPJTXPLHLTCEQVPDRSQFU2CY5LXZJRAGOLPPQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:57.996364233 (505h3m57.867447269s) [noise=true , address=145.79.227.31:4799 , piece_id=HSOHCLYHJMJYQQQDAT6JSPEJHRQBJ6MKQ44SFJ5Z6R2JEOV6DWUA, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:57.997930696 (505h3m57.869013731s) [noise=true , address=145.79.227.42:4121 , piece_id=UCZXAO5YFCNBGXRT2XEDF2R5HUPOQ7RSZEQXLQ47XGQQNX7GN5ZQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:57.998955959 (505h3m57.870039004s) [noise=true , address=145.79.227.50:4374 , piece_id=XBZNWUVPVIAFHRVTKCKI5DKRTWSTHCJXLKSMSJPOSJYMTJID34DA, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.002102520 (505h3m57.873185565s) [noise=true , address=145.79.227.49:4337 , piece_id=ZTTGLNTXF3MTD7KGCINBZJXY5XXIGOLFKLVQ2DT5K4UNRO5ERSAQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.002326246 (505h3m57.873409301s) [noise=true , address=145.79.227.38:4977 , piece_id=QSYMNKN5QE2HDV32SEHN457Y2ZNFDB3MY3WPDPUZYJ2SAZN3BDTQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.008062539 (505h3m57.879145584s) [noise=true , address=145.79.227.22:4511 , piece_id=OXZWIZUYMK3SZTWFUMRRKTSBAEGAC27QSOGNQD7MNTKCGYHDDO6Q, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.013517547 (505h3m57.884600582s) [noise=true , address=145.79.227.51:4411 , piece_id=2NILIF67FCLWX66GJ2K2TBEDK5WMZRRSFWTOJJ7E3OQRLOMGJFPQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.028055369 (505h3m57.899138405s) [noise=true , address=145.79.227.47:4266 , piece_id=5RYXQP3FJZID3RBGL4WFI3IBRJUNCSKGYJJNDV2X74XD5AK7UJYQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.046156557 (505h3m57.917239592s) [noise=true , address=145.79.227.6:4259 , piece_id=34OGAWRQRAR4DAUNSMXKSBR6HIYI47ITCUXE2QNXNV5ONASH3W4A, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.202158942 (505h3m58.073241977s) [noise=true , address=77.90.31.13:4629 , piece_id=QVBEA2BDFXXF5IQ5KPGVLCL3YW37WGD6KGROWSV26F3BG33WC46A, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.277974568 (505h3m58.149057613s) [noise=true , address=77.90.31.17:4773 , piece_id=GBGWGNZNF6ZXRGPXZC6NNVB6OUJW4BJ74CRTQ2UFJN4RX2462AWQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.376870194 (505h3m58.247953229s) [noise=true , address=77.90.31.15:4700 , piece_id=EGD5XSSWVOVCATQJHCNOQXRLD5K6FDDSIYUER6J7MC5H4LYRZ4GA, seg_pos={0 0}, upload=443196]: Done uploading piece. err: <nil> Dec 26 16:32:58.376896934 (505h3m58.247979979s) [seg_pos={0 0}, upload=443196]: Segment reached optimal threshold of 13 pieces. Dec 26 16:32:58.377175755 (505h3m58.2482588s) [noise=true , address=77.90.31.10:4518 , piece_id=NNDCPNYD6E7XGCGQQUJL2JVUWWFDZ5LDJ423EKRPUXQHEFYEKSJQ, seg_pos={0 0}, upload=443196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:49172->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:49172->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49172->77.90.31.10:4518: use of closed network connection Dec 26 16:32:58.377183550 (505h3m58.248266586s) [noise=true , address=77.90.31.10:4518 , piece_id=NNDCPNYD6E7XGCGQQUJL2JVUWWFDZ5LDJ423EKRPUXQHEFYEKSJQ, seg_pos={0 0}, upload=443196]: Long tail cancellation due to optimal threshold Dec 26 16:32:58.377471599 (505h3m58.248554644s) [noise=true , address=45.91.116.8:4045 , piece_id=HR5TEJZZR6DPQS2UXT6P3PSYBGAYSZKN6GHXA2PZL6BQZEQSAX5Q, seg_pos={0 0}, upload=443196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:47566->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:47566->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47566->45.91.116.8:4045: use of closed network connection Dec 26 16:32:58.377480505 (505h3m58.24856354s) [noise=true , address=45.91.116.8:4045 , piece_id=HR5TEJZZR6DPQS2UXT6P3PSYBGAYSZKN6GHXA2PZL6BQZEQSAX5Q, seg_pos={0 0}, upload=443196]: Long tail cancellation due to optimal threshold Dec 26 16:32:58.377585515 (505h3m58.24866856s) [noise=true , address=145.79.227.41:4087 , piece_id=PW35BAT23OD7M3FDPMPCXNNP3B5SKPPKRSXM2QJ7ULP7WKSZIM4Q, seg_pos={0 0}, upload=443196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:50806->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:50806->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50806->145.79.227.41:4087: use of closed network connection Dec 26 16:32:58.377594232 (505h3m58.248677267s) [noise=true , address=145.79.227.41:4087 , piece_id=PW35BAT23OD7M3FDPMPCXNNP3B5SKPPKRSXM2QJ7ULP7WKSZIM4Q, seg_pos={0 0}, upload=443196]: Long tail cancellation due to optimal threshold Dec 26 16:32:58.377612497 (505h3m58.248695542s) [seg_pos={0 0}, upload=443196]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:32:58.377694032 (505h3m58.248777067s) [upload=443196]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:32:58.377699973 (505h3m58.248783018s) [upload=443196]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:32:58.669645940 (505h3m58.540728975s) [upload=443196]: Done uploading segments... Dec 26 16:32:58.669665266 (505h3m58.540748301s) [upload=443196]: Upload finished. err: <nil> Duration:* 30.002768657s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [XX.........X.............XXX...X...XX.....XX..........X.....] duration: 5.000911488s trace: https://jaeger.dev.storj.io/trace/4d8eedf07808cbf FAIL (2025-12-26 16:33:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000969399s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X....X..X...X.......XX..........X.........X..X.............] duration: 51.899741497s trace: https://jaeger.dev.storj.io/trace/3e3dfe0984f19d4e FAIL (2025-12-26 16:33:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.466998847 (504h51m15.338081892s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467028614 (504h51m15.338111659s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467359594 (504h51m15.338442639s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467908379 (504h51m15.338991424s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.468520054 (504h51m15.339603089s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469055293 (504h51m15.340138328s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469497605 (504h51m15.34058064s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469973642 (504h51m15.341056687s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.470454397 (504h51m15.341537443s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471043630 (504h51m15.342126695s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471470552 (504h51m15.342553587s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471676735 (504h51m15.34275977s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472062900 (504h51m15.343145935s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472369224 (504h51m15.343452259s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472823849 (504h51m15.343906884s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:16.743410939 (504h51m16.614493984s) [upload=443005]: Next returned nil segment Dec 26 16:20:16.743428862 (504h51m16.614511907s) [upload=443005]: Waiting for error group managing segments... Dec 26 16:20:17.091086984 (504h51m16.962170029s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.389229539 (504h51m17.260312594s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.464846438 (504h51m17.335929483s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.542075705 (504h51m17.41315875s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.623422163 (504h51m17.494505208s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.665409184 (504h51m17.536492219s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.301995626 (504h51m18.173078671s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.311693532 (504h51m18.182776567s) [noise=true , address=145.79.227.34:4915 , piece_id=7ZBWNPUF54BEGIXTDRDI3J7HNEZ63MEXKUBMRZW6PFWWNCEAH3AA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.340201235 (504h51m18.21128428s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.379076200 (504h51m18.250159245s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.474362322 (504h51m18.345445367s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.555133063 (504h51m18.426216108s) [noise=true , address=77.90.31.17:4791 , piece_id=QFHBBE4B5DWZU7HTAARXW655MKX4YPKPOXVM77CH5DCPSLZ7UGNA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.707673447 (504h51m18.578756492s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734654664 (504h51m18.605737709s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734678970 (504h51m18.605762005s) [seg_pos={0 0}, upload=443005]: Segment reached optimal threshold of 14 pieces. Dec 26 16:20:18.734942853 (504h51m18.606025888s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.734952842 (504h51m18.606035887s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735555389 (504h51m18.606638434s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection Dec 26 16:20:18.735566701 (504h51m18.606649746s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735681349 (504h51m18.606764394s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735699113 (504h51m18.606782148s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735826605 (504h51m18.60690968s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735837737 (504h51m18.606920782s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735908392 (504h51m18.606991447s) [seg_pos={0 0}, upload=443005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:20:18.736024182 (504h51m18.607107227s) [upload=443005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:20:18.736031275 (504h51m18.60711431s) [upload=443005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:20:19.031573339 (504h51m18.902656384s) [upload=443005]: Done uploading segments... Dec 26 16:20:19.031602895 (504h51m18.90268594s) [upload=443005]: Upload finished. err: <nil> Duration: 51.899766143s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.XX.........X.............XXX...X...XX.....XX..........X....] duration: 5.001128362s trace: https://jaeger.dev.storj.io/trace/4d8eedf07808cbf FAIL (2025-12-26 16:32:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.0011491s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...............X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XX] duration: 30.002298713s trace: https://jaeger.dev.storj.io/trace/11bb7d8e93253651 FAIL (2025-12-26 16:32:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 5GOTVODIFH7M2R3Z7PO2D3JJQE4ARNAIMDIBLKXLYZ2M5O7LK4GQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: HEB3CXNCD5Q7Y74XVJQVPSXUAVPNS7ESR2LXNGR3VRGVII42IIXA) context deadline exceeded ======LOGS====== one starting upload Dec 26 16:32:03.809658639 (505h3m3.680741694s) [upload=443180]: Uploading segments... Dec 26 16:32:03.810590082 (505h3m3.681673128s) [upload=443180]: Got next segment. Inline: false Dec 26 16:32:03.810656439 (505h3m3.681739474s) [upload=443180]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:32:03.810665416 (505h3m3.681748451s) [upload=443180]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:32:04.072185325 (505h3m3.94326837s) [seg_pos={0 0}, upload=443180]: Begin upload segment... Dec 26 16:32:04.072233406 (505h3m3.943316451s) [seg_pos={0 0}, upload=443180]: Done begin upload segment. Dec 26 16:32:04.072260799 (505h3m3.943343844s) [noise=true , address=45.91.116.7:4010 , piece_id=MTVZQVM2C2ULZTGAYLAGSTRUG4GYBFTOY4YCUO2UDY7IGOLDJENA, seg_pos={0 0}, upload=443180]: Uploading piece... Dec 26 16:32:04.072652074 (505h3m3.943735119s) [noise=true , address=145.79.227.37:4565 , piece_id=YPJWMZGKZA22TZZBJ47W47M22WPAQZ4COLAWTUPRNX7XQJXBG4HA, seg_pos={0 0}, upload=443180]: Uploading piece... Dec 26 16:32:04.073283235 (505h3m3.94436628s) [noise=true , address=145.79.227.8:4330 , piece_id=UQID24T2YL2XGZQPJX37HS4LL2DDFGCEE5TSYODJVXFDJJYU5QPA, seg_pos={0 0}, upload=443180]: Uploading piece... Dec 26 16:32:04.073648271 (505h3m3.944731306s) [noise=true , address=145.79.227.48:4301 , piece_id=72C5TTLKLQ6MVN3YU3PKBWLIH26FLWOLFXCFH2C4OP6MZD7GXY2A, seg_pos={0 0}, upload=443180]: Uploading piece... Dec 26 16:32:04.074089010 (505h3m3.945172055s) [noise=true , address=145.79.227.39:4013 , piece_id=CSJGMPIEBTQPGSTYRIO2UDB767R6RWMP4SFOFRFPI2HYGVUJS63A, seg_pos={0 0}, upload=443180]: Uploading piece... Dec 26 16:32:04.074103568 (505h3m3.945186603s) [noise=true , address=77.90.31.16:4736 , piece_id=5GOTVODIFH7M2R3Z7PO2D3JJQE4ARNAIMDIBLKXLYZ2M5O7LK4GQ, seg_pos={0 0}, upload=443180]: Uploading piece... Dec 26 16:32:04.075832160 (505h3m3.946915195s) [noise=true , address=45.91.116.11:4119 , piece_id=VNVAHHAUCW47W6N5FI62YPUER6H4QPLE6E2MXFLGLF6KBGVQSYDA, seg_pos={0 0}, upload=443180]: Uploading piece... Dec 26 16:32:04.075952879 (505h3m3.947035924s) [noise=true , address=145.79.227.51:4411 , piece_id=GPG3F7LLAQI7ANAUDHGWBFZPYTVU64I2JUUIPNNAFDPWQOUY2HXA, seg_pos={0 0}, upload=443180]: Uploading piece... Dec 26 16:32:04.076527604 (505h3m3.947610649s) [noise=true , address=45.91.116.4:4937 , piece_id=MXHYTW5565W67JT5W5RTMAJ5HN5CSRIKUGYIKHUHA3FKMHD653DA, seg_pos={0 0}, upload=443180]: Uploading piece... Dec 26 16:32:04.076992940 (505h3m3.948075985s) [noise=true , address=45.91.116.5:4298 , piece_id=HEB3CXNCD5Q7Y74XVJQVPSXUAVPNS7ESR2LXNGR3VRGVII42IIXA, seg_pos={0 0}, upload=443180]: Uploading piece... Dec 26 16:32:04.077417317 (505h3m3.948500353s) [noise=true , address=145.79.227.47:4265 , piece_id=LRJX7XCJMFF5OIYZQTZ6RX65S3GAFB3OFDQOYMO2JOKR5EHIQ3UQ, seg_pos={0 0}, upload=443180]: Uploading piece... Dec 26 16:32:04.078133552 (505h3m3.949216587s) [noise=true , address=145.79.227.4:4187 , piece_id=FPEH2XTASHAMUTB25DLN5YWX3K6ZVURJQ6SNONX22C7Z5OCXTLFA, seg_pos={0 0}, upload=443180]: Uploading piece... Dec 26 16:32:04.078569191 (505h3m3.949652236s) [noise=true , address=145.79.227.2:4113 , piece_id=PWCRL7ZIOGQDSHA7PS2ESJJEAHRUI3JEFZK477FKLOJ3JPACLAVA, seg_pos={0 0}, upload=443180]: Uploading piece... Dec 26 16:32:04.079040739 (505h3m3.950123774s) [noise=true , address=145.79.227.36:4518 , piece_id=BJ32EZ252ZHXJBY5QES62SBNKN637BFMYVEO2ND35DYVQYAP7DJQ, seg_pos={0 0}, upload=443180]: Uploading piece... Dec 26 16:32:04.080200177 (505h3m3.951283242s) [noise=true , address=145.79.227.45:4229 , piece_id=BYQI6ELP2NQTZD7QPJEZB63ZVT3GXC6U7AV23AROY2TPXSVV67EA, seg_pos={0 0}, upload=443180]: Uploading piece... Dec 26 16:32:04.081020359 (505h3m3.952103404s) [noise=true , address=45.91.116.8:4045 , piece_id=UUBJGIKN54SXUFF3H25CTPAOZYIY5C72AJ6GJVXT4AKIF6BH5AVA, seg_pos={0 0}, upload=443180]: Uploading piece... Dec 26 16:32:05.754175908 (505h3m5.625258943s) [upload=443180]: Next returned nil segment Dec 26 16:32:05.754228088 (505h3m5.625311143s) [upload=443180]: Waiting for error group managing segments... Dec 26 16:32:06.083246286 (505h3m5.954329331s) [noise=true , address=145.79.227.45:4229 , piece_id=BYQI6ELP2NQTZD7QPJEZB63ZVT3GXC6U7AV23AROY2TPXSVV67EA, seg_pos={0 0}, upload=443180]: Done uploading piece. err: <nil> Dec 26 16:32:06.086065334 (505h3m5.957148379s) [noise=true , address=145.79.227.39:4013 , piece_id=CSJGMPIEBTQPGSTYRIO2UDB767R6RWMP4SFOFRFPI2HYGVUJS63A, seg_pos={0 0}, upload=443180]: Done uploading piece. err: <nil> Dec 26 16:32:06.088656808 (505h3m5.959739843s) [noise=true , address=145.79.227.51:4411 , piece_id=GPG3F7LLAQI7ANAUDHGWBFZPYTVU64I2JUUIPNNAFDPWQOUY2HXA, seg_pos={0 0}, upload=443180]: Done uploading piece. err: <nil> Dec 26 16:32:06.093014256 (505h3m5.964097291s) [noise=true , address=145.79.227.48:4301 , piece_id=72C5TTLKLQ6MVN3YU3PKBWLIH26FLWOLFXCFH2C4OP6MZD7GXY2A, seg_pos={0 0}, upload=443180]: Done uploading piece. err: <nil> Dec 26 16:32:06.096158824 (505h3m5.967241869s) [noise=true , address=145.79.227.8:4330 , piece_id=UQID24T2YL2XGZQPJX37HS4LL2DDFGCEE5TSYODJVXFDJJYU5QPA, seg_pos={0 0}, upload=443180]: Done uploading piece. err: <nil> Dec 26 16:32:06.100262448 (505h3m5.971345493s) [noise=true , address=145.79.227.37:4565 , piece_id=YPJWMZGKZA22TZZBJ47W47M22WPAQZ4COLAWTUPRNX7XQJXBG4HA, seg_pos={0 0}, upload=443180]: Done uploading piece. err: <nil> Dec 26 16:32:06.105284010 (505h3m5.976367045s) [noise=true , address=145.79.227.36:4518 , piece_id=BJ32EZ252ZHXJBY5QES62SBNKN637BFMYVEO2ND35DYVQYAP7DJQ, seg_pos={0 0}, upload=443180]: Done uploading piece. err: <nil> Dec 26 16:32:06.109982326 (505h3m5.981065371s) [noise=true , address=145.79.227.47:4265 , piece_id=LRJX7XCJMFF5OIYZQTZ6RX65S3GAFB3OFDQOYMO2JOKR5EHIQ3UQ, seg_pos={0 0}, upload=443180]: Done uploading piece. err: <nil> Dec 26 16:32:06.151757794 (505h3m6.022840829s) [noise=true , address=145.79.227.2:4113 , piece_id=PWCRL7ZIOGQDSHA7PS2ESJJEAHRUI3JEFZK477FKLOJ3JPACLAVA, seg_pos={0 0}, upload=443180]: Done uploading piece. err: <nil> Dec 26 16:32:06.365285799 (505h3m6.236368834s) [noise=true , address=77.90.31.16:4736 , piece_id=5GOTVODIFH7M2R3Z7PO2D3JJQE4ARNAIMDIBLKXLYZ2M5O7LK4GQ, seg_pos={0 0}, upload=443180]: Done uploading piece. err: <nil> Dec 26 16:32:06.474766429 (505h3m6.345849474s) [noise=true , address=45.91.116.8:4045 , piece_id=UUBJGIKN54SXUFF3H25CTPAOZYIY5C72AJ6GJVXT4AKIF6BH5AVA, seg_pos={0 0}, upload=443180]: Done uploading piece. err: <nil> Dec 26 16:32:06.544926285 (505h3m6.41600933s) [noise=true , address=45.91.116.5:4298 , piece_id=HEB3CXNCD5Q7Y74XVJQVPSXUAVPNS7ESR2LXNGR3VRGVII42IIXA, seg_pos={0 0}, upload=443180]: Done uploading piece. err: <nil> Dec 26 16:32:06.644312855 (505h3m6.5153959s) [noise=true , address=45.91.116.4:4937 , piece_id=MXHYTW5565W67JT5W5RTMAJ5HN5CSRIKUGYIKHUHA3FKMHD653DA, seg_pos={0 0}, upload=443180]: Done uploading piece. err: <nil> Dec 26 16:32:06.644338493 (505h3m6.515421538s) [seg_pos={0 0}, upload=443180]: Segment reached optimal threshold of 13 pieces. Dec 26 16:32:06.644905654 (505h3m6.515988689s) [noise=true , address=45.91.116.11:4119 , piece_id=VNVAHHAUCW47W6N5FI62YPUER6H4QPLE6E2MXFLGLF6KBGVQSYDA, seg_pos={0 0}, upload=443180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:57140->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:57140->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57140->45.91.116.11:4119: use of closed network connection Dec 26 16:32:06.644918097 (505h3m6.516001132s) [noise=true , address=45.91.116.11:4119 , piece_id=VNVAHHAUCW47W6N5FI62YPUER6H4QPLE6E2MXFLGLF6KBGVQSYDA, seg_pos={0 0}, upload=443180]: Long tail cancellation due to optimal threshold Dec 26 16:32:06.645046773 (505h3m6.516129808s) [noise=true , address=145.79.227.4:4187 , piece_id=FPEH2XTASHAMUTB25DLN5YWX3K6ZVURJQ6SNONX22C7Z5OCXTLFA, seg_pos={0 0}, upload=443180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:37752->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:37752->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37752->145.79.227.4:4187: use of closed network connection Dec 26 16:32:06.645053345 (505h3m6.51613638s) [noise=true , address=145.79.227.4:4187 , piece_id=FPEH2XTASHAMUTB25DLN5YWX3K6ZVURJQ6SNONX22C7Z5OCXTLFA, seg_pos={0 0}, upload=443180]: Long tail cancellation due to optimal threshold Dec 26 16:32:06.645348025 (505h3m6.516431061s) [noise=true , address=45.91.116.7:4010 , piece_id=MTVZQVM2C2ULZTGAYLAGSTRUG4GYBFTOY4YCUO2UDY7IGOLDJENA, seg_pos={0 0}, upload=443180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 26 16:32:06.645373114 (505h3m6.516456169s) [noise=true , address=45.91.116.7:4010 , piece_id=MTVZQVM2C2ULZTGAYLAGSTRUG4GYBFTOY4YCUO2UDY7IGOLDJENA, seg_pos={0 0}, upload=443180]: Long tail cancellation due to optimal threshold Dec 26 16:32:06.645395206 (505h3m6.516478251s) [seg_pos={0 0}, upload=443180]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:32:06.645477753 (505h3m6.516560798s) [upload=443180]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:32:06.645483353 (505h3m6.516566388s) [upload=443180]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:32:06.943481684 (505h3m6.814564719s) [upload=443180]: Done uploading segments... Dec 26 16:32:06.943501733 (505h3m6.814584778s) [upload=443180]: Upload finished. err: <nil> Duration: 30.002327979s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 11.668448664s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:32:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 11.668530389s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..XX.........X.............XXX...X...XX.....XX..........X...] duration: 5.000467172s trace: https://jaeger.dev.storj.io/trace/4d8eedf07808cbf FAIL (2025-12-26 16:31:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000494523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X....X..X...X.......XX..........X.........X..X............] duration: 43.464737406s trace: https://jaeger.dev.storj.io/trace/3e3dfe0984f19d4e FAIL (2025-12-26 16:31:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.466998847 (504h51m15.338081892s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467028614 (504h51m15.338111659s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467359594 (504h51m15.338442639s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467908379 (504h51m15.338991424s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.468520054 (504h51m15.339603089s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469055293 (504h51m15.340138328s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469497605 (504h51m15.34058064s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469973642 (504h51m15.341056687s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.470454397 (504h51m15.341537443s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471043630 (504h51m15.342126695s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471470552 (504h51m15.342553587s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471676735 (504h51m15.34275977s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472062900 (504h51m15.343145935s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472369224 (504h51m15.343452259s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472823849 (504h51m15.343906884s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:16.743410939 (504h51m16.614493984s) [upload=443005]: Next returned nil segment Dec 26 16:20:16.743428862 (504h51m16.614511907s) [upload=443005]: Waiting for error group managing segments... Dec 26 16:20:17.091086984 (504h51m16.962170029s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.389229539 (504h51m17.260312594s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.464846438 (504h51m17.335929483s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.542075705 (504h51m17.41315875s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.623422163 (504h51m17.494505208s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.665409184 (504h51m17.536492219s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.301995626 (504h51m18.173078671s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.311693532 (504h51m18.182776567s) [noise=true , address=145.79.227.34:4915 , piece_id=7ZBWNPUF54BEGIXTDRDI3J7HNEZ63MEXKUBMRZW6PFWWNCEAH3AA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.340201235 (504h51m18.21128428s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.379076200 (504h51m18.250159245s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.474362322 (504h51m18.345445367s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.555133063 (504h51m18.426216108s) [noise=true , address=77.90.31.17:4791 , piece_id=QFHBBE4B5DWZU7HTAARXW655MKX4YPKPOXVM77CH5DCPSLZ7UGNA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.707673447 (504h51m18.578756492s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734654664 (504h51m18.605737709s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734678970 (504h51m18.605762005s) [seg_pos={0 0}, upload=443005]: Segment reached optimal threshold of 14 pieces. Dec 26 16:20:18.734942853 (504h51m18.606025888s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.734952842 (504h51m18.606035887s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735555389 (504h51m18.606638434s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection Dec 26 16:20:18.735566701 (504h51m18.606649746s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735681349 (504h51m18.606764394s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735699113 (504h51m18.606782148s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735826605 (504h51m18.60690968s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735837737 (504h51m18.606920782s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735908392 (504h51m18.606991447s) [seg_pos={0 0}, upload=443005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:20:18.736024182 (504h51m18.607107227s) [upload=443005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:20:18.736031275 (504h51m18.60711431s) [upload=443005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:20:19.031573339 (504h51m18.902656384s) [upload=443005]: Done uploading segments... Dec 26 16:20:19.031602895 (504h51m18.90268594s) [upload=443005]: Upload finished. err: <nil> Duration: 43.464763455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [................X.....X..XX..XXX.X...XX.X.XX.X....XX...X...X] duration: 30.003266134s trace: https://jaeger.dev.storj.io/trace/3a1b9ad419ba9c2e FAIL (2025-12-26 16:31:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Done starting upload Dec 26 16:31:02.520243985 (505h2m2.39132703s) [upload=443167]: Uploading segments... Dec 26 16:31:02.520373302 (505h2m2.391456347s) [upload=443167]: Got next segment. Inline: false Dec 26 16:31:02.520451420 (505h2m2.391534455s) [upload=443167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:31:02.520457933 (505h2m2.391540978s) [upload=443167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:31:02.766182605 (505h2m2.63726566s) [seg_pos={0 0}, upload=443167]: Begin upload segment... Dec 26 16:31:02.766244784 (505h2m2.637327829s) [seg_pos={0 0}, upload=443167]: Done begin upload segment. Dec 26 16:31:02.766282245 (505h2m2.63736529s) [noise=true , address=145.79.227.47:4265 , piece_id=UJGWTN2YTG42UXP5MSXBH3GG3YFN7PCMH5L6OCSINBJAUYE2CLRA, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.766388217 (505h2m2.637471272s) [noise=true , address=145.79.227.13:4802 , piece_id=AF75MEXCINEL25WLU63QRY7AJ5WJ2Y63PY7OM6AJY4YTF6ANAMQA, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.766962440 (505h2m2.638045486s) [noise=true , address=145.79.227.46:4603 , piece_id=IXTJYVDACXWSDDV2CEYRRX7ZXZPZXFPNCGKFCMZ2B5MNMJXYVMGA, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.767331603 (505h2m2.638414648s) [noise=true , address=145.79.227.50:4375 , piece_id=IRG5LENZXD3JOXJUD43IAMO7GS4AY526F3ZFQXRZZKX57SWVOL5Q, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.767643537 (505h2m2.638726572s) [noise=true , address=145.79.227.2:4113 , piece_id=ZICIDCM7YK4JU3W3KF3E5IPBGPOE533P5PAK56NHBWSVYOKXAWDA, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.767946694 (505h2m2.639029729s) [noise=true , address=45.91.116.6:4973 , piece_id=YRRWYEAE2JP5HFFIOO3SBDVG7GBJHW42ELT7ITXEL3GYUUUZLJGA, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.768266263 (505h2m2.639349308s) [noise=true , address=145.79.227.11:4729 , piece_id=NPOSBCBL3ZM57WPZ5QXHISBZALEDTX62D2XGKYES3VJ7LDOC4QOA, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.768623412 (505h2m2.639706458s) [noise=true , address=45.91.116.5:4300 , piece_id=JRSZZVNIDTMLTBPJ4R2RFKO3TOCDM2Q32W2D7LROVV7VVZFLMDUQ, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.769083660 (505h2m2.640166705s) [noise=true , address=45.91.116.8:4046 , piece_id=CRM3FWNW3FKTB2ENA3MJX5BIUNOAE5U3KQ3MN3JHNMIVL7GXZVFA, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.771359813 (505h2m2.642442859s) [noise=true , address=145.79.227.48:4303 , piece_id=AFMR3KNKGEQI3FN2WUO5NVAIBBO6LO2F26FCBIB2NPH2S6GBZ65A, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.772834671 (505h2m2.643917707s) [noise=true , address=145.79.227.15:4874 , piece_id=K2VZXRDFGWNRDCQ65HRGFIZJ44ERHVGGTBDE5RIXVBNMF3JOT2HQ, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.774957223 (505h2m2.646040268s) [noise=true , address=145.79.227.37:4566 , piece_id=ESWX57X4F35FB67TE7LQY5IBZN4RQS5I57WRWXDC65QDEFJDMF2A, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.776754616 (505h2m2.647837661s) [noise=true , address=45.91.116.13:4154 , piece_id=RREHTH3WXYI2T3U3NVTTAQTUFF7PVCOBVFRGQ5PTI4IZL62OUQOA, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.778703467 (505h2m2.649786512s) [noise=true , address=145.79.227.52:4445 , piece_id=THBDPA46KZ2TUS3SMXEALVOZ7WHJ22AOCJ6V7BJ7RPYEGYQSRJHA, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.780156965 (505h2m2.65124001s) [noise=true , address=145.79.227.32:4836 , piece_id=UWEKC3WL5EO2ZBG2PIZSRXBTJII7OCO5BWGWUV24BSQHVXI3NDXA, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.781481337 (505h2m2.652564392s) [noise=true , address=145.79.227.16:4911 , piece_id=OJAMDI4LBR6XJ564IXBMEY644IBVZANNVWPMITHHW4LTJQATDF6Q, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:04.482347452 (505h2m4.353430487s) [upload=443167]: Next returned nil segment Dec 26 16:31:04.482371098 (505h2m4.353454143s) [upload=443167]: Waiting for error group managing segments... Dec 26 16:31:04.782650442 (505h2m4.653733487s) [noise=true , address=145.79.227.11:4729 , piece_id=NPOSBCBL3ZM57WPZ5QXHISBZALEDTX62D2XGKYES3VJ7LDOC4QOA, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:04.786628867 (505h2m4.657711903s) [noise=true , address=145.79.227.13:4802 , piece_id=AF75MEXCINEL25WLU63QRY7AJ5WJ2Y63PY7OM6AJY4YTF6ANAMQA, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:04.789265528 (505h2m4.660348573s) [noise=true , address=145.79.227.32:4836 , piece_id=UWEKC3WL5EO2ZBG2PIZSRXBTJII7OCO5BWGWUV24BSQHVXI3NDXA, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:04.791139075 (505h2m4.66222212s) [noise=true , address=145.79.227.50:4375 , piece_id=IRG5LENZXD3JOXJUD43IAMO7GS4AY526F3ZFQXRZZKX57SWVOL5Q, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:04.794570309 (505h2m4.665653354s) [noise=true , address=145.79.227.48:4303 , piece_id=AFMR3KNKGEQI3FN2WUO5NVAIBBO6LO2F26FCBIB2NPH2S6GBZ65A, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:04.801382942 (505h2m4.672465977s) [noise=true , address=145.79.227.46:4603 , piece_id=IXTJYVDACXWSDDV2CEYRRX7ZXZPZXFPNCGKFCMZ2B5MNMJXYVMGA, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:04.801510975 (505h2m4.67259401s) [noise=true , address=145.79.227.47:4265 , piece_id=UJGWTN2YTG42UXP5MSXBH3GG3YFN7PCMH5L6OCSINBJAUYE2CLRA, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:04.803507187 (505h2m4.674590232s) [noise=true , address=145.79.227.52:4445 , piece_id=THBDPA46KZ2TUS3SMXEALVOZ7WHJ22AOCJ6V7BJ7RPYEGYQSRJHA, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:04.805518878 (505h2m4.676601913s) [noise=true , address=145.79.227.37:4566 , piece_id=ESWX57X4F35FB67TE7LQY5IBZN4RQS5I57WRWXDC65QDEFJDMF2A, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:04.848540248 (505h2m4.719623313s) [noise=true , address=145.79.227.16:4911 , piece_id=OJAMDI4LBR6XJ564IXBMEY644IBVZANNVWPMITHHW4LTJQATDF6Q, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:04.858805444 (505h2m4.729888499s) [noise=true , address=145.79.227.15:4874 , piece_id=K2VZXRDFGWNRDCQ65HRGFIZJ44ERHVGGTBDE5RIXVBNMF3JOT2HQ, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:05.003427269 (505h2m4.874510304s) [noise=true , address=45.91.116.13:4154 , piece_id=RREHTH3WXYI2T3U3NVTTAQTUFF7PVCOBVFRGQ5PTI4IZL62OUQOA, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:05.036112778 (505h2m4.907195823s) [noise=true , address=45.91.116.6:4973 , piece_id=YRRWYEAE2JP5HFFIOO3SBDVG7GBJHW42ELT7ITXEL3GYUUUZLJGA, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:05.036133989 (505h2m4.907217034s) [seg_pos={0 0}, upload=443167]: Segment reached optimal threshold of 13 pieces. Dec 26 16:31:05.036337076 (505h2m4.907420111s) [noise=true , address=145.79.227.2:4113 , piece_id=ZICIDCM7YK4JU3W3KF3E5IPBGPOE533P5PAK56NHBWSVYOKXAWDA, seg_pos={0 0}, upload=443167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:36182->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:36182->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36182->145.79.227.2:4113: use of closed network connection Dec 26 16:31:05.036346293 (505h2m4.907429338s) [noise=true , address=145.79.227.2:4113 , piece_id=ZICIDCM7YK4JU3W3KF3E5IPBGPOE533P5PAK56NHBWSVYOKXAWDA, seg_pos={0 0}, upload=443167]: Long tail cancellation due to optimal threshold Dec 26 16:31:05.036435754 (505h2m4.907518789s) [noise=true , address=45.91.116.8:4046 , piece_id=CRM3FWNW3FKTB2ENA3MJX5BIUNOAE5U3KQ3MN3JHNMIVL7GXZVFA, seg_pos={0 0}, upload=443167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:35748->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:35748->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35748->45.91.116.8:4046: use of closed network connection Dec 26 16:31:05.036451423 (505h2m4.907534468s) [noise=true , address=45.91.116.8:4046 , piece_id=CRM3FWNW3FKTB2ENA3MJX5BIUNOAE5U3KQ3MN3JHNMIVL7GXZVFA, seg_pos={0 0}, upload=443167]: Long tail cancellation due to optimal threshold Dec 26 16:31:05.036670089 (505h2m4.907753134s) [noise=true , address=45.91.116.5:4300 , piece_id=JRSZZVNIDTMLTBPJ4R2RFKO3TOCDM2Q32W2D7LROVV7VVZFLMDUQ, seg_pos={0 0}, upload=443167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 26 16:31:05.036677633 (505h2m4.907760668s) [noise=true , address=45.91.116.5:4300 , piece_id=JRSZZVNIDTMLTBPJ4R2RFKO3TOCDM2Q32W2D7LROVV7VVZFLMDUQ, seg_pos={0 0}, upload=443167]: Long tail cancellation due to optimal threshold Dec 26 16:31:05.036697852 (505h2m4.907780887s) [seg_pos={0 0}, upload=443167]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:31:05.036765231 (505h2m4.907848266s) [upload=443167]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:31:05.036769849 (505h2m4.907852894s) [upload=443167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:31:05.339840279 (505h2m5.210923314s) [upload=443167]: Done uploading segments... Dec 26 16:31:05.339868072 (505h2m5.210951107s) [upload=443167]: Upload finished. err: <nil> Duration: 30.003296583s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 8.858772739s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:31:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 8.858807375s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...XX.........X.............XXX...X...XX.....XX..........X..] duration: 3.40084378s trace: https://jaeger.dev.storj.io/trace/4d8eedf07808cbf FAIL (2025-12-26 16:31:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.400877083s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X....X..X...X.......XX..........X.........X..X...........] duration: 53.516150561s trace: https://jaeger.dev.storj.io/trace/3e3dfe0984f19d4e FAIL (2025-12-26 16:31:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.466998847 (504h51m15.338081892s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467028614 (504h51m15.338111659s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467359594 (504h51m15.338442639s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467908379 (504h51m15.338991424s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.468520054 (504h51m15.339603089s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469055293 (504h51m15.340138328s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469497605 (504h51m15.34058064s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469973642 (504h51m15.341056687s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.470454397 (504h51m15.341537443s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471043630 (504h51m15.342126695s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471470552 (504h51m15.342553587s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471676735 (504h51m15.34275977s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472062900 (504h51m15.343145935s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472369224 (504h51m15.343452259s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472823849 (504h51m15.343906884s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:16.743410939 (504h51m16.614493984s) [upload=443005]: Next returned nil segment Dec 26 16:20:16.743428862 (504h51m16.614511907s) [upload=443005]: Waiting for error group managing segments... Dec 26 16:20:17.091086984 (504h51m16.962170029s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.389229539 (504h51m17.260312594s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.464846438 (504h51m17.335929483s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.542075705 (504h51m17.41315875s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.623422163 (504h51m17.494505208s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.665409184 (504h51m17.536492219s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.301995626 (504h51m18.173078671s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.311693532 (504h51m18.182776567s) [noise=true , address=145.79.227.34:4915 , piece_id=7ZBWNPUF54BEGIXTDRDI3J7HNEZ63MEXKUBMRZW6PFWWNCEAH3AA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.340201235 (504h51m18.21128428s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.379076200 (504h51m18.250159245s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.474362322 (504h51m18.345445367s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.555133063 (504h51m18.426216108s) [noise=true , address=77.90.31.17:4791 , piece_id=QFHBBE4B5DWZU7HTAARXW655MKX4YPKPOXVM77CH5DCPSLZ7UGNA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.707673447 (504h51m18.578756492s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734654664 (504h51m18.605737709s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734678970 (504h51m18.605762005s) [seg_pos={0 0}, upload=443005]: Segment reached optimal threshold of 14 pieces. Dec 26 16:20:18.734942853 (504h51m18.606025888s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.734952842 (504h51m18.606035887s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735555389 (504h51m18.606638434s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection Dec 26 16:20:18.735566701 (504h51m18.606649746s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735681349 (504h51m18.606764394s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735699113 (504h51m18.606782148s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735826605 (504h51m18.60690968s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735837737 (504h51m18.606920782s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735908392 (504h51m18.606991447s) [seg_pos={0 0}, upload=443005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:20:18.736024182 (504h51m18.607107227s) [upload=443005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:20:18.736031275 (504h51m18.60711431s) [upload=443005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:20:19.031573339 (504h51m18.902656384s) [upload=443005]: Done uploading segments... Dec 26 16:20:19.031602895 (504h51m18.90268594s) [upload=443005]: Upload finished. err: <nil> Duration: 53.51626503s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 7.5092174s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:31:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 7.509244913s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 20.691476905s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:30:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 20.691503786s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.................X.....X..XX..XXX.X...XX.X.XX.X....XX...X...] duration: 6.175656838s trace: https://jaeger.dev.storj.io/trace/7db2f5600219fec2 FAIL (2025-12-26 16:30:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8 (504h58m51.642207893s) [upload=443121]: Done starting upload Dec 26 16:27:51.771241651 (504h58m51.642324686s) [upload=443121]: Uploading segments... Dec 26 16:27:51.771383371 (504h58m51.642466406s) [upload=443121]: Got next segment. Inline: false Dec 26 16:27:51.771422144 (504h58m51.642505189s) [upload=443121]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:27:51.771427394 (504h58m51.642510429s) [upload=443121]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:27:52.016309964 (504h58m51.887393009s) [seg_pos={0 0}, upload=443121]: Begin upload segment... Dec 26 16:27:52.016379056 (504h58m51.887462101s) [seg_pos={0 0}, upload=443121]: Done begin upload segment. Dec 26 16:27:52.016423610 (504h58m51.887506645s) [noise=true , address=45.91.116.8:4047 , piece_id=EN74CX7XC4RLGJ4VNKSHTNZ6YQ5ADR7UXMNNUSE3XOQD6GCOPXRA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.016433769 (504h58m51.887516804s) [noise=true , address=145.79.227.46:4602 , piece_id=EBVUOKSVSRKQDFBOCP7BZWU4HOSWXIJ5BNSGQ6QX5VTNBY7RB36Q, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.016635383 (504h58m51.887718429s) [noise=true , address=145.79.227.37:4566 , piece_id=BUJYPGEELG5UXS7NNWAC7TMLTOFGB4J5Z4B6U6VHMWG4SBY25WWA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.017118383 (504h58m51.888201418s) [noise=true , address=145.79.227.50:4374 , piece_id=76IQX5RDRU7J3ORVVBC5CIJZKWDFAD3C43F3FLJLOSIKLX5RPF6A, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.017289469 (504h58m51.888372504s) [noise=true , address=45.91.116.12:4226 , piece_id=K3KQIRVUGUWCSMIYNJKTU77UDPF7T2PBK5KKOZSAYQNMD7B6TSKA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.018048635 (504h58m51.88913167s) [noise=true , address=45.91.116.13:4156 , piece_id=XTWDUNXJGISATJMKRJMQFNWTSHMEGBT6A7PSZ6YBD7JLW3H63EFQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.018174414 (504h58m51.889257459s) [noise=true , address=45.91.116.5:4298 , piece_id=WODIDTWZ7FYCDPI6BT2YFUIPMETGNAS5HVUFQ3E6ZYHQFO6IWGUA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.018563374 (504h58m51.889646419s) [noise=true , address=145.79.227.51:4409 , piece_id=QDVTB5JBDKWJZZIV7OXY2KGN6YTSOZRTKZOLSWINHDZPLL2US4CQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.019222299 (504h58m51.890305344s) [noise=true , address=77.90.31.14:4662 , piece_id=GBYWCBM27G5RW5UMQF3ZLDLRT45WUKY3MUMGV3JCV4MYAVT6PW5A, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.019571023 (504h58m51.890654068s) [noise=true , address=145.79.227.48:4301 , piece_id=KRQUIU5B3VGUL6YJ22KBBJVPMK5LWFMCCSS2GMJPNGZCYAVKGJ2Q, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.019723183 (504h58m51.890806229s) [noise=true , address=45.91.116.11:4120 , piece_id=6C6XZBBDJBRZREMQ56T4HBSMIJHFOMCCRNAOSYQO4A4KS6QLC5GQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020133524 (504h58m51.891216559s) [noise=true , address=145.79.227.36:4518 , piece_id=STUHSYSV7MU6GUIAQP5N5JIIYCAUBLZBQ2PWBYLJR35MH73Z7QKQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020149455 (504h58m51.89123249s) [noise=true , address=145.79.227.16:4909 , piece_id=B5OKFWF6TCDCJCYHQO3FRP5OPLO4T4O4FIZAKNUKXTPBZV62AURQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020566770 (504h58m51.891649815s) [noise=true , address=145.79.227.47:4267 , piece_id=FHYMP7JM64IBGVK5KP57F45AE67JYLSL24WMJZP5ROMOL37ENXGQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020699993 (504h58m51.891783038s) [noise=true , address=145.79.227.41:4085 , piece_id=EOFAXKFJZH5WGXDREREJ23WOPW6ZKQWM6PJESRFANSXTNGVABPBA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.023082460 (504h58m51.894165505s) [noise=true , address=145.79.227.32:4836 , piece_id=SW25F4SS4GDPSE6QJT6W5QVIBPEQ7VMI54ZF3EGWWOOD7TQ4BNBA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:53.754638374 (504h58m53.625721419s) [upload=443121]: Next returned nil segment Dec 26 16:27:53.754655587 (504h58m53.625738632s) [upload=443121]: Waiting for error group managing segments... Dec 26 16:27:54.042259825 (504h58m53.91334286s) [noise=true , address=145.79.227.32:4836 , piece_id=SW25F4SS4GDPSE6QJT6W5QVIBPEQ7VMI54ZF3EGWWOOD7TQ4BNBA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.042937176 (504h58m53.914020221s) [noise=true , address=145.79.227.51:4409 , piece_id=QDVTB5JBDKWJZZIV7OXY2KGN6YTSOZRTKZOLSWINHDZPLL2US4CQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.044245196 (504h58m53.915328231s) [noise=true , address=145.79.227.36:4518 , piece_id=STUHSYSV7MU6GUIAQP5N5JIIYCAUBLZBQ2PWBYLJR35MH73Z7QKQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.053542530 (504h58m53.924625585s) [noise=true , address=145.79.227.37:4566 , piece_id=BUJYPGEELG5UXS7NNWAC7TMLTOFGB4J5Z4B6U6VHMWG4SBY25WWA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.054374083 (504h58m53.925457118s) [noise=true , address=145.79.227.48:4301 , piece_id=KRQUIU5B3VGUL6YJ22KBBJVPMK5LWFMCCSS2GMJPNGZCYAVKGJ2Q, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.062316276 (504h58m53.933399321s) [noise=true , address=145.79.227.50:4374 , piece_id=76IQX5RDRU7J3ORVVBC5CIJZKWDFAD3C43F3FLJLOSIKLX5RPF6A, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.065628374 (504h58m53.936711409s) [noise=true , address=145.79.227.46:4602 , piece_id=EBVUOKSVSRKQDFBOCP7BZWU4HOSWXIJ5BNSGQ6QX5VTNBY7RB36Q, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.078233186 (504h58m53.949316221s) [noise=true , address=145.79.227.41:4085 , piece_id=EOFAXKFJZH5WGXDREREJ23WOPW6ZKQWM6PJESRFANSXTNGVABPBA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.095259877 (504h58m53.966342922s) [noise=true , address=145.79.227.16:4909 , piece_id=B5OKFWF6TCDCJCYHQO3FRP5OPLO4T4O4FIZAKNUKXTPBZV62AURQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.101046285 (504h58m53.97212933s) [noise=true , address=145.79.227.47:4267 , piece_id=FHYMP7JM64IBGVK5KP57F45AE67JYLSL24WMJZP5ROMOL37ENXGQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.286875276 (504h58m54.157958322s) [noise=true , address=45.91.116.11:4120 , piece_id=6C6XZBBDJBRZREMQ56T4HBSMIJHFOMCCRNAOSYQO4A4KS6QLC5GQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.381576113 (504h58m54.252659158s) [noise=true , address=45.91.116.12:4226 , piece_id=K3KQIRVUGUWCSMIYNJKTU77UDPF7T2PBK5KKOZSAYQNMD7B6TSKA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.404723019 (504h58m54.275806064s) [noise=true , address=45.91.116.8:4047 , piece_id=EN74CX7XC4RLGJ4VNKSHTNZ6YQ5ADR7UXMNNUSE3XOQD6GCOPXRA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.404747315 (504h58m54.27583035s) [seg_pos={0 0}, upload=443121]: Segment reached optimal threshold of 13 pieces. Dec 26 16:27:54.405263158 (504h58m54.276346203s) [noise=true , address=45.91.116.5:4298 , piece_id=WODIDTWZ7FYCDPI6BT2YFUIPMETGNAS5HVUFQ3E6ZYHQFO6IWGUA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:60970->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 16:27:54.405273627 (504h58m54.276356682s) [noise=true , address=45.91.116.5:4298 , piece_id=WODIDTWZ7FYCDPI6BT2YFUIPMETGNAS5HVUFQ3E6ZYHQFO6IWGUA, seg_pos={0 0}, upload=443121]: Long tail cancellation due to optimal threshold Dec 26 16:27:54.405349602 (504h58m54.276432637s) [noise=true , address=45.91.116.13:4156 , piece_id=XTWDUNXJGISATJMKRJMQFNWTSHMEGBT6A7PSZ6YBD7JLW3H63EFQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 16:27:54.405365733 (504h58m54.276448768s) [noise=true , address=45.91.116.13:4156 , piece_id=XTWDUNXJGISATJMKRJMQFNWTSHMEGBT6A7PSZ6YBD7JLW3H63EFQ, seg_pos={0 0}, upload=443121]: Long tail cancellation due to optimal threshold Dec 26 16:27:54.405542509 (504h58m54.276625544s) [noise=true , address=77.90.31.14:4662 , piece_id=GBYWCBM27G5RW5UMQF3ZLDLRT45WUKY3MUMGV3JCV4MYAVT6PW5A, seg_pos={0 0}, upload=443121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:35100->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:35100->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35100->77.90.31.14:4662: use of closed network connection Dec 26 16:27:54.405549372 (504h58m54.276632417s) [noise=true , address=77.90.31.14:4662 , piece_id=GBYWCBM27G5RW5UMQF3ZLDLRT45WUKY3MUMGV3JCV4MYAVT6PW5A, seg_pos={0 0}, upload=443121]: Long tail cancellation due to optimal threshold Dec 26 16:27:54.405594718 (504h58m54.276677763s) [seg_pos={0 0}, upload=443121]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:27:54.405861196 (504h58m54.276944241s) [upload=443121]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:27:54.405868820 (504h58m54.276951855s) [upload=443121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:27:54.704991443 (504h58m54.576074478s) [upload=443121]: Done uploading segments... Dec 26 16:27:54.705011470 (504h58m54.576094505s) [upload=443121]: Upload finished. err: <nil> Duration: 6.175687968s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X...XX.........X.............XXX...X...XX.....XX..........X.] duration: 3.445083551s trace: https://jaeger.dev.storj.io/trace/4d8eedf07808cbf FAIL (2025-12-26 16:29:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.44510997s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X....X..X...X.......XX..........X.........X..X..........] duration: 27.133721071s trace: https://jaeger.dev.storj.io/trace/3e3dfe0984f19d4e FAIL (2025-12-26 16:29:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.466998847 (504h51m15.338081892s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467028614 (504h51m15.338111659s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467359594 (504h51m15.338442639s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467908379 (504h51m15.338991424s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.468520054 (504h51m15.339603089s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469055293 (504h51m15.340138328s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469497605 (504h51m15.34058064s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469973642 (504h51m15.341056687s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.470454397 (504h51m15.341537443s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471043630 (504h51m15.342126695s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471470552 (504h51m15.342553587s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471676735 (504h51m15.34275977s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472062900 (504h51m15.343145935s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472369224 (504h51m15.343452259s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472823849 (504h51m15.343906884s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:16.743410939 (504h51m16.614493984s) [upload=443005]: Next returned nil segment Dec 26 16:20:16.743428862 (504h51m16.614511907s) [upload=443005]: Waiting for error group managing segments... Dec 26 16:20:17.091086984 (504h51m16.962170029s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.389229539 (504h51m17.260312594s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.464846438 (504h51m17.335929483s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.542075705 (504h51m17.41315875s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.623422163 (504h51m17.494505208s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.665409184 (504h51m17.536492219s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.301995626 (504h51m18.173078671s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.311693532 (504h51m18.182776567s) [noise=true , address=145.79.227.34:4915 , piece_id=7ZBWNPUF54BEGIXTDRDI3J7HNEZ63MEXKUBMRZW6PFWWNCEAH3AA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.340201235 (504h51m18.21128428s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.379076200 (504h51m18.250159245s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.474362322 (504h51m18.345445367s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.555133063 (504h51m18.426216108s) [noise=true , address=77.90.31.17:4791 , piece_id=QFHBBE4B5DWZU7HTAARXW655MKX4YPKPOXVM77CH5DCPSLZ7UGNA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.707673447 (504h51m18.578756492s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734654664 (504h51m18.605737709s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734678970 (504h51m18.605762005s) [seg_pos={0 0}, upload=443005]: Segment reached optimal threshold of 14 pieces. Dec 26 16:20:18.734942853 (504h51m18.606025888s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.734952842 (504h51m18.606035887s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735555389 (504h51m18.606638434s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection Dec 26 16:20:18.735566701 (504h51m18.606649746s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735681349 (504h51m18.606764394s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735699113 (504h51m18.606782148s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735826605 (504h51m18.60690968s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735837737 (504h51m18.606920782s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735908392 (504h51m18.606991447s) [seg_pos={0 0}, upload=443005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:20:18.736024182 (504h51m18.607107227s) [upload=443005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:20:18.736031275 (504h51m18.60711431s) [upload=443005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:20:19.031573339 (504h51m18.902656384s) [upload=443005]: Done uploading segments... Dec 26 16:20:19.031602895 (504h51m18.90268594s) [upload=443005]: Upload finished. err: <nil> Duration: 27.133750216s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 12.176248292s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:29:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 12.176303537s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 9.827094533s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:29:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 9.827201708s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..................X.....X..XX..XXX.X...XX.X.XX.X....XX...X..] duration: 6.813591537s trace: https://jaeger.dev.storj.io/trace/7db2f5600219fec2 FAIL (2025-12-26 16:29:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8 (504h58m51.642207893s) [upload=443121]: Done starting upload Dec 26 16:27:51.771241651 (504h58m51.642324686s) [upload=443121]: Uploading segments... Dec 26 16:27:51.771383371 (504h58m51.642466406s) [upload=443121]: Got next segment. Inline: false Dec 26 16:27:51.771422144 (504h58m51.642505189s) [upload=443121]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:27:51.771427394 (504h58m51.642510429s) [upload=443121]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:27:52.016309964 (504h58m51.887393009s) [seg_pos={0 0}, upload=443121]: Begin upload segment... Dec 26 16:27:52.016379056 (504h58m51.887462101s) [seg_pos={0 0}, upload=443121]: Done begin upload segment. Dec 26 16:27:52.016423610 (504h58m51.887506645s) [noise=true , address=45.91.116.8:4047 , piece_id=EN74CX7XC4RLGJ4VNKSHTNZ6YQ5ADR7UXMNNUSE3XOQD6GCOPXRA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.016433769 (504h58m51.887516804s) [noise=true , address=145.79.227.46:4602 , piece_id=EBVUOKSVSRKQDFBOCP7BZWU4HOSWXIJ5BNSGQ6QX5VTNBY7RB36Q, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.016635383 (504h58m51.887718429s) [noise=true , address=145.79.227.37:4566 , piece_id=BUJYPGEELG5UXS7NNWAC7TMLTOFGB4J5Z4B6U6VHMWG4SBY25WWA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.017118383 (504h58m51.888201418s) [noise=true , address=145.79.227.50:4374 , piece_id=76IQX5RDRU7J3ORVVBC5CIJZKWDFAD3C43F3FLJLOSIKLX5RPF6A, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.017289469 (504h58m51.888372504s) [noise=true , address=45.91.116.12:4226 , piece_id=K3KQIRVUGUWCSMIYNJKTU77UDPF7T2PBK5KKOZSAYQNMD7B6TSKA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.018048635 (504h58m51.88913167s) [noise=true , address=45.91.116.13:4156 , piece_id=XTWDUNXJGISATJMKRJMQFNWTSHMEGBT6A7PSZ6YBD7JLW3H63EFQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.018174414 (504h58m51.889257459s) [noise=true , address=45.91.116.5:4298 , piece_id=WODIDTWZ7FYCDPI6BT2YFUIPMETGNAS5HVUFQ3E6ZYHQFO6IWGUA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.018563374 (504h58m51.889646419s) [noise=true , address=145.79.227.51:4409 , piece_id=QDVTB5JBDKWJZZIV7OXY2KGN6YTSOZRTKZOLSWINHDZPLL2US4CQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.019222299 (504h58m51.890305344s) [noise=true , address=77.90.31.14:4662 , piece_id=GBYWCBM27G5RW5UMQF3ZLDLRT45WUKY3MUMGV3JCV4MYAVT6PW5A, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.019571023 (504h58m51.890654068s) [noise=true , address=145.79.227.48:4301 , piece_id=KRQUIU5B3VGUL6YJ22KBBJVPMK5LWFMCCSS2GMJPNGZCYAVKGJ2Q, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.019723183 (504h58m51.890806229s) [noise=true , address=45.91.116.11:4120 , piece_id=6C6XZBBDJBRZREMQ56T4HBSMIJHFOMCCRNAOSYQO4A4KS6QLC5GQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020133524 (504h58m51.891216559s) [noise=true , address=145.79.227.36:4518 , piece_id=STUHSYSV7MU6GUIAQP5N5JIIYCAUBLZBQ2PWBYLJR35MH73Z7QKQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020149455 (504h58m51.89123249s) [noise=true , address=145.79.227.16:4909 , piece_id=B5OKFWF6TCDCJCYHQO3FRP5OPLO4T4O4FIZAKNUKXTPBZV62AURQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020566770 (504h58m51.891649815s) [noise=true , address=145.79.227.47:4267 , piece_id=FHYMP7JM64IBGVK5KP57F45AE67JYLSL24WMJZP5ROMOL37ENXGQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020699993 (504h58m51.891783038s) [noise=true , address=145.79.227.41:4085 , piece_id=EOFAXKFJZH5WGXDREREJ23WOPW6ZKQWM6PJESRFANSXTNGVABPBA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.023082460 (504h58m51.894165505s) [noise=true , address=145.79.227.32:4836 , piece_id=SW25F4SS4GDPSE6QJT6W5QVIBPEQ7VMI54ZF3EGWWOOD7TQ4BNBA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:53.754638374 (504h58m53.625721419s) [upload=443121]: Next returned nil segment Dec 26 16:27:53.754655587 (504h58m53.625738632s) [upload=443121]: Waiting for error group managing segments... Dec 26 16:27:54.042259825 (504h58m53.91334286s) [noise=true , address=145.79.227.32:4836 , piece_id=SW25F4SS4GDPSE6QJT6W5QVIBPEQ7VMI54ZF3EGWWOOD7TQ4BNBA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.042937176 (504h58m53.914020221s) [noise=true , address=145.79.227.51:4409 , piece_id=QDVTB5JBDKWJZZIV7OXY2KGN6YTSOZRTKZOLSWINHDZPLL2US4CQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.044245196 (504h58m53.915328231s) [noise=true , address=145.79.227.36:4518 , piece_id=STUHSYSV7MU6GUIAQP5N5JIIYCAUBLZBQ2PWBYLJR35MH73Z7QKQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.053542530 (504h58m53.924625585s) [noise=true , address=145.79.227.37:4566 , piece_id=BUJYPGEELG5UXS7NNWAC7TMLTOFGB4J5Z4B6U6VHMWG4SBY25WWA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.054374083 (504h58m53.925457118s) [noise=true , address=145.79.227.48:4301 , piece_id=KRQUIU5B3VGUL6YJ22KBBJVPMK5LWFMCCSS2GMJPNGZCYAVKGJ2Q, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.062316276 (504h58m53.933399321s) [noise=true , address=145.79.227.50:4374 , piece_id=76IQX5RDRU7J3ORVVBC5CIJZKWDFAD3C43F3FLJLOSIKLX5RPF6A, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.065628374 (504h58m53.936711409s) [noise=true , address=145.79.227.46:4602 , piece_id=EBVUOKSVSRKQDFBOCP7BZWU4HOSWXIJ5BNSGQ6QX5VTNBY7RB36Q, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.078233186 (504h58m53.949316221s) [noise=true , address=145.79.227.41:4085 , piece_id=EOFAXKFJZH5WGXDREREJ23WOPW6ZKQWM6PJESRFANSXTNGVABPBA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.095259877 (504h58m53.966342922s) [noise=true , address=145.79.227.16:4909 , piece_id=B5OKFWF6TCDCJCYHQO3FRP5OPLO4T4O4FIZAKNUKXTPBZV62AURQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.101046285 (504h58m53.97212933s) [noise=true , address=145.79.227.47:4267 , piece_id=FHYMP7JM64IBGVK5KP57F45AE67JYLSL24WMJZP5ROMOL37ENXGQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.286875276 (504h58m54.157958322s) [noise=true , address=45.91.116.11:4120 , piece_id=6C6XZBBDJBRZREMQ56T4HBSMIJHFOMCCRNAOSYQO4A4KS6QLC5GQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.381576113 (504h58m54.252659158s) [noise=true , address=45.91.116.12:4226 , piece_id=K3KQIRVUGUWCSMIYNJKTU77UDPF7T2PBK5KKOZSAYQNMD7B6TSKA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.404723019 (504h58m54.275806064s) [noise=true , address=45.91.116.8:4047 , piece_id=EN74CX7XC4RLGJ4VNKSHTNZ6YQ5ADR7UXMNNUSE3XOQD6GCOPXRA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.404747315 (504h58m54.27583035s) [seg_pos={0 0}, upload=443121]: Segment reached optimal threshold of 13 pieces. Dec 26 16:27:54.405263158 (504h58m54.276346203s) [noise=true , address=45.91.116.5:4298 , piece_id=WODIDTWZ7FYCDPI6BT2YFUIPMETGNAS5HVUFQ3E6ZYHQFO6IWGUA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:60970->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 16:27:54.405273627 (504h58m54.276356682s) [noise=true , address=45.91.116.5:4298 , piece_id=WODIDTWZ7FYCDPI6BT2YFUIPMETGNAS5HVUFQ3E6ZYHQFO6IWGUA, seg_pos={0 0}, upload=443121]: Long tail cancellation due to optimal threshold Dec 26 16:27:54.405349602 (504h58m54.276432637s) [noise=true , address=45.91.116.13:4156 , piece_id=XTWDUNXJGISATJMKRJMQFNWTSHMEGBT6A7PSZ6YBD7JLW3H63EFQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 16:27:54.405365733 (504h58m54.276448768s) [noise=true , address=45.91.116.13:4156 , piece_id=XTWDUNXJGISATJMKRJMQFNWTSHMEGBT6A7PSZ6YBD7JLW3H63EFQ, seg_pos={0 0}, upload=443121]: Long tail cancellation due to optimal threshold Dec 26 16:27:54.405542509 (504h58m54.276625544s) [noise=true , address=77.90.31.14:4662 , piece_id=GBYWCBM27G5RW5UMQF3ZLDLRT45WUKY3MUMGV3JCV4MYAVT6PW5A, seg_pos={0 0}, upload=443121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:35100->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:35100->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35100->77.90.31.14:4662: use of closed network connection Dec 26 16:27:54.405549372 (504h58m54.276632417s) [noise=true , address=77.90.31.14:4662 , piece_id=GBYWCBM27G5RW5UMQF3ZLDLRT45WUKY3MUMGV3JCV4MYAVT6PW5A, seg_pos={0 0}, upload=443121]: Long tail cancellation due to optimal threshold Dec 26 16:27:54.405594718 (504h58m54.276677763s) [seg_pos={0 0}, upload=443121]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:27:54.405861196 (504h58m54.276944241s) [upload=443121]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:27:54.405868820 (504h58m54.276951855s) [upload=443121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:27:54.704991443 (504h58m54.576074478s) [upload=443121]: Done uploading segments... Dec 26 16:27:54.705011470 (504h58m54.576094505s) [upload=443121]: Upload finished. err: <nil> Duration: 6.813618179s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [X....X....X..X...X.......XX..........X.........X..X.........] duration: 37.125611345s trace: https://jaeger.dev.storj.io/trace/3e3dfe0984f19d4e FAIL (2025-12-26 16:28:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.466998847 (504h51m15.338081892s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467028614 (504h51m15.338111659s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467359594 (504h51m15.338442639s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467908379 (504h51m15.338991424s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.468520054 (504h51m15.339603089s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469055293 (504h51m15.340138328s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469497605 (504h51m15.34058064s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469973642 (504h51m15.341056687s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.470454397 (504h51m15.341537443s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471043630 (504h51m15.342126695s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471470552 (504h51m15.342553587s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471676735 (504h51m15.34275977s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472062900 (504h51m15.343145935s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472369224 (504h51m15.343452259s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472823849 (504h51m15.343906884s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:16.743410939 (504h51m16.614493984s) [upload=443005]: Next returned nil segment Dec 26 16:20:16.743428862 (504h51m16.614511907s) [upload=443005]: Waiting for error group managing segments... Dec 26 16:20:17.091086984 (504h51m16.962170029s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.389229539 (504h51m17.260312594s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.464846438 (504h51m17.335929483s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.542075705 (504h51m17.41315875s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.623422163 (504h51m17.494505208s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.665409184 (504h51m17.536492219s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.301995626 (504h51m18.173078671s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.311693532 (504h51m18.182776567s) [noise=true , address=145.79.227.34:4915 , piece_id=7ZBWNPUF54BEGIXTDRDI3J7HNEZ63MEXKUBMRZW6PFWWNCEAH3AA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.340201235 (504h51m18.21128428s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.379076200 (504h51m18.250159245s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.474362322 (504h51m18.345445367s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.555133063 (504h51m18.426216108s) [noise=true , address=77.90.31.17:4791 , piece_id=QFHBBE4B5DWZU7HTAARXW655MKX4YPKPOXVM77CH5DCPSLZ7UGNA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.707673447 (504h51m18.578756492s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734654664 (504h51m18.605737709s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734678970 (504h51m18.605762005s) [seg_pos={0 0}, upload=443005]: Segment reached optimal threshold of 14 pieces. Dec 26 16:20:18.734942853 (504h51m18.606025888s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.734952842 (504h51m18.606035887s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735555389 (504h51m18.606638434s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection Dec 26 16:20:18.735566701 (504h51m18.606649746s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735681349 (504h51m18.606764394s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735699113 (504h51m18.606782148s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735826605 (504h51m18.60690968s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735837737 (504h51m18.606920782s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735908392 (504h51m18.606991447s) [seg_pos={0 0}, upload=443005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:20:18.736024182 (504h51m18.607107227s) [upload=443005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:20:18.736031275 (504h51m18.60711431s) [upload=443005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:20:19.031573339 (504h51m18.902656384s) [upload=443005]: Done uploading segments... Dec 26 16:20:19.031602895 (504h51m18.90268594s) [upload=443005]: Upload finished. err: <nil> Duration: 37.12564043s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [XX...XX.........X.............XXX...X...XX.....XX..........X] duration: 5.001933354s trace: https://jaeger.dev.storj.io/trace/4d8eedf07808cbf FAIL (2025-12-26 16:28:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001955977s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 16.624073778s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:28:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 16.624103634s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.XX...XX.........X.............XXX...X...XX.....XX..........] duration: 3.454150996s trace: https://jaeger.dev.storj.io/trace/7e43da40a4822a39 FAIL (2025-12-26 16:28:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.454178068s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...................X.....X..XX..XXX.X...XX.X.XX.X....XX...X.] duration: 6.096754806s trace: https://jaeger.dev.storj.io/trace/7db2f5600219fec2 FAIL (2025-12-26 16:28:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8 (504h58m51.642207893s) [upload=443121]: Done starting upload Dec 26 16:27:51.771241651 (504h58m51.642324686s) [upload=443121]: Uploading segments... Dec 26 16:27:51.771383371 (504h58m51.642466406s) [upload=443121]: Got next segment. Inline: false Dec 26 16:27:51.771422144 (504h58m51.642505189s) [upload=443121]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:27:51.771427394 (504h58m51.642510429s) [upload=443121]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:27:52.016309964 (504h58m51.887393009s) [seg_pos={0 0}, upload=443121]: Begin upload segment... Dec 26 16:27:52.016379056 (504h58m51.887462101s) [seg_pos={0 0}, upload=443121]: Done begin upload segment. Dec 26 16:27:52.016423610 (504h58m51.887506645s) [noise=true , address=45.91.116.8:4047 , piece_id=EN74CX7XC4RLGJ4VNKSHTNZ6YQ5ADR7UXMNNUSE3XOQD6GCOPXRA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.016433769 (504h58m51.887516804s) [noise=true , address=145.79.227.46:4602 , piece_id=EBVUOKSVSRKQDFBOCP7BZWU4HOSWXIJ5BNSGQ6QX5VTNBY7RB36Q, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.016635383 (504h58m51.887718429s) [noise=true , address=145.79.227.37:4566 , piece_id=BUJYPGEELG5UXS7NNWAC7TMLTOFGB4J5Z4B6U6VHMWG4SBY25WWA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.017118383 (504h58m51.888201418s) [noise=true , address=145.79.227.50:4374 , piece_id=76IQX5RDRU7J3ORVVBC5CIJZKWDFAD3C43F3FLJLOSIKLX5RPF6A, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.017289469 (504h58m51.888372504s) [noise=true , address=45.91.116.12:4226 , piece_id=K3KQIRVUGUWCSMIYNJKTU77UDPF7T2PBK5KKOZSAYQNMD7B6TSKA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.018048635 (504h58m51.88913167s) [noise=true , address=45.91.116.13:4156 , piece_id=XTWDUNXJGISATJMKRJMQFNWTSHMEGBT6A7PSZ6YBD7JLW3H63EFQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.018174414 (504h58m51.889257459s) [noise=true , address=45.91.116.5:4298 , piece_id=WODIDTWZ7FYCDPI6BT2YFUIPMETGNAS5HVUFQ3E6ZYHQFO6IWGUA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.018563374 (504h58m51.889646419s) [noise=true , address=145.79.227.51:4409 , piece_id=QDVTB5JBDKWJZZIV7OXY2KGN6YTSOZRTKZOLSWINHDZPLL2US4CQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.019222299 (504h58m51.890305344s) [noise=true , address=77.90.31.14:4662 , piece_id=GBYWCBM27G5RW5UMQF3ZLDLRT45WUKY3MUMGV3JCV4MYAVT6PW5A, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.019571023 (504h58m51.890654068s) [noise=true , address=145.79.227.48:4301 , piece_id=KRQUIU5B3VGUL6YJ22KBBJVPMK5LWFMCCSS2GMJPNGZCYAVKGJ2Q, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.019723183 (504h58m51.890806229s) [noise=true , address=45.91.116.11:4120 , piece_id=6C6XZBBDJBRZREMQ56T4HBSMIJHFOMCCRNAOSYQO4A4KS6QLC5GQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020133524 (504h58m51.891216559s) [noise=true , address=145.79.227.36:4518 , piece_id=STUHSYSV7MU6GUIAQP5N5JIIYCAUBLZBQ2PWBYLJR35MH73Z7QKQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020149455 (504h58m51.89123249s) [noise=true , address=145.79.227.16:4909 , piece_id=B5OKFWF6TCDCJCYHQO3FRP5OPLO4T4O4FIZAKNUKXTPBZV62AURQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020566770 (504h58m51.891649815s) [noise=true , address=145.79.227.47:4267 , piece_id=FHYMP7JM64IBGVK5KP57F45AE67JYLSL24WMJZP5ROMOL37ENXGQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020699993 (504h58m51.891783038s) [noise=true , address=145.79.227.41:4085 , piece_id=EOFAXKFJZH5WGXDREREJ23WOPW6ZKQWM6PJESRFANSXTNGVABPBA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.023082460 (504h58m51.894165505s) [noise=true , address=145.79.227.32:4836 , piece_id=SW25F4SS4GDPSE6QJT6W5QVIBPEQ7VMI54ZF3EGWWOOD7TQ4BNBA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:53.754638374 (504h58m53.625721419s) [upload=443121]: Next returned nil segment Dec 26 16:27:53.754655587 (504h58m53.625738632s) [upload=443121]: Waiting for error group managing segments... Dec 26 16:27:54.042259825 (504h58m53.91334286s) [noise=true , address=145.79.227.32:4836 , piece_id=SW25F4SS4GDPSE6QJT6W5QVIBPEQ7VMI54ZF3EGWWOOD7TQ4BNBA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.042937176 (504h58m53.914020221s) [noise=true , address=145.79.227.51:4409 , piece_id=QDVTB5JBDKWJZZIV7OXY2KGN6YTSOZRTKZOLSWINHDZPLL2US4CQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.044245196 (504h58m53.915328231s) [noise=true , address=145.79.227.36:4518 , piece_id=STUHSYSV7MU6GUIAQP5N5JIIYCAUBLZBQ2PWBYLJR35MH73Z7QKQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.053542530 (504h58m53.924625585s) [noise=true , address=145.79.227.37:4566 , piece_id=BUJYPGEELG5UXS7NNWAC7TMLTOFGB4J5Z4B6U6VHMWG4SBY25WWA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.054374083 (504h58m53.925457118s) [noise=true , address=145.79.227.48:4301 , piece_id=KRQUIU5B3VGUL6YJ22KBBJVPMK5LWFMCCSS2GMJPNGZCYAVKGJ2Q, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.062316276 (504h58m53.933399321s) [noise=true , address=145.79.227.50:4374 , piece_id=76IQX5RDRU7J3ORVVBC5CIJZKWDFAD3C43F3FLJLOSIKLX5RPF6A, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.065628374 (504h58m53.936711409s) [noise=true , address=145.79.227.46:4602 , piece_id=EBVUOKSVSRKQDFBOCP7BZWU4HOSWXIJ5BNSGQ6QX5VTNBY7RB36Q, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.078233186 (504h58m53.949316221s) [noise=true , address=145.79.227.41:4085 , piece_id=EOFAXKFJZH5WGXDREREJ23WOPW6ZKQWM6PJESRFANSXTNGVABPBA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.095259877 (504h58m53.966342922s) [noise=true , address=145.79.227.16:4909 , piece_id=B5OKFWF6TCDCJCYHQO3FRP5OPLO4T4O4FIZAKNUKXTPBZV62AURQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.101046285 (504h58m53.97212933s) [noise=true , address=145.79.227.47:4267 , piece_id=FHYMP7JM64IBGVK5KP57F45AE67JYLSL24WMJZP5ROMOL37ENXGQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.286875276 (504h58m54.157958322s) [noise=true , address=45.91.116.11:4120 , piece_id=6C6XZBBDJBRZREMQ56T4HBSMIJHFOMCCRNAOSYQO4A4KS6QLC5GQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.381576113 (504h58m54.252659158s) [noise=true , address=45.91.116.12:4226 , piece_id=K3KQIRVUGUWCSMIYNJKTU77UDPF7T2PBK5KKOZSAYQNMD7B6TSKA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.404723019 (504h58m54.275806064s) [noise=true , address=45.91.116.8:4047 , piece_id=EN74CX7XC4RLGJ4VNKSHTNZ6YQ5ADR7UXMNNUSE3XOQD6GCOPXRA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.404747315 (504h58m54.27583035s) [seg_pos={0 0}, upload=443121]: Segment reached optimal threshold of 13 pieces. Dec 26 16:27:54.405263158 (504h58m54.276346203s) [noise=true , address=45.91.116.5:4298 , piece_id=WODIDTWZ7FYCDPI6BT2YFUIPMETGNAS5HVUFQ3E6ZYHQFO6IWGUA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:60970->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 16:27:54.405273627 (504h58m54.276356682s) [noise=true , address=45.91.116.5:4298 , piece_id=WODIDTWZ7FYCDPI6BT2YFUIPMETGNAS5HVUFQ3E6ZYHQFO6IWGUA, seg_pos={0 0}, upload=443121]: Long tail cancellation due to optimal threshold Dec 26 16:27:54.405349602 (504h58m54.276432637s) [noise=true , address=45.91.116.13:4156 , piece_id=XTWDUNXJGISATJMKRJMQFNWTSHMEGBT6A7PSZ6YBD7JLW3H63EFQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 16:27:54.405365733 (504h58m54.276448768s) [noise=true , address=45.91.116.13:4156 , piece_id=XTWDUNXJGISATJMKRJMQFNWTSHMEGBT6A7PSZ6YBD7JLW3H63EFQ, seg_pos={0 0}, upload=443121]: Long tail cancellation due to optimal threshold Dec 26 16:27:54.405542509 (504h58m54.276625544s) [noise=true , address=77.90.31.14:4662 , piece_id=GBYWCBM27G5RW5UMQF3ZLDLRT45WUKY3MUMGV3JCV4MYAVT6PW5A, seg_pos={0 0}, upload=443121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:35100->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:35100->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35100->77.90.31.14:4662: use of closed network connection Dec 26 16:27:54.405549372 (504h58m54.276632417s) [noise=true , address=77.90.31.14:4662 , piece_id=GBYWCBM27G5RW5UMQF3ZLDLRT45WUKY3MUMGV3JCV4MYAVT6PW5A, seg_pos={0 0}, upload=443121]: Long tail cancellation due to optimal threshold Dec 26 16:27:54.405594718 (504h58m54.276677763s) [seg_pos={0 0}, upload=443121]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:27:54.405861196 (504h58m54.276944241s) [upload=443121]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:27:54.405868820 (504h58m54.276951855s) [upload=443121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:27:54.704991443 (504h58m54.576074478s) [upload=443121]: Done uploading segments... Dec 26 16:27:54.705011470 (504h58m54.576094505s) [upload=443121]: Upload finished. err: <nil> Duration: 6.096769022s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)