Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 11.668448664s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:32:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 11.668530389s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..XX.........X.............XXX...X...XX.....XX..........X...] duration: 5.000467172s trace: https://jaeger.dev.storj.io/trace/4d8eedf07808cbf FAIL (2025-12-26 16:31:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000494523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X....X..X...X.......XX..........X.........X..X............] duration: 43.464737406s trace: https://jaeger.dev.storj.io/trace/3e3dfe0984f19d4e FAIL (2025-12-26 16:31:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.466998847 (504h51m15.338081892s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467028614 (504h51m15.338111659s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467359594 (504h51m15.338442639s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467908379 (504h51m15.338991424s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.468520054 (504h51m15.339603089s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469055293 (504h51m15.340138328s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469497605 (504h51m15.34058064s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469973642 (504h51m15.341056687s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.470454397 (504h51m15.341537443s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471043630 (504h51m15.342126695s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471470552 (504h51m15.342553587s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471676735 (504h51m15.34275977s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472062900 (504h51m15.343145935s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472369224 (504h51m15.343452259s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472823849 (504h51m15.343906884s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:16.743410939 (504h51m16.614493984s) [upload=443005]: Next returned nil segment Dec 26 16:20:16.743428862 (504h51m16.614511907s) [upload=443005]: Waiting for error group managing segments... Dec 26 16:20:17.091086984 (504h51m16.962170029s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.389229539 (504h51m17.260312594s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.464846438 (504h51m17.335929483s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.542075705 (504h51m17.41315875s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.623422163 (504h51m17.494505208s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.665409184 (504h51m17.536492219s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.301995626 (504h51m18.173078671s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.311693532 (504h51m18.182776567s) [noise=true , address=145.79.227.34:4915 , piece_id=7ZBWNPUF54BEGIXTDRDI3J7HNEZ63MEXKUBMRZW6PFWWNCEAH3AA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.340201235 (504h51m18.21128428s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.379076200 (504h51m18.250159245s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.474362322 (504h51m18.345445367s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.555133063 (504h51m18.426216108s) [noise=true , address=77.90.31.17:4791 , piece_id=QFHBBE4B5DWZU7HTAARXW655MKX4YPKPOXVM77CH5DCPSLZ7UGNA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.707673447 (504h51m18.578756492s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734654664 (504h51m18.605737709s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734678970 (504h51m18.605762005s) [seg_pos={0 0}, upload=443005]: Segment reached optimal threshold of 14 pieces. Dec 26 16:20:18.734942853 (504h51m18.606025888s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.734952842 (504h51m18.606035887s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735555389 (504h51m18.606638434s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection Dec 26 16:20:18.735566701 (504h51m18.606649746s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735681349 (504h51m18.606764394s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735699113 (504h51m18.606782148s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735826605 (504h51m18.60690968s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735837737 (504h51m18.606920782s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735908392 (504h51m18.606991447s) [seg_pos={0 0}, upload=443005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:20:18.736024182 (504h51m18.607107227s) [upload=443005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:20:18.736031275 (504h51m18.60711431s) [upload=443005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:20:19.031573339 (504h51m18.902656384s) [upload=443005]: Done uploading segments... Dec 26 16:20:19.031602895 (504h51m18.90268594s) [upload=443005]: Upload finished. err: <nil> Duration: 43.464763455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [................X.....X..XX..XXX.X...XX.X.XX.X....XX...X...X] duration: 30.003266134s trace: https://jaeger.dev.storj.io/trace/3a1b9ad419ba9c2e FAIL (2025-12-26 16:31:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Done starting upload Dec 26 16:31:02.520243985 (505h2m2.39132703s) [upload=443167]: Uploading segments... Dec 26 16:31:02.520373302 (505h2m2.391456347s) [upload=443167]: Got next segment. Inline: false Dec 26 16:31:02.520451420 (505h2m2.391534455s) [upload=443167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:31:02.520457933 (505h2m2.391540978s) [upload=443167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:31:02.766182605 (505h2m2.63726566s) [seg_pos={0 0}, upload=443167]: Begin upload segment... Dec 26 16:31:02.766244784 (505h2m2.637327829s) [seg_pos={0 0}, upload=443167]: Done begin upload segment. Dec 26 16:31:02.766282245 (505h2m2.63736529s) [noise=true , address=145.79.227.47:4265 , piece_id=UJGWTN2YTG42UXP5MSXBH3GG3YFN7PCMH5L6OCSINBJAUYE2CLRA, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.766388217 (505h2m2.637471272s) [noise=true , address=145.79.227.13:4802 , piece_id=AF75MEXCINEL25WLU63QRY7AJ5WJ2Y63PY7OM6AJY4YTF6ANAMQA, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.766962440 (505h2m2.638045486s) [noise=true , address=145.79.227.46:4603 , piece_id=IXTJYVDACXWSDDV2CEYRRX7ZXZPZXFPNCGKFCMZ2B5MNMJXYVMGA, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.767331603 (505h2m2.638414648s) [noise=true , address=145.79.227.50:4375 , piece_id=IRG5LENZXD3JOXJUD43IAMO7GS4AY526F3ZFQXRZZKX57SWVOL5Q, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.767643537 (505h2m2.638726572s) [noise=true , address=145.79.227.2:4113 , piece_id=ZICIDCM7YK4JU3W3KF3E5IPBGPOE533P5PAK56NHBWSVYOKXAWDA, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.767946694 (505h2m2.639029729s) [noise=true , address=45.91.116.6:4973 , piece_id=YRRWYEAE2JP5HFFIOO3SBDVG7GBJHW42ELT7ITXEL3GYUUUZLJGA, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.768266263 (505h2m2.639349308s) [noise=true , address=145.79.227.11:4729 , piece_id=NPOSBCBL3ZM57WPZ5QXHISBZALEDTX62D2XGKYES3VJ7LDOC4QOA, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.768623412 (505h2m2.639706458s) [noise=true , address=45.91.116.5:4300 , piece_id=JRSZZVNIDTMLTBPJ4R2RFKO3TOCDM2Q32W2D7LROVV7VVZFLMDUQ, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.769083660 (505h2m2.640166705s) [noise=true , address=45.91.116.8:4046 , piece_id=CRM3FWNW3FKTB2ENA3MJX5BIUNOAE5U3KQ3MN3JHNMIVL7GXZVFA, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.771359813 (505h2m2.642442859s) [noise=true , address=145.79.227.48:4303 , piece_id=AFMR3KNKGEQI3FN2WUO5NVAIBBO6LO2F26FCBIB2NPH2S6GBZ65A, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.772834671 (505h2m2.643917707s) [noise=true , address=145.79.227.15:4874 , piece_id=K2VZXRDFGWNRDCQ65HRGFIZJ44ERHVGGTBDE5RIXVBNMF3JOT2HQ, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.774957223 (505h2m2.646040268s) [noise=true , address=145.79.227.37:4566 , piece_id=ESWX57X4F35FB67TE7LQY5IBZN4RQS5I57WRWXDC65QDEFJDMF2A, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.776754616 (505h2m2.647837661s) [noise=true , address=45.91.116.13:4154 , piece_id=RREHTH3WXYI2T3U3NVTTAQTUFF7PVCOBVFRGQ5PTI4IZL62OUQOA, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.778703467 (505h2m2.649786512s) [noise=true , address=145.79.227.52:4445 , piece_id=THBDPA46KZ2TUS3SMXEALVOZ7WHJ22AOCJ6V7BJ7RPYEGYQSRJHA, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.780156965 (505h2m2.65124001s) [noise=true , address=145.79.227.32:4836 , piece_id=UWEKC3WL5EO2ZBG2PIZSRXBTJII7OCO5BWGWUV24BSQHVXI3NDXA, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:02.781481337 (505h2m2.652564392s) [noise=true , address=145.79.227.16:4911 , piece_id=OJAMDI4LBR6XJ564IXBMEY644IBVZANNVWPMITHHW4LTJQATDF6Q, seg_pos={0 0}, upload=443167]: Uploading piece... Dec 26 16:31:04.482347452 (505h2m4.353430487s) [upload=443167]: Next returned nil segment Dec 26 16:31:04.482371098 (505h2m4.353454143s) [upload=443167]: Waiting for error group managing segments... Dec 26 16:31:04.782650442 (505h2m4.653733487s) [noise=true , address=145.79.227.11:4729 , piece_id=NPOSBCBL3ZM57WPZ5QXHISBZALEDTX62D2XGKYES3VJ7LDOC4QOA, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:04.786628867 (505h2m4.657711903s) [noise=true , address=145.79.227.13:4802 , piece_id=AF75MEXCINEL25WLU63QRY7AJ5WJ2Y63PY7OM6AJY4YTF6ANAMQA, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:04.789265528 (505h2m4.660348573s) [noise=true , address=145.79.227.32:4836 , piece_id=UWEKC3WL5EO2ZBG2PIZSRXBTJII7OCO5BWGWUV24BSQHVXI3NDXA, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:04.791139075 (505h2m4.66222212s) [noise=true , address=145.79.227.50:4375 , piece_id=IRG5LENZXD3JOXJUD43IAMO7GS4AY526F3ZFQXRZZKX57SWVOL5Q, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:04.794570309 (505h2m4.665653354s) [noise=true , address=145.79.227.48:4303 , piece_id=AFMR3KNKGEQI3FN2WUO5NVAIBBO6LO2F26FCBIB2NPH2S6GBZ65A, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:04.801382942 (505h2m4.672465977s) [noise=true , address=145.79.227.46:4603 , piece_id=IXTJYVDACXWSDDV2CEYRRX7ZXZPZXFPNCGKFCMZ2B5MNMJXYVMGA, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:04.801510975 (505h2m4.67259401s) [noise=true , address=145.79.227.47:4265 , piece_id=UJGWTN2YTG42UXP5MSXBH3GG3YFN7PCMH5L6OCSINBJAUYE2CLRA, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:04.803507187 (505h2m4.674590232s) [noise=true , address=145.79.227.52:4445 , piece_id=THBDPA46KZ2TUS3SMXEALVOZ7WHJ22AOCJ6V7BJ7RPYEGYQSRJHA, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:04.805518878 (505h2m4.676601913s) [noise=true , address=145.79.227.37:4566 , piece_id=ESWX57X4F35FB67TE7LQY5IBZN4RQS5I57WRWXDC65QDEFJDMF2A, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:04.848540248 (505h2m4.719623313s) [noise=true , address=145.79.227.16:4911 , piece_id=OJAMDI4LBR6XJ564IXBMEY644IBVZANNVWPMITHHW4LTJQATDF6Q, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:04.858805444 (505h2m4.729888499s) [noise=true , address=145.79.227.15:4874 , piece_id=K2VZXRDFGWNRDCQ65HRGFIZJ44ERHVGGTBDE5RIXVBNMF3JOT2HQ, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:05.003427269 (505h2m4.874510304s) [noise=true , address=45.91.116.13:4154 , piece_id=RREHTH3WXYI2T3U3NVTTAQTUFF7PVCOBVFRGQ5PTI4IZL62OUQOA, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:05.036112778 (505h2m4.907195823s) [noise=true , address=45.91.116.6:4973 , piece_id=YRRWYEAE2JP5HFFIOO3SBDVG7GBJHW42ELT7ITXEL3GYUUUZLJGA, seg_pos={0 0}, upload=443167]: Done uploading piece. err: <nil> Dec 26 16:31:05.036133989 (505h2m4.907217034s) [seg_pos={0 0}, upload=443167]: Segment reached optimal threshold of 13 pieces. Dec 26 16:31:05.036337076 (505h2m4.907420111s) [noise=true , address=145.79.227.2:4113 , piece_id=ZICIDCM7YK4JU3W3KF3E5IPBGPOE533P5PAK56NHBWSVYOKXAWDA, seg_pos={0 0}, upload=443167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:36182->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:36182->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36182->145.79.227.2:4113: use of closed network connection Dec 26 16:31:05.036346293 (505h2m4.907429338s) [noise=true , address=145.79.227.2:4113 , piece_id=ZICIDCM7YK4JU3W3KF3E5IPBGPOE533P5PAK56NHBWSVYOKXAWDA, seg_pos={0 0}, upload=443167]: Long tail cancellation due to optimal threshold Dec 26 16:31:05.036435754 (505h2m4.907518789s) [noise=true , address=45.91.116.8:4046 , piece_id=CRM3FWNW3FKTB2ENA3MJX5BIUNOAE5U3KQ3MN3JHNMIVL7GXZVFA, seg_pos={0 0}, upload=443167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:35748->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:35748->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35748->45.91.116.8:4046: use of closed network connection Dec 26 16:31:05.036451423 (505h2m4.907534468s) [noise=true , address=45.91.116.8:4046 , piece_id=CRM3FWNW3FKTB2ENA3MJX5BIUNOAE5U3KQ3MN3JHNMIVL7GXZVFA, seg_pos={0 0}, upload=443167]: Long tail cancellation due to optimal threshold Dec 26 16:31:05.036670089 (505h2m4.907753134s) [noise=true , address=45.91.116.5:4300 , piece_id=JRSZZVNIDTMLTBPJ4R2RFKO3TOCDM2Q32W2D7LROVV7VVZFLMDUQ, seg_pos={0 0}, upload=443167]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 26 16:31:05.036677633 (505h2m4.907760668s) [noise=true , address=45.91.116.5:4300 , piece_id=JRSZZVNIDTMLTBPJ4R2RFKO3TOCDM2Q32W2D7LROVV7VVZFLMDUQ, seg_pos={0 0}, upload=443167]: Long tail cancellation due to optimal threshold Dec 26 16:31:05.036697852 (505h2m4.907780887s) [seg_pos={0 0}, upload=443167]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:31:05.036765231 (505h2m4.907848266s) [upload=443167]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:31:05.036769849 (505h2m4.907852894s) [upload=443167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:31:05.339840279 (505h2m5.210923314s) [upload=443167]: Done uploading segments... Dec 26 16:31:05.339868072 (505h2m5.210951107s) [upload=443167]: Upload finished. err: <nil> Duration: 30.003296583s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 8.858772739s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:31:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 8.858807375s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...XX.........X.............XXX...X...XX.....XX..........X..] duration: 3.40084378s trace: https://jaeger.dev.storj.io/trace/4d8eedf07808cbf FAIL (2025-12-26 16:31:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.400877083s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X....X..X...X.......XX..........X.........X..X...........] duration: 53.516150561s trace: https://jaeger.dev.storj.io/trace/3e3dfe0984f19d4e FAIL (2025-12-26 16:31:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.466998847 (504h51m15.338081892s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467028614 (504h51m15.338111659s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467359594 (504h51m15.338442639s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467908379 (504h51m15.338991424s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.468520054 (504h51m15.339603089s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469055293 (504h51m15.340138328s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469497605 (504h51m15.34058064s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469973642 (504h51m15.341056687s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.470454397 (504h51m15.341537443s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471043630 (504h51m15.342126695s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471470552 (504h51m15.342553587s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471676735 (504h51m15.34275977s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472062900 (504h51m15.343145935s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472369224 (504h51m15.343452259s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472823849 (504h51m15.343906884s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:16.743410939 (504h51m16.614493984s) [upload=443005]: Next returned nil segment Dec 26 16:20:16.743428862 (504h51m16.614511907s) [upload=443005]: Waiting for error group managing segments... Dec 26 16:20:17.091086984 (504h51m16.962170029s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.389229539 (504h51m17.260312594s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.464846438 (504h51m17.335929483s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.542075705 (504h51m17.41315875s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.623422163 (504h51m17.494505208s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.665409184 (504h51m17.536492219s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.301995626 (504h51m18.173078671s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.311693532 (504h51m18.182776567s) [noise=true , address=145.79.227.34:4915 , piece_id=7ZBWNPUF54BEGIXTDRDI3J7HNEZ63MEXKUBMRZW6PFWWNCEAH3AA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.340201235 (504h51m18.21128428s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.379076200 (504h51m18.250159245s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.474362322 (504h51m18.345445367s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.555133063 (504h51m18.426216108s) [noise=true , address=77.90.31.17:4791 , piece_id=QFHBBE4B5DWZU7HTAARXW655MKX4YPKPOXVM77CH5DCPSLZ7UGNA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.707673447 (504h51m18.578756492s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734654664 (504h51m18.605737709s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734678970 (504h51m18.605762005s) [seg_pos={0 0}, upload=443005]: Segment reached optimal threshold of 14 pieces. Dec 26 16:20:18.734942853 (504h51m18.606025888s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.734952842 (504h51m18.606035887s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735555389 (504h51m18.606638434s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection Dec 26 16:20:18.735566701 (504h51m18.606649746s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735681349 (504h51m18.606764394s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735699113 (504h51m18.606782148s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735826605 (504h51m18.60690968s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735837737 (504h51m18.606920782s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735908392 (504h51m18.606991447s) [seg_pos={0 0}, upload=443005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:20:18.736024182 (504h51m18.607107227s) [upload=443005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:20:18.736031275 (504h51m18.60711431s) [upload=443005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:20:19.031573339 (504h51m18.902656384s) [upload=443005]: Done uploading segments... Dec 26 16:20:19.031602895 (504h51m18.90268594s) [upload=443005]: Upload finished. err: <nil> Duration: 53.51626503s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 7.5092174s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:31:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 7.509244913s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 20.691476905s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:30:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 20.691503786s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.................X.....X..XX..XXX.X...XX.X.XX.X....XX...X...] duration: 6.175656838s trace: https://jaeger.dev.storj.io/trace/7db2f5600219fec2 FAIL (2025-12-26 16:30:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8 (504h58m51.642207893s) [upload=443121]: Done starting upload Dec 26 16:27:51.771241651 (504h58m51.642324686s) [upload=443121]: Uploading segments... Dec 26 16:27:51.771383371 (504h58m51.642466406s) [upload=443121]: Got next segment. Inline: false Dec 26 16:27:51.771422144 (504h58m51.642505189s) [upload=443121]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:27:51.771427394 (504h58m51.642510429s) [upload=443121]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:27:52.016309964 (504h58m51.887393009s) [seg_pos={0 0}, upload=443121]: Begin upload segment... Dec 26 16:27:52.016379056 (504h58m51.887462101s) [seg_pos={0 0}, upload=443121]: Done begin upload segment. Dec 26 16:27:52.016423610 (504h58m51.887506645s) [noise=true , address=45.91.116.8:4047 , piece_id=EN74CX7XC4RLGJ4VNKSHTNZ6YQ5ADR7UXMNNUSE3XOQD6GCOPXRA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.016433769 (504h58m51.887516804s) [noise=true , address=145.79.227.46:4602 , piece_id=EBVUOKSVSRKQDFBOCP7BZWU4HOSWXIJ5BNSGQ6QX5VTNBY7RB36Q, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.016635383 (504h58m51.887718429s) [noise=true , address=145.79.227.37:4566 , piece_id=BUJYPGEELG5UXS7NNWAC7TMLTOFGB4J5Z4B6U6VHMWG4SBY25WWA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.017118383 (504h58m51.888201418s) [noise=true , address=145.79.227.50:4374 , piece_id=76IQX5RDRU7J3ORVVBC5CIJZKWDFAD3C43F3FLJLOSIKLX5RPF6A, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.017289469 (504h58m51.888372504s) [noise=true , address=45.91.116.12:4226 , piece_id=K3KQIRVUGUWCSMIYNJKTU77UDPF7T2PBK5KKOZSAYQNMD7B6TSKA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.018048635 (504h58m51.88913167s) [noise=true , address=45.91.116.13:4156 , piece_id=XTWDUNXJGISATJMKRJMQFNWTSHMEGBT6A7PSZ6YBD7JLW3H63EFQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.018174414 (504h58m51.889257459s) [noise=true , address=45.91.116.5:4298 , piece_id=WODIDTWZ7FYCDPI6BT2YFUIPMETGNAS5HVUFQ3E6ZYHQFO6IWGUA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.018563374 (504h58m51.889646419s) [noise=true , address=145.79.227.51:4409 , piece_id=QDVTB5JBDKWJZZIV7OXY2KGN6YTSOZRTKZOLSWINHDZPLL2US4CQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.019222299 (504h58m51.890305344s) [noise=true , address=77.90.31.14:4662 , piece_id=GBYWCBM27G5RW5UMQF3ZLDLRT45WUKY3MUMGV3JCV4MYAVT6PW5A, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.019571023 (504h58m51.890654068s) [noise=true , address=145.79.227.48:4301 , piece_id=KRQUIU5B3VGUL6YJ22KBBJVPMK5LWFMCCSS2GMJPNGZCYAVKGJ2Q, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.019723183 (504h58m51.890806229s) [noise=true , address=45.91.116.11:4120 , piece_id=6C6XZBBDJBRZREMQ56T4HBSMIJHFOMCCRNAOSYQO4A4KS6QLC5GQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020133524 (504h58m51.891216559s) [noise=true , address=145.79.227.36:4518 , piece_id=STUHSYSV7MU6GUIAQP5N5JIIYCAUBLZBQ2PWBYLJR35MH73Z7QKQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020149455 (504h58m51.89123249s) [noise=true , address=145.79.227.16:4909 , piece_id=B5OKFWF6TCDCJCYHQO3FRP5OPLO4T4O4FIZAKNUKXTPBZV62AURQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020566770 (504h58m51.891649815s) [noise=true , address=145.79.227.47:4267 , piece_id=FHYMP7JM64IBGVK5KP57F45AE67JYLSL24WMJZP5ROMOL37ENXGQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020699993 (504h58m51.891783038s) [noise=true , address=145.79.227.41:4085 , piece_id=EOFAXKFJZH5WGXDREREJ23WOPW6ZKQWM6PJESRFANSXTNGVABPBA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.023082460 (504h58m51.894165505s) [noise=true , address=145.79.227.32:4836 , piece_id=SW25F4SS4GDPSE6QJT6W5QVIBPEQ7VMI54ZF3EGWWOOD7TQ4BNBA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:53.754638374 (504h58m53.625721419s) [upload=443121]: Next returned nil segment Dec 26 16:27:53.754655587 (504h58m53.625738632s) [upload=443121]: Waiting for error group managing segments... Dec 26 16:27:54.042259825 (504h58m53.91334286s) [noise=true , address=145.79.227.32:4836 , piece_id=SW25F4SS4GDPSE6QJT6W5QVIBPEQ7VMI54ZF3EGWWOOD7TQ4BNBA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.042937176 (504h58m53.914020221s) [noise=true , address=145.79.227.51:4409 , piece_id=QDVTB5JBDKWJZZIV7OXY2KGN6YTSOZRTKZOLSWINHDZPLL2US4CQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.044245196 (504h58m53.915328231s) [noise=true , address=145.79.227.36:4518 , piece_id=STUHSYSV7MU6GUIAQP5N5JIIYCAUBLZBQ2PWBYLJR35MH73Z7QKQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.053542530 (504h58m53.924625585s) [noise=true , address=145.79.227.37:4566 , piece_id=BUJYPGEELG5UXS7NNWAC7TMLTOFGB4J5Z4B6U6VHMWG4SBY25WWA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.054374083 (504h58m53.925457118s) [noise=true , address=145.79.227.48:4301 , piece_id=KRQUIU5B3VGUL6YJ22KBBJVPMK5LWFMCCSS2GMJPNGZCYAVKGJ2Q, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.062316276 (504h58m53.933399321s) [noise=true , address=145.79.227.50:4374 , piece_id=76IQX5RDRU7J3ORVVBC5CIJZKWDFAD3C43F3FLJLOSIKLX5RPF6A, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.065628374 (504h58m53.936711409s) [noise=true , address=145.79.227.46:4602 , piece_id=EBVUOKSVSRKQDFBOCP7BZWU4HOSWXIJ5BNSGQ6QX5VTNBY7RB36Q, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.078233186 (504h58m53.949316221s) [noise=true , address=145.79.227.41:4085 , piece_id=EOFAXKFJZH5WGXDREREJ23WOPW6ZKQWM6PJESRFANSXTNGVABPBA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.095259877 (504h58m53.966342922s) [noise=true , address=145.79.227.16:4909 , piece_id=B5OKFWF6TCDCJCYHQO3FRP5OPLO4T4O4FIZAKNUKXTPBZV62AURQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.101046285 (504h58m53.97212933s) [noise=true , address=145.79.227.47:4267 , piece_id=FHYMP7JM64IBGVK5KP57F45AE67JYLSL24WMJZP5ROMOL37ENXGQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.286875276 (504h58m54.157958322s) [noise=true , address=45.91.116.11:4120 , piece_id=6C6XZBBDJBRZREMQ56T4HBSMIJHFOMCCRNAOSYQO4A4KS6QLC5GQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.381576113 (504h58m54.252659158s) [noise=true , address=45.91.116.12:4226 , piece_id=K3KQIRVUGUWCSMIYNJKTU77UDPF7T2PBK5KKOZSAYQNMD7B6TSKA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.404723019 (504h58m54.275806064s) [noise=true , address=45.91.116.8:4047 , piece_id=EN74CX7XC4RLGJ4VNKSHTNZ6YQ5ADR7UXMNNUSE3XOQD6GCOPXRA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.404747315 (504h58m54.27583035s) [seg_pos={0 0}, upload=443121]: Segment reached optimal threshold of 13 pieces. Dec 26 16:27:54.405263158 (504h58m54.276346203s) [noise=true , address=45.91.116.5:4298 , piece_id=WODIDTWZ7FYCDPI6BT2YFUIPMETGNAS5HVUFQ3E6ZYHQFO6IWGUA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:60970->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 16:27:54.405273627 (504h58m54.276356682s) [noise=true , address=45.91.116.5:4298 , piece_id=WODIDTWZ7FYCDPI6BT2YFUIPMETGNAS5HVUFQ3E6ZYHQFO6IWGUA, seg_pos={0 0}, upload=443121]: Long tail cancellation due to optimal threshold Dec 26 16:27:54.405349602 (504h58m54.276432637s) [noise=true , address=45.91.116.13:4156 , piece_id=XTWDUNXJGISATJMKRJMQFNWTSHMEGBT6A7PSZ6YBD7JLW3H63EFQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 16:27:54.405365733 (504h58m54.276448768s) [noise=true , address=45.91.116.13:4156 , piece_id=XTWDUNXJGISATJMKRJMQFNWTSHMEGBT6A7PSZ6YBD7JLW3H63EFQ, seg_pos={0 0}, upload=443121]: Long tail cancellation due to optimal threshold Dec 26 16:27:54.405542509 (504h58m54.276625544s) [noise=true , address=77.90.31.14:4662 , piece_id=GBYWCBM27G5RW5UMQF3ZLDLRT45WUKY3MUMGV3JCV4MYAVT6PW5A, seg_pos={0 0}, upload=443121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:35100->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:35100->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35100->77.90.31.14:4662: use of closed network connection Dec 26 16:27:54.405549372 (504h58m54.276632417s) [noise=true , address=77.90.31.14:4662 , piece_id=GBYWCBM27G5RW5UMQF3ZLDLRT45WUKY3MUMGV3JCV4MYAVT6PW5A, seg_pos={0 0}, upload=443121]: Long tail cancellation due to optimal threshold Dec 26 16:27:54.405594718 (504h58m54.276677763s) [seg_pos={0 0}, upload=443121]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:27:54.405861196 (504h58m54.276944241s) [upload=443121]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:27:54.405868820 (504h58m54.276951855s) [upload=443121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:27:54.704991443 (504h58m54.576074478s) [upload=443121]: Done uploading segments... Dec 26 16:27:54.705011470 (504h58m54.576094505s) [upload=443121]: Upload finished. err: <nil> Duration: 6.175687968s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X...XX.........X.............XXX...X...XX.....XX..........X.] duration: 3.445083551s trace: https://jaeger.dev.storj.io/trace/4d8eedf07808cbf FAIL (2025-12-26 16:29:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.44510997s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X....X..X...X.......XX..........X.........X..X..........] duration: 27.133721071s trace: https://jaeger.dev.storj.io/trace/3e3dfe0984f19d4e FAIL (2025-12-26 16:29:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.466998847 (504h51m15.338081892s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467028614 (504h51m15.338111659s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467359594 (504h51m15.338442639s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467908379 (504h51m15.338991424s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.468520054 (504h51m15.339603089s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469055293 (504h51m15.340138328s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469497605 (504h51m15.34058064s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469973642 (504h51m15.341056687s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.470454397 (504h51m15.341537443s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471043630 (504h51m15.342126695s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471470552 (504h51m15.342553587s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471676735 (504h51m15.34275977s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472062900 (504h51m15.343145935s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472369224 (504h51m15.343452259s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472823849 (504h51m15.343906884s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:16.743410939 (504h51m16.614493984s) [upload=443005]: Next returned nil segment Dec 26 16:20:16.743428862 (504h51m16.614511907s) [upload=443005]: Waiting for error group managing segments... Dec 26 16:20:17.091086984 (504h51m16.962170029s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.389229539 (504h51m17.260312594s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.464846438 (504h51m17.335929483s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.542075705 (504h51m17.41315875s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.623422163 (504h51m17.494505208s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.665409184 (504h51m17.536492219s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.301995626 (504h51m18.173078671s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.311693532 (504h51m18.182776567s) [noise=true , address=145.79.227.34:4915 , piece_id=7ZBWNPUF54BEGIXTDRDI3J7HNEZ63MEXKUBMRZW6PFWWNCEAH3AA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.340201235 (504h51m18.21128428s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.379076200 (504h51m18.250159245s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.474362322 (504h51m18.345445367s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.555133063 (504h51m18.426216108s) [noise=true , address=77.90.31.17:4791 , piece_id=QFHBBE4B5DWZU7HTAARXW655MKX4YPKPOXVM77CH5DCPSLZ7UGNA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.707673447 (504h51m18.578756492s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734654664 (504h51m18.605737709s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734678970 (504h51m18.605762005s) [seg_pos={0 0}, upload=443005]: Segment reached optimal threshold of 14 pieces. Dec 26 16:20:18.734942853 (504h51m18.606025888s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.734952842 (504h51m18.606035887s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735555389 (504h51m18.606638434s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection Dec 26 16:20:18.735566701 (504h51m18.606649746s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735681349 (504h51m18.606764394s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735699113 (504h51m18.606782148s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735826605 (504h51m18.60690968s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735837737 (504h51m18.606920782s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735908392 (504h51m18.606991447s) [seg_pos={0 0}, upload=443005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:20:18.736024182 (504h51m18.607107227s) [upload=443005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:20:18.736031275 (504h51m18.60711431s) [upload=443005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:20:19.031573339 (504h51m18.902656384s) [upload=443005]: Done uploading segments... Dec 26 16:20:19.031602895 (504h51m18.90268594s) [upload=443005]: Upload finished. err: <nil> Duration: 27.133750216s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 12.176248292s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:29:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 12.176303537s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 9.827094533s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:29:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 9.827201708s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..................X.....X..XX..XXX.X...XX.X.XX.X....XX...X..] duration: 6.813591537s trace: https://jaeger.dev.storj.io/trace/7db2f5600219fec2 FAIL (2025-12-26 16:29:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8 (504h58m51.642207893s) [upload=443121]: Done starting upload Dec 26 16:27:51.771241651 (504h58m51.642324686s) [upload=443121]: Uploading segments... Dec 26 16:27:51.771383371 (504h58m51.642466406s) [upload=443121]: Got next segment. Inline: false Dec 26 16:27:51.771422144 (504h58m51.642505189s) [upload=443121]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:27:51.771427394 (504h58m51.642510429s) [upload=443121]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:27:52.016309964 (504h58m51.887393009s) [seg_pos={0 0}, upload=443121]: Begin upload segment... Dec 26 16:27:52.016379056 (504h58m51.887462101s) [seg_pos={0 0}, upload=443121]: Done begin upload segment. Dec 26 16:27:52.016423610 (504h58m51.887506645s) [noise=true , address=45.91.116.8:4047 , piece_id=EN74CX7XC4RLGJ4VNKSHTNZ6YQ5ADR7UXMNNUSE3XOQD6GCOPXRA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.016433769 (504h58m51.887516804s) [noise=true , address=145.79.227.46:4602 , piece_id=EBVUOKSVSRKQDFBOCP7BZWU4HOSWXIJ5BNSGQ6QX5VTNBY7RB36Q, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.016635383 (504h58m51.887718429s) [noise=true , address=145.79.227.37:4566 , piece_id=BUJYPGEELG5UXS7NNWAC7TMLTOFGB4J5Z4B6U6VHMWG4SBY25WWA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.017118383 (504h58m51.888201418s) [noise=true , address=145.79.227.50:4374 , piece_id=76IQX5RDRU7J3ORVVBC5CIJZKWDFAD3C43F3FLJLOSIKLX5RPF6A, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.017289469 (504h58m51.888372504s) [noise=true , address=45.91.116.12:4226 , piece_id=K3KQIRVUGUWCSMIYNJKTU77UDPF7T2PBK5KKOZSAYQNMD7B6TSKA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.018048635 (504h58m51.88913167s) [noise=true , address=45.91.116.13:4156 , piece_id=XTWDUNXJGISATJMKRJMQFNWTSHMEGBT6A7PSZ6YBD7JLW3H63EFQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.018174414 (504h58m51.889257459s) [noise=true , address=45.91.116.5:4298 , piece_id=WODIDTWZ7FYCDPI6BT2YFUIPMETGNAS5HVUFQ3E6ZYHQFO6IWGUA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.018563374 (504h58m51.889646419s) [noise=true , address=145.79.227.51:4409 , piece_id=QDVTB5JBDKWJZZIV7OXY2KGN6YTSOZRTKZOLSWINHDZPLL2US4CQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.019222299 (504h58m51.890305344s) [noise=true , address=77.90.31.14:4662 , piece_id=GBYWCBM27G5RW5UMQF3ZLDLRT45WUKY3MUMGV3JCV4MYAVT6PW5A, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.019571023 (504h58m51.890654068s) [noise=true , address=145.79.227.48:4301 , piece_id=KRQUIU5B3VGUL6YJ22KBBJVPMK5LWFMCCSS2GMJPNGZCYAVKGJ2Q, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.019723183 (504h58m51.890806229s) [noise=true , address=45.91.116.11:4120 , piece_id=6C6XZBBDJBRZREMQ56T4HBSMIJHFOMCCRNAOSYQO4A4KS6QLC5GQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020133524 (504h58m51.891216559s) [noise=true , address=145.79.227.36:4518 , piece_id=STUHSYSV7MU6GUIAQP5N5JIIYCAUBLZBQ2PWBYLJR35MH73Z7QKQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020149455 (504h58m51.89123249s) [noise=true , address=145.79.227.16:4909 , piece_id=B5OKFWF6TCDCJCYHQO3FRP5OPLO4T4O4FIZAKNUKXTPBZV62AURQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020566770 (504h58m51.891649815s) [noise=true , address=145.79.227.47:4267 , piece_id=FHYMP7JM64IBGVK5KP57F45AE67JYLSL24WMJZP5ROMOL37ENXGQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020699993 (504h58m51.891783038s) [noise=true , address=145.79.227.41:4085 , piece_id=EOFAXKFJZH5WGXDREREJ23WOPW6ZKQWM6PJESRFANSXTNGVABPBA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.023082460 (504h58m51.894165505s) [noise=true , address=145.79.227.32:4836 , piece_id=SW25F4SS4GDPSE6QJT6W5QVIBPEQ7VMI54ZF3EGWWOOD7TQ4BNBA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:53.754638374 (504h58m53.625721419s) [upload=443121]: Next returned nil segment Dec 26 16:27:53.754655587 (504h58m53.625738632s) [upload=443121]: Waiting for error group managing segments... Dec 26 16:27:54.042259825 (504h58m53.91334286s) [noise=true , address=145.79.227.32:4836 , piece_id=SW25F4SS4GDPSE6QJT6W5QVIBPEQ7VMI54ZF3EGWWOOD7TQ4BNBA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.042937176 (504h58m53.914020221s) [noise=true , address=145.79.227.51:4409 , piece_id=QDVTB5JBDKWJZZIV7OXY2KGN6YTSOZRTKZOLSWINHDZPLL2US4CQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.044245196 (504h58m53.915328231s) [noise=true , address=145.79.227.36:4518 , piece_id=STUHSYSV7MU6GUIAQP5N5JIIYCAUBLZBQ2PWBYLJR35MH73Z7QKQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.053542530 (504h58m53.924625585s) [noise=true , address=145.79.227.37:4566 , piece_id=BUJYPGEELG5UXS7NNWAC7TMLTOFGB4J5Z4B6U6VHMWG4SBY25WWA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.054374083 (504h58m53.925457118s) [noise=true , address=145.79.227.48:4301 , piece_id=KRQUIU5B3VGUL6YJ22KBBJVPMK5LWFMCCSS2GMJPNGZCYAVKGJ2Q, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.062316276 (504h58m53.933399321s) [noise=true , address=145.79.227.50:4374 , piece_id=76IQX5RDRU7J3ORVVBC5CIJZKWDFAD3C43F3FLJLOSIKLX5RPF6A, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.065628374 (504h58m53.936711409s) [noise=true , address=145.79.227.46:4602 , piece_id=EBVUOKSVSRKQDFBOCP7BZWU4HOSWXIJ5BNSGQ6QX5VTNBY7RB36Q, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.078233186 (504h58m53.949316221s) [noise=true , address=145.79.227.41:4085 , piece_id=EOFAXKFJZH5WGXDREREJ23WOPW6ZKQWM6PJESRFANSXTNGVABPBA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.095259877 (504h58m53.966342922s) [noise=true , address=145.79.227.16:4909 , piece_id=B5OKFWF6TCDCJCYHQO3FRP5OPLO4T4O4FIZAKNUKXTPBZV62AURQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.101046285 (504h58m53.97212933s) [noise=true , address=145.79.227.47:4267 , piece_id=FHYMP7JM64IBGVK5KP57F45AE67JYLSL24WMJZP5ROMOL37ENXGQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.286875276 (504h58m54.157958322s) [noise=true , address=45.91.116.11:4120 , piece_id=6C6XZBBDJBRZREMQ56T4HBSMIJHFOMCCRNAOSYQO4A4KS6QLC5GQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.381576113 (504h58m54.252659158s) [noise=true , address=45.91.116.12:4226 , piece_id=K3KQIRVUGUWCSMIYNJKTU77UDPF7T2PBK5KKOZSAYQNMD7B6TSKA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.404723019 (504h58m54.275806064s) [noise=true , address=45.91.116.8:4047 , piece_id=EN74CX7XC4RLGJ4VNKSHTNZ6YQ5ADR7UXMNNUSE3XOQD6GCOPXRA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.404747315 (504h58m54.27583035s) [seg_pos={0 0}, upload=443121]: Segment reached optimal threshold of 13 pieces. Dec 26 16:27:54.405263158 (504h58m54.276346203s) [noise=true , address=45.91.116.5:4298 , piece_id=WODIDTWZ7FYCDPI6BT2YFUIPMETGNAS5HVUFQ3E6ZYHQFO6IWGUA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:60970->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 16:27:54.405273627 (504h58m54.276356682s) [noise=true , address=45.91.116.5:4298 , piece_id=WODIDTWZ7FYCDPI6BT2YFUIPMETGNAS5HVUFQ3E6ZYHQFO6IWGUA, seg_pos={0 0}, upload=443121]: Long tail cancellation due to optimal threshold Dec 26 16:27:54.405349602 (504h58m54.276432637s) [noise=true , address=45.91.116.13:4156 , piece_id=XTWDUNXJGISATJMKRJMQFNWTSHMEGBT6A7PSZ6YBD7JLW3H63EFQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 16:27:54.405365733 (504h58m54.276448768s) [noise=true , address=45.91.116.13:4156 , piece_id=XTWDUNXJGISATJMKRJMQFNWTSHMEGBT6A7PSZ6YBD7JLW3H63EFQ, seg_pos={0 0}, upload=443121]: Long tail cancellation due to optimal threshold Dec 26 16:27:54.405542509 (504h58m54.276625544s) [noise=true , address=77.90.31.14:4662 , piece_id=GBYWCBM27G5RW5UMQF3ZLDLRT45WUKY3MUMGV3JCV4MYAVT6PW5A, seg_pos={0 0}, upload=443121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:35100->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:35100->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35100->77.90.31.14:4662: use of closed network connection Dec 26 16:27:54.405549372 (504h58m54.276632417s) [noise=true , address=77.90.31.14:4662 , piece_id=GBYWCBM27G5RW5UMQF3ZLDLRT45WUKY3MUMGV3JCV4MYAVT6PW5A, seg_pos={0 0}, upload=443121]: Long tail cancellation due to optimal threshold Dec 26 16:27:54.405594718 (504h58m54.276677763s) [seg_pos={0 0}, upload=443121]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:27:54.405861196 (504h58m54.276944241s) [upload=443121]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:27:54.405868820 (504h58m54.276951855s) [upload=443121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:27:54.704991443 (504h58m54.576074478s) [upload=443121]: Done uploading segments... Dec 26 16:27:54.705011470 (504h58m54.576094505s) [upload=443121]: Upload finished. err: <nil> Duration: 6.813618179s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [X....X....X..X...X.......XX..........X.........X..X.........] duration: 37.125611345s trace: https://jaeger.dev.storj.io/trace/3e3dfe0984f19d4e FAIL (2025-12-26 16:28:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.466998847 (504h51m15.338081892s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467028614 (504h51m15.338111659s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467359594 (504h51m15.338442639s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467908379 (504h51m15.338991424s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.468520054 (504h51m15.339603089s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469055293 (504h51m15.340138328s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469497605 (504h51m15.34058064s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469973642 (504h51m15.341056687s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.470454397 (504h51m15.341537443s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471043630 (504h51m15.342126695s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471470552 (504h51m15.342553587s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471676735 (504h51m15.34275977s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472062900 (504h51m15.343145935s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472369224 (504h51m15.343452259s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472823849 (504h51m15.343906884s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:16.743410939 (504h51m16.614493984s) [upload=443005]: Next returned nil segment Dec 26 16:20:16.743428862 (504h51m16.614511907s) [upload=443005]: Waiting for error group managing segments... Dec 26 16:20:17.091086984 (504h51m16.962170029s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.389229539 (504h51m17.260312594s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.464846438 (504h51m17.335929483s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.542075705 (504h51m17.41315875s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.623422163 (504h51m17.494505208s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.665409184 (504h51m17.536492219s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.301995626 (504h51m18.173078671s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.311693532 (504h51m18.182776567s) [noise=true , address=145.79.227.34:4915 , piece_id=7ZBWNPUF54BEGIXTDRDI3J7HNEZ63MEXKUBMRZW6PFWWNCEAH3AA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.340201235 (504h51m18.21128428s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.379076200 (504h51m18.250159245s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.474362322 (504h51m18.345445367s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.555133063 (504h51m18.426216108s) [noise=true , address=77.90.31.17:4791 , piece_id=QFHBBE4B5DWZU7HTAARXW655MKX4YPKPOXVM77CH5DCPSLZ7UGNA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.707673447 (504h51m18.578756492s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734654664 (504h51m18.605737709s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734678970 (504h51m18.605762005s) [seg_pos={0 0}, upload=443005]: Segment reached optimal threshold of 14 pieces. Dec 26 16:20:18.734942853 (504h51m18.606025888s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.734952842 (504h51m18.606035887s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735555389 (504h51m18.606638434s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection Dec 26 16:20:18.735566701 (504h51m18.606649746s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735681349 (504h51m18.606764394s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735699113 (504h51m18.606782148s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735826605 (504h51m18.60690968s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735837737 (504h51m18.606920782s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735908392 (504h51m18.606991447s) [seg_pos={0 0}, upload=443005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:20:18.736024182 (504h51m18.607107227s) [upload=443005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:20:18.736031275 (504h51m18.60711431s) [upload=443005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:20:19.031573339 (504h51m18.902656384s) [upload=443005]: Done uploading segments... Dec 26 16:20:19.031602895 (504h51m18.90268594s) [upload=443005]: Upload finished. err: <nil> Duration: 37.12564043s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [XX...XX.........X.............XXX...X...XX.....XX..........X] duration: 5.001933354s trace: https://jaeger.dev.storj.io/trace/4d8eedf07808cbf FAIL (2025-12-26 16:28:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001955977s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 16.624073778s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:28:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 16.624103634s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.XX...XX.........X.............XXX...X...XX.....XX..........] duration: 3.454150996s trace: https://jaeger.dev.storj.io/trace/7e43da40a4822a39 FAIL (2025-12-26 16:28:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.454178068s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...................X.....X..XX..XXX.X...XX.X.XX.X....XX...X.] duration: 6.096754806s trace: https://jaeger.dev.storj.io/trace/7db2f5600219fec2 FAIL (2025-12-26 16:28:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8 (504h58m51.642207893s) [upload=443121]: Done starting upload Dec 26 16:27:51.771241651 (504h58m51.642324686s) [upload=443121]: Uploading segments... Dec 26 16:27:51.771383371 (504h58m51.642466406s) [upload=443121]: Got next segment. Inline: false Dec 26 16:27:51.771422144 (504h58m51.642505189s) [upload=443121]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:27:51.771427394 (504h58m51.642510429s) [upload=443121]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:27:52.016309964 (504h58m51.887393009s) [seg_pos={0 0}, upload=443121]: Begin upload segment... Dec 26 16:27:52.016379056 (504h58m51.887462101s) [seg_pos={0 0}, upload=443121]: Done begin upload segment. Dec 26 16:27:52.016423610 (504h58m51.887506645s) [noise=true , address=45.91.116.8:4047 , piece_id=EN74CX7XC4RLGJ4VNKSHTNZ6YQ5ADR7UXMNNUSE3XOQD6GCOPXRA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.016433769 (504h58m51.887516804s) [noise=true , address=145.79.227.46:4602 , piece_id=EBVUOKSVSRKQDFBOCP7BZWU4HOSWXIJ5BNSGQ6QX5VTNBY7RB36Q, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.016635383 (504h58m51.887718429s) [noise=true , address=145.79.227.37:4566 , piece_id=BUJYPGEELG5UXS7NNWAC7TMLTOFGB4J5Z4B6U6VHMWG4SBY25WWA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.017118383 (504h58m51.888201418s) [noise=true , address=145.79.227.50:4374 , piece_id=76IQX5RDRU7J3ORVVBC5CIJZKWDFAD3C43F3FLJLOSIKLX5RPF6A, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.017289469 (504h58m51.888372504s) [noise=true , address=45.91.116.12:4226 , piece_id=K3KQIRVUGUWCSMIYNJKTU77UDPF7T2PBK5KKOZSAYQNMD7B6TSKA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.018048635 (504h58m51.88913167s) [noise=true , address=45.91.116.13:4156 , piece_id=XTWDUNXJGISATJMKRJMQFNWTSHMEGBT6A7PSZ6YBD7JLW3H63EFQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.018174414 (504h58m51.889257459s) [noise=true , address=45.91.116.5:4298 , piece_id=WODIDTWZ7FYCDPI6BT2YFUIPMETGNAS5HVUFQ3E6ZYHQFO6IWGUA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.018563374 (504h58m51.889646419s) [noise=true , address=145.79.227.51:4409 , piece_id=QDVTB5JBDKWJZZIV7OXY2KGN6YTSOZRTKZOLSWINHDZPLL2US4CQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.019222299 (504h58m51.890305344s) [noise=true , address=77.90.31.14:4662 , piece_id=GBYWCBM27G5RW5UMQF3ZLDLRT45WUKY3MUMGV3JCV4MYAVT6PW5A, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.019571023 (504h58m51.890654068s) [noise=true , address=145.79.227.48:4301 , piece_id=KRQUIU5B3VGUL6YJ22KBBJVPMK5LWFMCCSS2GMJPNGZCYAVKGJ2Q, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.019723183 (504h58m51.890806229s) [noise=true , address=45.91.116.11:4120 , piece_id=6C6XZBBDJBRZREMQ56T4HBSMIJHFOMCCRNAOSYQO4A4KS6QLC5GQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020133524 (504h58m51.891216559s) [noise=true , address=145.79.227.36:4518 , piece_id=STUHSYSV7MU6GUIAQP5N5JIIYCAUBLZBQ2PWBYLJR35MH73Z7QKQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020149455 (504h58m51.89123249s) [noise=true , address=145.79.227.16:4909 , piece_id=B5OKFWF6TCDCJCYHQO3FRP5OPLO4T4O4FIZAKNUKXTPBZV62AURQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020566770 (504h58m51.891649815s) [noise=true , address=145.79.227.47:4267 , piece_id=FHYMP7JM64IBGVK5KP57F45AE67JYLSL24WMJZP5ROMOL37ENXGQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020699993 (504h58m51.891783038s) [noise=true , address=145.79.227.41:4085 , piece_id=EOFAXKFJZH5WGXDREREJ23WOPW6ZKQWM6PJESRFANSXTNGVABPBA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.023082460 (504h58m51.894165505s) [noise=true , address=145.79.227.32:4836 , piece_id=SW25F4SS4GDPSE6QJT6W5QVIBPEQ7VMI54ZF3EGWWOOD7TQ4BNBA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:53.754638374 (504h58m53.625721419s) [upload=443121]: Next returned nil segment Dec 26 16:27:53.754655587 (504h58m53.625738632s) [upload=443121]: Waiting for error group managing segments... Dec 26 16:27:54.042259825 (504h58m53.91334286s) [noise=true , address=145.79.227.32:4836 , piece_id=SW25F4SS4GDPSE6QJT6W5QVIBPEQ7VMI54ZF3EGWWOOD7TQ4BNBA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.042937176 (504h58m53.914020221s) [noise=true , address=145.79.227.51:4409 , piece_id=QDVTB5JBDKWJZZIV7OXY2KGN6YTSOZRTKZOLSWINHDZPLL2US4CQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.044245196 (504h58m53.915328231s) [noise=true , address=145.79.227.36:4518 , piece_id=STUHSYSV7MU6GUIAQP5N5JIIYCAUBLZBQ2PWBYLJR35MH73Z7QKQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.053542530 (504h58m53.924625585s) [noise=true , address=145.79.227.37:4566 , piece_id=BUJYPGEELG5UXS7NNWAC7TMLTOFGB4J5Z4B6U6VHMWG4SBY25WWA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.054374083 (504h58m53.925457118s) [noise=true , address=145.79.227.48:4301 , piece_id=KRQUIU5B3VGUL6YJ22KBBJVPMK5LWFMCCSS2GMJPNGZCYAVKGJ2Q, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.062316276 (504h58m53.933399321s) [noise=true , address=145.79.227.50:4374 , piece_id=76IQX5RDRU7J3ORVVBC5CIJZKWDFAD3C43F3FLJLOSIKLX5RPF6A, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.065628374 (504h58m53.936711409s) [noise=true , address=145.79.227.46:4602 , piece_id=EBVUOKSVSRKQDFBOCP7BZWU4HOSWXIJ5BNSGQ6QX5VTNBY7RB36Q, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.078233186 (504h58m53.949316221s) [noise=true , address=145.79.227.41:4085 , piece_id=EOFAXKFJZH5WGXDREREJ23WOPW6ZKQWM6PJESRFANSXTNGVABPBA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.095259877 (504h58m53.966342922s) [noise=true , address=145.79.227.16:4909 , piece_id=B5OKFWF6TCDCJCYHQO3FRP5OPLO4T4O4FIZAKNUKXTPBZV62AURQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.101046285 (504h58m53.97212933s) [noise=true , address=145.79.227.47:4267 , piece_id=FHYMP7JM64IBGVK5KP57F45AE67JYLSL24WMJZP5ROMOL37ENXGQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.286875276 (504h58m54.157958322s) [noise=true , address=45.91.116.11:4120 , piece_id=6C6XZBBDJBRZREMQ56T4HBSMIJHFOMCCRNAOSYQO4A4KS6QLC5GQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.381576113 (504h58m54.252659158s) [noise=true , address=45.91.116.12:4226 , piece_id=K3KQIRVUGUWCSMIYNJKTU77UDPF7T2PBK5KKOZSAYQNMD7B6TSKA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.404723019 (504h58m54.275806064s) [noise=true , address=45.91.116.8:4047 , piece_id=EN74CX7XC4RLGJ4VNKSHTNZ6YQ5ADR7UXMNNUSE3XOQD6GCOPXRA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.404747315 (504h58m54.27583035s) [seg_pos={0 0}, upload=443121]: Segment reached optimal threshold of 13 pieces. Dec 26 16:27:54.405263158 (504h58m54.276346203s) [noise=true , address=45.91.116.5:4298 , piece_id=WODIDTWZ7FYCDPI6BT2YFUIPMETGNAS5HVUFQ3E6ZYHQFO6IWGUA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:60970->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 16:27:54.405273627 (504h58m54.276356682s) [noise=true , address=45.91.116.5:4298 , piece_id=WODIDTWZ7FYCDPI6BT2YFUIPMETGNAS5HVUFQ3E6ZYHQFO6IWGUA, seg_pos={0 0}, upload=443121]: Long tail cancellation due to optimal threshold Dec 26 16:27:54.405349602 (504h58m54.276432637s) [noise=true , address=45.91.116.13:4156 , piece_id=XTWDUNXJGISATJMKRJMQFNWTSHMEGBT6A7PSZ6YBD7JLW3H63EFQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 16:27:54.405365733 (504h58m54.276448768s) [noise=true , address=45.91.116.13:4156 , piece_id=XTWDUNXJGISATJMKRJMQFNWTSHMEGBT6A7PSZ6YBD7JLW3H63EFQ, seg_pos={0 0}, upload=443121]: Long tail cancellation due to optimal threshold Dec 26 16:27:54.405542509 (504h58m54.276625544s) [noise=true , address=77.90.31.14:4662 , piece_id=GBYWCBM27G5RW5UMQF3ZLDLRT45WUKY3MUMGV3JCV4MYAVT6PW5A, seg_pos={0 0}, upload=443121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:35100->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:35100->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35100->77.90.31.14:4662: use of closed network connection Dec 26 16:27:54.405549372 (504h58m54.276632417s) [noise=true , address=77.90.31.14:4662 , piece_id=GBYWCBM27G5RW5UMQF3ZLDLRT45WUKY3MUMGV3JCV4MYAVT6PW5A, seg_pos={0 0}, upload=443121]: Long tail cancellation due to optimal threshold Dec 26 16:27:54.405594718 (504h58m54.276677763s) [seg_pos={0 0}, upload=443121]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:27:54.405861196 (504h58m54.276944241s) [upload=443121]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:27:54.405868820 (504h58m54.276951855s) [upload=443121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:27:54.704991443 (504h58m54.576074478s) [upload=443121]: Done uploading segments... Dec 26 16:27:54.705011470 (504h58m54.576094505s) [upload=443121]: Upload finished. err: <nil> Duration: 6.096769022s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X...................X.....X..XX..XXX.X...XX.X.XX.X....XX...X] duration: 30.003578301s trace: https://jaeger.dev.storj.io/trace/7db2f5600219fec2 FAIL (2025-12-26 16:28:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8 (504h58m51.642207893s) [upload=443121]: Done starting upload Dec 26 16:27:51.771241651 (504h58m51.642324686s) [upload=443121]: Uploading segments... Dec 26 16:27:51.771383371 (504h58m51.642466406s) [upload=443121]: Got next segment. Inline: false Dec 26 16:27:51.771422144 (504h58m51.642505189s) [upload=443121]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:27:51.771427394 (504h58m51.642510429s) [upload=443121]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:27:52.016309964 (504h58m51.887393009s) [seg_pos={0 0}, upload=443121]: Begin upload segment... Dec 26 16:27:52.016379056 (504h58m51.887462101s) [seg_pos={0 0}, upload=443121]: Done begin upload segment. Dec 26 16:27:52.016423610 (504h58m51.887506645s) [noise=true , address=45.91.116.8:4047 , piece_id=EN74CX7XC4RLGJ4VNKSHTNZ6YQ5ADR7UXMNNUSE3XOQD6GCOPXRA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.016433769 (504h58m51.887516804s) [noise=true , address=145.79.227.46:4602 , piece_id=EBVUOKSVSRKQDFBOCP7BZWU4HOSWXIJ5BNSGQ6QX5VTNBY7RB36Q, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.016635383 (504h58m51.887718429s) [noise=true , address=145.79.227.37:4566 , piece_id=BUJYPGEELG5UXS7NNWAC7TMLTOFGB4J5Z4B6U6VHMWG4SBY25WWA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.017118383 (504h58m51.888201418s) [noise=true , address=145.79.227.50:4374 , piece_id=76IQX5RDRU7J3ORVVBC5CIJZKWDFAD3C43F3FLJLOSIKLX5RPF6A, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.017289469 (504h58m51.888372504s) [noise=true , address=45.91.116.12:4226 , piece_id=K3KQIRVUGUWCSMIYNJKTU77UDPF7T2PBK5KKOZSAYQNMD7B6TSKA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.018048635 (504h58m51.88913167s) [noise=true , address=45.91.116.13:4156 , piece_id=XTWDUNXJGISATJMKRJMQFNWTSHMEGBT6A7PSZ6YBD7JLW3H63EFQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.018174414 (504h58m51.889257459s) [noise=true , address=45.91.116.5:4298 , piece_id=WODIDTWZ7FYCDPI6BT2YFUIPMETGNAS5HVUFQ3E6ZYHQFO6IWGUA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.018563374 (504h58m51.889646419s) [noise=true , address=145.79.227.51:4409 , piece_id=QDVTB5JBDKWJZZIV7OXY2KGN6YTSOZRTKZOLSWINHDZPLL2US4CQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.019222299 (504h58m51.890305344s) [noise=true , address=77.90.31.14:4662 , piece_id=GBYWCBM27G5RW5UMQF3ZLDLRT45WUKY3MUMGV3JCV4MYAVT6PW5A, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.019571023 (504h58m51.890654068s) [noise=true , address=145.79.227.48:4301 , piece_id=KRQUIU5B3VGUL6YJ22KBBJVPMK5LWFMCCSS2GMJPNGZCYAVKGJ2Q, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.019723183 (504h58m51.890806229s) [noise=true , address=45.91.116.11:4120 , piece_id=6C6XZBBDJBRZREMQ56T4HBSMIJHFOMCCRNAOSYQO4A4KS6QLC5GQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020133524 (504h58m51.891216559s) [noise=true , address=145.79.227.36:4518 , piece_id=STUHSYSV7MU6GUIAQP5N5JIIYCAUBLZBQ2PWBYLJR35MH73Z7QKQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020149455 (504h58m51.89123249s) [noise=true , address=145.79.227.16:4909 , piece_id=B5OKFWF6TCDCJCYHQO3FRP5OPLO4T4O4FIZAKNUKXTPBZV62AURQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020566770 (504h58m51.891649815s) [noise=true , address=145.79.227.47:4267 , piece_id=FHYMP7JM64IBGVK5KP57F45AE67JYLSL24WMJZP5ROMOL37ENXGQ, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.020699993 (504h58m51.891783038s) [noise=true , address=145.79.227.41:4085 , piece_id=EOFAXKFJZH5WGXDREREJ23WOPW6ZKQWM6PJESRFANSXTNGVABPBA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:52.023082460 (504h58m51.894165505s) [noise=true , address=145.79.227.32:4836 , piece_id=SW25F4SS4GDPSE6QJT6W5QVIBPEQ7VMI54ZF3EGWWOOD7TQ4BNBA, seg_pos={0 0}, upload=443121]: Uploading piece... Dec 26 16:27:53.754638374 (504h58m53.625721419s) [upload=443121]: Next returned nil segment Dec 26 16:27:53.754655587 (504h58m53.625738632s) [upload=443121]: Waiting for error group managing segments... Dec 26 16:27:54.042259825 (504h58m53.91334286s) [noise=true , address=145.79.227.32:4836 , piece_id=SW25F4SS4GDPSE6QJT6W5QVIBPEQ7VMI54ZF3EGWWOOD7TQ4BNBA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.042937176 (504h58m53.914020221s) [noise=true , address=145.79.227.51:4409 , piece_id=QDVTB5JBDKWJZZIV7OXY2KGN6YTSOZRTKZOLSWINHDZPLL2US4CQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.044245196 (504h58m53.915328231s) [noise=true , address=145.79.227.36:4518 , piece_id=STUHSYSV7MU6GUIAQP5N5JIIYCAUBLZBQ2PWBYLJR35MH73Z7QKQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.053542530 (504h58m53.924625585s) [noise=true , address=145.79.227.37:4566 , piece_id=BUJYPGEELG5UXS7NNWAC7TMLTOFGB4J5Z4B6U6VHMWG4SBY25WWA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.054374083 (504h58m53.925457118s) [noise=true , address=145.79.227.48:4301 , piece_id=KRQUIU5B3VGUL6YJ22KBBJVPMK5LWFMCCSS2GMJPNGZCYAVKGJ2Q, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.062316276 (504h58m53.933399321s) [noise=true , address=145.79.227.50:4374 , piece_id=76IQX5RDRU7J3ORVVBC5CIJZKWDFAD3C43F3FLJLOSIKLX5RPF6A, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.065628374 (504h58m53.936711409s) [noise=true , address=145.79.227.46:4602 , piece_id=EBVUOKSVSRKQDFBOCP7BZWU4HOSWXIJ5BNSGQ6QX5VTNBY7RB36Q, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.078233186 (504h58m53.949316221s) [noise=true , address=145.79.227.41:4085 , piece_id=EOFAXKFJZH5WGXDREREJ23WOPW6ZKQWM6PJESRFANSXTNGVABPBA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.095259877 (504h58m53.966342922s) [noise=true , address=145.79.227.16:4909 , piece_id=B5OKFWF6TCDCJCYHQO3FRP5OPLO4T4O4FIZAKNUKXTPBZV62AURQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.101046285 (504h58m53.97212933s) [noise=true , address=145.79.227.47:4267 , piece_id=FHYMP7JM64IBGVK5KP57F45AE67JYLSL24WMJZP5ROMOL37ENXGQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.286875276 (504h58m54.157958322s) [noise=true , address=45.91.116.11:4120 , piece_id=6C6XZBBDJBRZREMQ56T4HBSMIJHFOMCCRNAOSYQO4A4KS6QLC5GQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.381576113 (504h58m54.252659158s) [noise=true , address=45.91.116.12:4226 , piece_id=K3KQIRVUGUWCSMIYNJKTU77UDPF7T2PBK5KKOZSAYQNMD7B6TSKA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.404723019 (504h58m54.275806064s) [noise=true , address=45.91.116.8:4047 , piece_id=EN74CX7XC4RLGJ4VNKSHTNZ6YQ5ADR7UXMNNUSE3XOQD6GCOPXRA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: <nil> Dec 26 16:27:54.404747315 (504h58m54.27583035s) [seg_pos={0 0}, upload=443121]: Segment reached optimal threshold of 13 pieces. Dec 26 16:27:54.405263158 (504h58m54.276346203s) [noise=true , address=45.91.116.5:4298 , piece_id=WODIDTWZ7FYCDPI6BT2YFUIPMETGNAS5HVUFQ3E6ZYHQFO6IWGUA, seg_pos={0 0}, upload=443121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:60970->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 16:27:54.405273627 (504h58m54.276356682s) [noise=true , address=45.91.116.5:4298 , piece_id=WODIDTWZ7FYCDPI6BT2YFUIPMETGNAS5HVUFQ3E6ZYHQFO6IWGUA, seg_pos={0 0}, upload=443121]: Long tail cancellation due to optimal threshold Dec 26 16:27:54.405349602 (504h58m54.276432637s) [noise=true , address=45.91.116.13:4156 , piece_id=XTWDUNXJGISATJMKRJMQFNWTSHMEGBT6A7PSZ6YBD7JLW3H63EFQ, seg_pos={0 0}, upload=443121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 16:27:54.405365733 (504h58m54.276448768s) [noise=true , address=45.91.116.13:4156 , piece_id=XTWDUNXJGISATJMKRJMQFNWTSHMEGBT6A7PSZ6YBD7JLW3H63EFQ, seg_pos={0 0}, upload=443121]: Long tail cancellation due to optimal threshold Dec 26 16:27:54.405542509 (504h58m54.276625544s) [noise=true , address=77.90.31.14:4662 , piece_id=GBYWCBM27G5RW5UMQF3ZLDLRT45WUKY3MUMGV3JCV4MYAVT6PW5A, seg_pos={0 0}, upload=443121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:35100->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:35100->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35100->77.90.31.14:4662: use of closed network connection Dec 26 16:27:54.405549372 (504h58m54.276632417s) [noise=true , address=77.90.31.14:4662 , piece_id=GBYWCBM27G5RW5UMQF3ZLDLRT45WUKY3MUMGV3JCV4MYAVT6PW5A, seg_pos={0 0}, upload=443121]: Long tail cancellation due to optimal threshold Dec 26 16:27:54.405594718 (504h58m54.276677763s) [seg_pos={0 0}, upload=443121]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:27:54.405861196 (504h58m54.276944241s) [upload=443121]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:27:54.405868820 (504h58m54.276951855s) [upload=443121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:27:54.704991443 (504h58m54.576074478s) [upload=443121]: Done uploading segments... Dec 26 16:27:54.705011470 (504h58m54.576094505s) [upload=443121]: Upload finished. err: <nil> Duration: 30.003601183s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 4.997712967s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:28:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 4.997743757s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..XX...XX.........X.............XXX...X...XX.....XX.........] duration: 4.370941625s trace: https://jaeger.dev.storj.io/trace/7e43da40a4822a39 FAIL (2025-12-26 16:28:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.370973025s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X....X....X..X...X.......XX..........X.........X..X........] duration: 39.592314658s trace: https://jaeger.dev.storj.io/trace/3e3dfe0984f19d4e FAIL (2025-12-26 16:27:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.466998847 (504h51m15.338081892s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467028614 (504h51m15.338111659s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467359594 (504h51m15.338442639s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467908379 (504h51m15.338991424s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.468520054 (504h51m15.339603089s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469055293 (504h51m15.340138328s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469497605 (504h51m15.34058064s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469973642 (504h51m15.341056687s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.470454397 (504h51m15.341537443s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471043630 (504h51m15.342126695s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471470552 (504h51m15.342553587s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471676735 (504h51m15.34275977s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472062900 (504h51m15.343145935s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472369224 (504h51m15.343452259s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472823849 (504h51m15.343906884s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:16.743410939 (504h51m16.614493984s) [upload=443005]: Next returned nil segment Dec 26 16:20:16.743428862 (504h51m16.614511907s) [upload=443005]: Waiting for error group managing segments... Dec 26 16:20:17.091086984 (504h51m16.962170029s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.389229539 (504h51m17.260312594s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.464846438 (504h51m17.335929483s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.542075705 (504h51m17.41315875s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.623422163 (504h51m17.494505208s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.665409184 (504h51m17.536492219s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.301995626 (504h51m18.173078671s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.311693532 (504h51m18.182776567s) [noise=true , address=145.79.227.34:4915 , piece_id=7ZBWNPUF54BEGIXTDRDI3J7HNEZ63MEXKUBMRZW6PFWWNCEAH3AA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.340201235 (504h51m18.21128428s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.379076200 (504h51m18.250159245s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.474362322 (504h51m18.345445367s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.555133063 (504h51m18.426216108s) [noise=true , address=77.90.31.17:4791 , piece_id=QFHBBE4B5DWZU7HTAARXW655MKX4YPKPOXVM77CH5DCPSLZ7UGNA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.707673447 (504h51m18.578756492s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734654664 (504h51m18.605737709s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734678970 (504h51m18.605762005s) [seg_pos={0 0}, upload=443005]: Segment reached optimal threshold of 14 pieces. Dec 26 16:20:18.734942853 (504h51m18.606025888s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.734952842 (504h51m18.606035887s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735555389 (504h51m18.606638434s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection Dec 26 16:20:18.735566701 (504h51m18.606649746s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735681349 (504h51m18.606764394s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735699113 (504h51m18.606782148s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735826605 (504h51m18.60690968s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735837737 (504h51m18.606920782s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735908392 (504h51m18.606991447s) [seg_pos={0 0}, upload=443005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:20:18.736024182 (504h51m18.607107227s) [upload=443005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:20:18.736031275 (504h51m18.60711431s) [upload=443005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:20:19.031573339 (504h51m18.902656384s) [upload=443005]: Done uploading segments... Dec 26 16:20:19.031602895 (504h51m18.90268594s) [upload=443005]: Upload finished. err: <nil> Duration: 39.592329636s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 10.926909553s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:27:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 10.926935943s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...XX...XX.........X.............XXX...X...XX.....XX........] duration: 5.000524011s trace: https://jaeger.dev.storj.io/trace/7e43da40a4822a39 FAIL (2025-12-26 16:27:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000548327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX...................X.....X..XX..XXX.X...XX.X.XX.X....XX...] duration: 5.880939715s trace: https://jaeger.dev.storj.io/trace/3f4ee18d6c20a87 FAIL (2025-12-26 16:27:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 16:25:12.724473263 (504h56m12.595556298s) [upload=443079]: Starting upload Dec 26 16:25:12.724540882 (504h56m12.595623917s) [upload=443079]: Done starting upload Dec 26 16:25:12.724757634 (504h56m12.595840669s) [upload=443079]: Uploading segments... Dec 26 16:25:12.724851774 (504h56m12.595934809s) [upload=443079]: Got next segment. Inline: false Dec 26 16:25:12.724881711 (504h56m12.595964756s) [upload=443079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:25:12.724885949 (504h56m12.595968994s) [upload=443079]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:25:12.967631409 (504h56m12.838714454s) [seg_pos={0 0}, upload=443079]: Begin upload segment... Dec 26 16:25:12.967672637 (504h56m12.838755682s) [seg_pos={0 0}, upload=443079]: Done begin upload segment. Dec 26 16:25:12.967717332 (504h56m12.838800367s) [noise=true , address=145.79.227.49:4338 , piece_id=7YHEDJHWYYKEMOJOT7WDSYWMVCCRGGCH5XD3GRWSAQAT3T6XKLYA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.967727943 (504h56m12.838810978s) [noise=true , address=145.79.227.13:4804 , piece_id=5CAQV5NZHKLXGXLJMKXR74FWW46SOYOPRVLK2PR5PMKRW7E5JGVQ, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.968427765 (504h56m12.83951081s) [noise=true , address=145.79.227.51:4409 , piece_id=FPENA4MRRTBEPG2I6IE52NXEQMAHJCNRI4FE67ZXEHJOILJMPFCQ, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.968554516 (504h56m12.839637561s) [noise=true , address=145.79.227.37:4566 , piece_id=P26XQX6NSU2NLXXOW4MJWNB23KK5ZO7EVI3UOTIJGUTTT32GODHA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.968891548 (504h56m12.839974583s) [noise=true , address=45.91.116.12:4227 , piece_id=BMMRT4Z57MH4V3AVKLL5VBRZXU54PCSUPVVEEIITNL6MAECIWYVA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.968939940 (504h56m12.840022985s) [noise=true , address=145.79.227.5:4222 , piece_id=ZORDZIWYYSJ72IWQOUG57K2XQJJWTQPXQ46PEFGJTBDXMCUG67OA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.969328460 (504h56m12.840411495s) [noise=true , address=145.79.227.46:4603 , piece_id=BGDTN5EXNU2EKMRRNDYFEHK75JITOPRTNW5FXDN66IGSSBFZIFJQ, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.969337808 (504h56m12.840420843s) [noise=true , address=145.79.227.16:4911 , piece_id=D3FD463XSS4WWDSLHVMZCQICZMAM7OXNTS5JNVDRXBPQOHDSDRWA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.969781403 (504h56m12.840864448s) [noise=true , address=145.79.227.32:4836 , piece_id=HUZJ3Z46K4IHBOUBLN7GHMYKD67QLJ5EWWNJGWXHO3B4WQO6KCLA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.969942158 (504h56m12.841025193s) [noise=true , address=77.90.31.13:4629 , piece_id=E5DLCMAOGNWZ3NQZNL6YMYYKFBCWIHXTX4DUTY2OFREZNMDUFBRA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.970172437 (504h56m12.841255472s) [noise=true , address=45.91.116.10:4084 , piece_id=HRQGIXXVFG7GHGK3C7BWKG7JIOZKXTJDOPVNF22VBURLMKUDMKDA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.970560686 (504h56m12.841643721s) [noise=true , address=45.91.116.6:4973 , piece_id=A243H6Z5EPE73AR24WOE3JL4HCTUCW6MRAZZQOKOOJ27DKVOZPGA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.970972701 (504h56m12.842055736s) [noise=true , address=45.91.116.8:4048 , piece_id=WX6WACST3J2JELW765CS4U3TCNREHRQTTUINYQGLW7WOM4RE72YA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.971342765 (504h56m12.84242581s) [noise=true , address=145.79.227.2:4114 , piece_id=7QGITYUBVGZHJGN6DVR3ZOSFIL4LMO5T6J3BNZEUNF366FHFMUUQ, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.971722348 (504h56m12.842805393s) [noise=true , address=145.79.227.11:4730 , piece_id=D35YMJIHH7ABNWVSAYXO2RC3SDE3GGWOAXWHQ2ERDEQXOINRHAGA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.972829446 (504h56m12.843912491s) [noise=true , address=145.79.227.35:4483 , piece_id=P3WYWEADNCLVRAYBUKQ7DLHDGLFPCFIMZYAA5MXDX42X5G7TFPMA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:14.675166383 (504h56m14.546249418s) [upload=443079]: Next returned nil segment Dec 26 16:25:14.675179578 (504h56m14.546262613s) [upload=443079]: Waiting for error group managing segments... Dec 26 16:25:14.965434624 (504h56m14.83651766s) [noise=true , address=145.79.227.32:4836 , piece_id=HUZJ3Z46K4IHBOUBLN7GHMYKD67QLJ5EWWNJGWXHO3B4WQO6KCLA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.965969002 (504h56m14.837052047s) [noise=true , address=145.79.227.49:4338 , piece_id=7YHEDJHWYYKEMOJOT7WDSYWMVCCRGGCH5XD3GRWSAQAT3T6XKLYA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.973444016 (504h56m14.844527051s) [noise=true , address=145.79.227.5:4222 , piece_id=ZORDZIWYYSJ72IWQOUG57K2XQJJWTQPXQ46PEFGJTBDXMCUG67OA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.974671563 (504h56m14.845754598s) [noise=true , address=145.79.227.11:4730 , piece_id=D35YMJIHH7ABNWVSAYXO2RC3SDE3GGWOAXWHQ2ERDEQXOINRHAGA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.986783416 (504h56m14.857866461s) [noise=true , address=145.79.227.46:4603 , piece_id=BGDTN5EXNU2EKMRRNDYFEHK75JITOPRTNW5FXDN66IGSSBFZIFJQ, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.988435422 (504h56m14.859518457s) [noise=true , address=145.79.227.51:4409 , piece_id=FPENA4MRRTBEPG2I6IE52NXEQMAHJCNRI4FE67ZXEHJOILJMPFCQ, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.992004167 (504h56m14.863087212s) [noise=true , address=145.79.227.37:4566 , piece_id=P26XQX6NSU2NLXXOW4MJWNB23KK5ZO7EVI3UOTIJGUTTT32GODHA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:15.004627965 (504h56m14.875711011s) [noise=true , address=145.79.227.13:4804 , piece_id=5CAQV5NZHKLXGXLJMKXR74FWW46SOYOPRVLK2PR5PMKRW7E5JGVQ, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:15.006485713 (504h56m14.877568748s) [noise=true , address=145.79.227.2:4114 , piece_id=7QGITYUBVGZHJGN6DVR3ZOSFIL4LMO5T6J3BNZEUNF366FHFMUUQ, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:15.225128197 (504h56m15.096211242s) [noise=true , address=45.91.116.12:4227 , piece_id=BMMRT4Z57MH4V3AVKLL5VBRZXU54PCSUPVVEEIITNL6MAECIWYVA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:15.236042749 (504h56m15.107125794s) [noise=true , address=45.91.116.8:4048 , piece_id=WX6WACST3J2JELW765CS4U3TCNREHRQTTUINYQGLW7WOM4RE72YA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:15.371498702 (504h56m15.242581737s) [noise=true , address=45.91.116.6:4973 , piece_id=A243H6Z5EPE73AR24WOE3JL4HCTUCW6MRAZZQOKOOJ27DKVOZPGA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:16.500662196 (504h56m16.371745231s) [noise=true , address=145.79.227.16:4911 , piece_id=D3FD463XSS4WWDSLHVMZCQICZMAM7OXNTS5JNVDRXBPQOHDSDRWA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:16.500699437 (504h56m16.371782472s) [seg_pos={0 0}, upload=443079]: Segment reached optimal threshold of 13 pieces. Dec 26 16:25:16.500923443 (504h56m16.372006488s) [noise=true , address=77.90.31.13:4629 , piece_id=E5DLCMAOGNWZ3NQZNL6YMYYKFBCWIHXTX4DUTY2OFREZNMDUFBRA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:16.500932931 (504h56m16.372015966s) [noise=true , address=77.90.31.13:4629 , piece_id=E5DLCMAOGNWZ3NQZNL6YMYYKFBCWIHXTX4DUTY2OFREZNMDUFBRA, seg_pos={0 0}, upload=443079]: Long tail cancellation due to optimal threshold Dec 26 16:25:16.500994638 (504h56m16.372077683s) [noise=true , address=45.91.116.10:4084 , piece_id=HRQGIXXVFG7GHGK3C7BWKG7JIOZKXTJDOPVNF22VBURLMKUDMKDA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:16.501009487 (504h56m16.372092522s) [noise=true , address=45.91.116.10:4084 , piece_id=HRQGIXXVFG7GHGK3C7BWKG7JIOZKXTJDOPVNF22VBURLMKUDMKDA, seg_pos={0 0}, upload=443079]: Long tail cancellation due to optimal threshold Dec 26 16:25:16.501111121 (504h56m16.372194176s) [noise=true , address=145.79.227.35:4483 , piece_id=P3WYWEADNCLVRAYBUKQ7DLHDGLFPCFIMZYAA5MXDX42X5G7TFPMA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:16.501121580 (504h56m16.372204615s) [noise=true , address=145.79.227.35:4483 , piece_id=P3WYWEADNCLVRAYBUKQ7DLHDGLFPCFIMZYAA5MXDX42X5G7TFPMA, seg_pos={0 0}, upload=443079]: Long tail cancellation due to optimal threshold Dec 26 16:25:16.501165484 (504h56m16.372248519s) [seg_pos={0 0}, upload=443079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:25:16.501257439 (504h56m16.372340474s) [upload=443079]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:16.501264282 (504h56m16.372347327s) [upload=443079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:16.795278376 (504h56m16.666361421s) [upload=443079]: Done uploading segments... Dec 26 16:25:16.795300788 (504h56m16.666383854s) [upload=443079]: Upload finished. err: <nil> Duration: 5.880971396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.X....X....X..X...X.......XX..........X.........X..X.......] duration: 59.357650406s trace: https://jaeger.dev.storj.io/trace/3e3dfe0984f19d4e FAIL (2025-12-26 16:27:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.466998847 (504h51m15.338081892s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467028614 (504h51m15.338111659s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467359594 (504h51m15.338442639s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467908379 (504h51m15.338991424s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.468520054 (504h51m15.339603089s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469055293 (504h51m15.340138328s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469497605 (504h51m15.34058064s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469973642 (504h51m15.341056687s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.470454397 (504h51m15.341537443s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471043630 (504h51m15.342126695s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471470552 (504h51m15.342553587s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471676735 (504h51m15.34275977s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472062900 (504h51m15.343145935s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472369224 (504h51m15.343452259s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472823849 (504h51m15.343906884s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:16.743410939 (504h51m16.614493984s) [upload=443005]: Next returned nil segment Dec 26 16:20:16.743428862 (504h51m16.614511907s) [upload=443005]: Waiting for error group managing segments... Dec 26 16:20:17.091086984 (504h51m16.962170029s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.389229539 (504h51m17.260312594s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.464846438 (504h51m17.335929483s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.542075705 (504h51m17.41315875s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.623422163 (504h51m17.494505208s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.665409184 (504h51m17.536492219s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.301995626 (504h51m18.173078671s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.311693532 (504h51m18.182776567s) [noise=true , address=145.79.227.34:4915 , piece_id=7ZBWNPUF54BEGIXTDRDI3J7HNEZ63MEXKUBMRZW6PFWWNCEAH3AA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.340201235 (504h51m18.21128428s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.379076200 (504h51m18.250159245s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.474362322 (504h51m18.345445367s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.555133063 (504h51m18.426216108s) [noise=true , address=77.90.31.17:4791 , piece_id=QFHBBE4B5DWZU7HTAARXW655MKX4YPKPOXVM77CH5DCPSLZ7UGNA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.707673447 (504h51m18.578756492s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734654664 (504h51m18.605737709s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734678970 (504h51m18.605762005s) [seg_pos={0 0}, upload=443005]: Segment reached optimal threshold of 14 pieces. Dec 26 16:20:18.734942853 (504h51m18.606025888s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.734952842 (504h51m18.606035887s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735555389 (504h51m18.606638434s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection Dec 26 16:20:18.735566701 (504h51m18.606649746s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735681349 (504h51m18.606764394s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735699113 (504h51m18.606782148s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735826605 (504h51m18.60690968s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735837737 (504h51m18.606920782s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735908392 (504h51m18.606991447s) [seg_pos={0 0}, upload=443005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:20:18.736024182 (504h51m18.607107227s) [upload=443005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:20:18.736031275 (504h51m18.60711431s) [upload=443005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:20:19.031573339 (504h51m18.902656384s) [upload=443005]: Done uploading segments... Dec 26 16:20:19.031602895 (504h51m18.90268594s) [upload=443005]: Upload finished. err: <nil> Duration: 59.357673541s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [....XX...XX.........X.............XXX...X...XX.....XX.......] duration: 3.998802528s trace: https://jaeger.dev.storj.io/trace/7e43da40a4822a39 FAIL (2025-12-26 16:27:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.998887029s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 11.609530542s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:26:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 11.609554479s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XX...................X.....X..XX..XXX.X...XX.X.XX.X....XX..] duration: 6.10230896s trace: https://jaeger.dev.storj.io/trace/3f4ee18d6c20a87 FAIL (2025-12-26 16:26:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 16:25:12.724473263 (504h56m12.595556298s) [upload=443079]: Starting upload Dec 26 16:25:12.724540882 (504h56m12.595623917s) [upload=443079]: Done starting upload Dec 26 16:25:12.724757634 (504h56m12.595840669s) [upload=443079]: Uploading segments... Dec 26 16:25:12.724851774 (504h56m12.595934809s) [upload=443079]: Got next segment. Inline: false Dec 26 16:25:12.724881711 (504h56m12.595964756s) [upload=443079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:25:12.724885949 (504h56m12.595968994s) [upload=443079]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:25:12.967631409 (504h56m12.838714454s) [seg_pos={0 0}, upload=443079]: Begin upload segment... Dec 26 16:25:12.967672637 (504h56m12.838755682s) [seg_pos={0 0}, upload=443079]: Done begin upload segment. Dec 26 16:25:12.967717332 (504h56m12.838800367s) [noise=true , address=145.79.227.49:4338 , piece_id=7YHEDJHWYYKEMOJOT7WDSYWMVCCRGGCH5XD3GRWSAQAT3T6XKLYA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.967727943 (504h56m12.838810978s) [noise=true , address=145.79.227.13:4804 , piece_id=5CAQV5NZHKLXGXLJMKXR74FWW46SOYOPRVLK2PR5PMKRW7E5JGVQ, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.968427765 (504h56m12.83951081s) [noise=true , address=145.79.227.51:4409 , piece_id=FPENA4MRRTBEPG2I6IE52NXEQMAHJCNRI4FE67ZXEHJOILJMPFCQ, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.968554516 (504h56m12.839637561s) [noise=true , address=145.79.227.37:4566 , piece_id=P26XQX6NSU2NLXXOW4MJWNB23KK5ZO7EVI3UOTIJGUTTT32GODHA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.968891548 (504h56m12.839974583s) [noise=true , address=45.91.116.12:4227 , piece_id=BMMRT4Z57MH4V3AVKLL5VBRZXU54PCSUPVVEEIITNL6MAECIWYVA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.968939940 (504h56m12.840022985s) [noise=true , address=145.79.227.5:4222 , piece_id=ZORDZIWYYSJ72IWQOUG57K2XQJJWTQPXQ46PEFGJTBDXMCUG67OA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.969328460 (504h56m12.840411495s) [noise=true , address=145.79.227.46:4603 , piece_id=BGDTN5EXNU2EKMRRNDYFEHK75JITOPRTNW5FXDN66IGSSBFZIFJQ, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.969337808 (504h56m12.840420843s) [noise=true , address=145.79.227.16:4911 , piece_id=D3FD463XSS4WWDSLHVMZCQICZMAM7OXNTS5JNVDRXBPQOHDSDRWA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.969781403 (504h56m12.840864448s) [noise=true , address=145.79.227.32:4836 , piece_id=HUZJ3Z46K4IHBOUBLN7GHMYKD67QLJ5EWWNJGWXHO3B4WQO6KCLA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.969942158 (504h56m12.841025193s) [noise=true , address=77.90.31.13:4629 , piece_id=E5DLCMAOGNWZ3NQZNL6YMYYKFBCWIHXTX4DUTY2OFREZNMDUFBRA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.970172437 (504h56m12.841255472s) [noise=true , address=45.91.116.10:4084 , piece_id=HRQGIXXVFG7GHGK3C7BWKG7JIOZKXTJDOPVNF22VBURLMKUDMKDA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.970560686 (504h56m12.841643721s) [noise=true , address=45.91.116.6:4973 , piece_id=A243H6Z5EPE73AR24WOE3JL4HCTUCW6MRAZZQOKOOJ27DKVOZPGA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.970972701 (504h56m12.842055736s) [noise=true , address=45.91.116.8:4048 , piece_id=WX6WACST3J2JELW765CS4U3TCNREHRQTTUINYQGLW7WOM4RE72YA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.971342765 (504h56m12.84242581s) [noise=true , address=145.79.227.2:4114 , piece_id=7QGITYUBVGZHJGN6DVR3ZOSFIL4LMO5T6J3BNZEUNF366FHFMUUQ, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.971722348 (504h56m12.842805393s) [noise=true , address=145.79.227.11:4730 , piece_id=D35YMJIHH7ABNWVSAYXO2RC3SDE3GGWOAXWHQ2ERDEQXOINRHAGA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.972829446 (504h56m12.843912491s) [noise=true , address=145.79.227.35:4483 , piece_id=P3WYWEADNCLVRAYBUKQ7DLHDGLFPCFIMZYAA5MXDX42X5G7TFPMA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:14.675166383 (504h56m14.546249418s) [upload=443079]: Next returned nil segment Dec 26 16:25:14.675179578 (504h56m14.546262613s) [upload=443079]: Waiting for error group managing segments... Dec 26 16:25:14.965434624 (504h56m14.83651766s) [noise=true , address=145.79.227.32:4836 , piece_id=HUZJ3Z46K4IHBOUBLN7GHMYKD67QLJ5EWWNJGWXHO3B4WQO6KCLA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.965969002 (504h56m14.837052047s) [noise=true , address=145.79.227.49:4338 , piece_id=7YHEDJHWYYKEMOJOT7WDSYWMVCCRGGCH5XD3GRWSAQAT3T6XKLYA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.973444016 (504h56m14.844527051s) [noise=true , address=145.79.227.5:4222 , piece_id=ZORDZIWYYSJ72IWQOUG57K2XQJJWTQPXQ46PEFGJTBDXMCUG67OA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.974671563 (504h56m14.845754598s) [noise=true , address=145.79.227.11:4730 , piece_id=D35YMJIHH7ABNWVSAYXO2RC3SDE3GGWOAXWHQ2ERDEQXOINRHAGA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.986783416 (504h56m14.857866461s) [noise=true , address=145.79.227.46:4603 , piece_id=BGDTN5EXNU2EKMRRNDYFEHK75JITOPRTNW5FXDN66IGSSBFZIFJQ, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.988435422 (504h56m14.859518457s) [noise=true , address=145.79.227.51:4409 , piece_id=FPENA4MRRTBEPG2I6IE52NXEQMAHJCNRI4FE67ZXEHJOILJMPFCQ, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.992004167 (504h56m14.863087212s) [noise=true , address=145.79.227.37:4566 , piece_id=P26XQX6NSU2NLXXOW4MJWNB23KK5ZO7EVI3UOTIJGUTTT32GODHA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:15.004627965 (504h56m14.875711011s) [noise=true , address=145.79.227.13:4804 , piece_id=5CAQV5NZHKLXGXLJMKXR74FWW46SOYOPRVLK2PR5PMKRW7E5JGVQ, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:15.006485713 (504h56m14.877568748s) [noise=true , address=145.79.227.2:4114 , piece_id=7QGITYUBVGZHJGN6DVR3ZOSFIL4LMO5T6J3BNZEUNF366FHFMUUQ, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:15.225128197 (504h56m15.096211242s) [noise=true , address=45.91.116.12:4227 , piece_id=BMMRT4Z57MH4V3AVKLL5VBRZXU54PCSUPVVEEIITNL6MAECIWYVA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:15.236042749 (504h56m15.107125794s) [noise=true , address=45.91.116.8:4048 , piece_id=WX6WACST3J2JELW765CS4U3TCNREHRQTTUINYQGLW7WOM4RE72YA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:15.371498702 (504h56m15.242581737s) [noise=true , address=45.91.116.6:4973 , piece_id=A243H6Z5EPE73AR24WOE3JL4HCTUCW6MRAZZQOKOOJ27DKVOZPGA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:16.500662196 (504h56m16.371745231s) [noise=true , address=145.79.227.16:4911 , piece_id=D3FD463XSS4WWDSLHVMZCQICZMAM7OXNTS5JNVDRXBPQOHDSDRWA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:16.500699437 (504h56m16.371782472s) [seg_pos={0 0}, upload=443079]: Segment reached optimal threshold of 13 pieces. Dec 26 16:25:16.500923443 (504h56m16.372006488s) [noise=true , address=77.90.31.13:4629 , piece_id=E5DLCMAOGNWZ3NQZNL6YMYYKFBCWIHXTX4DUTY2OFREZNMDUFBRA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:16.500932931 (504h56m16.372015966s) [noise=true , address=77.90.31.13:4629 , piece_id=E5DLCMAOGNWZ3NQZNL6YMYYKFBCWIHXTX4DUTY2OFREZNMDUFBRA, seg_pos={0 0}, upload=443079]: Long tail cancellation due to optimal threshold Dec 26 16:25:16.500994638 (504h56m16.372077683s) [noise=true , address=45.91.116.10:4084 , piece_id=HRQGIXXVFG7GHGK3C7BWKG7JIOZKXTJDOPVNF22VBURLMKUDMKDA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:16.501009487 (504h56m16.372092522s) [noise=true , address=45.91.116.10:4084 , piece_id=HRQGIXXVFG7GHGK3C7BWKG7JIOZKXTJDOPVNF22VBURLMKUDMKDA, seg_pos={0 0}, upload=443079]: Long tail cancellation due to optimal threshold Dec 26 16:25:16.501111121 (504h56m16.372194176s) [noise=true , address=145.79.227.35:4483 , piece_id=P3WYWEADNCLVRAYBUKQ7DLHDGLFPCFIMZYAA5MXDX42X5G7TFPMA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:16.501121580 (504h56m16.372204615s) [noise=true , address=145.79.227.35:4483 , piece_id=P3WYWEADNCLVRAYBUKQ7DLHDGLFPCFIMZYAA5MXDX42X5G7TFPMA, seg_pos={0 0}, upload=443079]: Long tail cancellation due to optimal threshold Dec 26 16:25:16.501165484 (504h56m16.372248519s) [seg_pos={0 0}, upload=443079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:25:16.501257439 (504h56m16.372340474s) [upload=443079]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:16.501264282 (504h56m16.372347327s) [upload=443079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:16.795278376 (504h56m16.666361421s) [upload=443079]: Done uploading segments... Dec 26 16:25:16.795300788 (504h56m16.666383854s) [upload=443079]: Upload finished. err: <nil> Duration: 6.102327186s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX.X....X....X..X...X.......XX..........X.........X..X......] duration: 56.340614563s trace: https://jaeger.dev.storj.io/trace/3e3dfe0984f19d4e FAIL (2025-12-26 16:26:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.466998847 (504h51m15.338081892s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467028614 (504h51m15.338111659s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467359594 (504h51m15.338442639s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467908379 (504h51m15.338991424s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.468520054 (504h51m15.339603089s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469055293 (504h51m15.340138328s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469497605 (504h51m15.34058064s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469973642 (504h51m15.341056687s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.470454397 (504h51m15.341537443s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471043630 (504h51m15.342126695s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471470552 (504h51m15.342553587s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471676735 (504h51m15.34275977s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472062900 (504h51m15.343145935s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472369224 (504h51m15.343452259s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472823849 (504h51m15.343906884s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:16.743410939 (504h51m16.614493984s) [upload=443005]: Next returned nil segment Dec 26 16:20:16.743428862 (504h51m16.614511907s) [upload=443005]: Waiting for error group managing segments... Dec 26 16:20:17.091086984 (504h51m16.962170029s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.389229539 (504h51m17.260312594s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.464846438 (504h51m17.335929483s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.542075705 (504h51m17.41315875s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.623422163 (504h51m17.494505208s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.665409184 (504h51m17.536492219s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.301995626 (504h51m18.173078671s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.311693532 (504h51m18.182776567s) [noise=true , address=145.79.227.34:4915 , piece_id=7ZBWNPUF54BEGIXTDRDI3J7HNEZ63MEXKUBMRZW6PFWWNCEAH3AA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.340201235 (504h51m18.21128428s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.379076200 (504h51m18.250159245s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.474362322 (504h51m18.345445367s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.555133063 (504h51m18.426216108s) [noise=true , address=77.90.31.17:4791 , piece_id=QFHBBE4B5DWZU7HTAARXW655MKX4YPKPOXVM77CH5DCPSLZ7UGNA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.707673447 (504h51m18.578756492s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734654664 (504h51m18.605737709s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734678970 (504h51m18.605762005s) [seg_pos={0 0}, upload=443005]: Segment reached optimal threshold of 14 pieces. Dec 26 16:20:18.734942853 (504h51m18.606025888s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.734952842 (504h51m18.606035887s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735555389 (504h51m18.606638434s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection Dec 26 16:20:18.735566701 (504h51m18.606649746s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735681349 (504h51m18.606764394s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735699113 (504h51m18.606782148s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735826605 (504h51m18.60690968s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735837737 (504h51m18.606920782s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735908392 (504h51m18.606991447s) [seg_pos={0 0}, upload=443005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:20:18.736024182 (504h51m18.607107227s) [upload=443005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:20:18.736031275 (504h51m18.60711431s) [upload=443005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:20:19.031573339 (504h51m18.902656384s) [upload=443005]: Done uploading segments... Dec 26 16:20:19.031602895 (504h51m18.90268594s) [upload=443005]: Upload finished. err: <nil> Duration: 56.340633288s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 17.265202589s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:26:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 17.265221395s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..XX...................X.....X..XX..XXX.X...XX.X.XX.X....XX.] duration: 5.865725775s trace: https://jaeger.dev.storj.io/trace/3f4ee18d6c20a87 FAIL (2025-12-26 16:25:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 16:25:12.724473263 (504h56m12.595556298s) [upload=443079]: Starting upload Dec 26 16:25:12.724540882 (504h56m12.595623917s) [upload=443079]: Done starting upload Dec 26 16:25:12.724757634 (504h56m12.595840669s) [upload=443079]: Uploading segments... Dec 26 16:25:12.724851774 (504h56m12.595934809s) [upload=443079]: Got next segment. Inline: false Dec 26 16:25:12.724881711 (504h56m12.595964756s) [upload=443079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:25:12.724885949 (504h56m12.595968994s) [upload=443079]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:25:12.967631409 (504h56m12.838714454s) [seg_pos={0 0}, upload=443079]: Begin upload segment... Dec 26 16:25:12.967672637 (504h56m12.838755682s) [seg_pos={0 0}, upload=443079]: Done begin upload segment. Dec 26 16:25:12.967717332 (504h56m12.838800367s) [noise=true , address=145.79.227.49:4338 , piece_id=7YHEDJHWYYKEMOJOT7WDSYWMVCCRGGCH5XD3GRWSAQAT3T6XKLYA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.967727943 (504h56m12.838810978s) [noise=true , address=145.79.227.13:4804 , piece_id=5CAQV5NZHKLXGXLJMKXR74FWW46SOYOPRVLK2PR5PMKRW7E5JGVQ, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.968427765 (504h56m12.83951081s) [noise=true , address=145.79.227.51:4409 , piece_id=FPENA4MRRTBEPG2I6IE52NXEQMAHJCNRI4FE67ZXEHJOILJMPFCQ, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.968554516 (504h56m12.839637561s) [noise=true , address=145.79.227.37:4566 , piece_id=P26XQX6NSU2NLXXOW4MJWNB23KK5ZO7EVI3UOTIJGUTTT32GODHA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.968891548 (504h56m12.839974583s) [noise=true , address=45.91.116.12:4227 , piece_id=BMMRT4Z57MH4V3AVKLL5VBRZXU54PCSUPVVEEIITNL6MAECIWYVA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.968939940 (504h56m12.840022985s) [noise=true , address=145.79.227.5:4222 , piece_id=ZORDZIWYYSJ72IWQOUG57K2XQJJWTQPXQ46PEFGJTBDXMCUG67OA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.969328460 (504h56m12.840411495s) [noise=true , address=145.79.227.46:4603 , piece_id=BGDTN5EXNU2EKMRRNDYFEHK75JITOPRTNW5FXDN66IGSSBFZIFJQ, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.969337808 (504h56m12.840420843s) [noise=true , address=145.79.227.16:4911 , piece_id=D3FD463XSS4WWDSLHVMZCQICZMAM7OXNTS5JNVDRXBPQOHDSDRWA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.969781403 (504h56m12.840864448s) [noise=true , address=145.79.227.32:4836 , piece_id=HUZJ3Z46K4IHBOUBLN7GHMYKD67QLJ5EWWNJGWXHO3B4WQO6KCLA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.969942158 (504h56m12.841025193s) [noise=true , address=77.90.31.13:4629 , piece_id=E5DLCMAOGNWZ3NQZNL6YMYYKFBCWIHXTX4DUTY2OFREZNMDUFBRA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.970172437 (504h56m12.841255472s) [noise=true , address=45.91.116.10:4084 , piece_id=HRQGIXXVFG7GHGK3C7BWKG7JIOZKXTJDOPVNF22VBURLMKUDMKDA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.970560686 (504h56m12.841643721s) [noise=true , address=45.91.116.6:4973 , piece_id=A243H6Z5EPE73AR24WOE3JL4HCTUCW6MRAZZQOKOOJ27DKVOZPGA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.970972701 (504h56m12.842055736s) [noise=true , address=45.91.116.8:4048 , piece_id=WX6WACST3J2JELW765CS4U3TCNREHRQTTUINYQGLW7WOM4RE72YA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.971342765 (504h56m12.84242581s) [noise=true , address=145.79.227.2:4114 , piece_id=7QGITYUBVGZHJGN6DVR3ZOSFIL4LMO5T6J3BNZEUNF366FHFMUUQ, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.971722348 (504h56m12.842805393s) [noise=true , address=145.79.227.11:4730 , piece_id=D35YMJIHH7ABNWVSAYXO2RC3SDE3GGWOAXWHQ2ERDEQXOINRHAGA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.972829446 (504h56m12.843912491s) [noise=true , address=145.79.227.35:4483 , piece_id=P3WYWEADNCLVRAYBUKQ7DLHDGLFPCFIMZYAA5MXDX42X5G7TFPMA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:14.675166383 (504h56m14.546249418s) [upload=443079]: Next returned nil segment Dec 26 16:25:14.675179578 (504h56m14.546262613s) [upload=443079]: Waiting for error group managing segments... Dec 26 16:25:14.965434624 (504h56m14.83651766s) [noise=true , address=145.79.227.32:4836 , piece_id=HUZJ3Z46K4IHBOUBLN7GHMYKD67QLJ5EWWNJGWXHO3B4WQO6KCLA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.965969002 (504h56m14.837052047s) [noise=true , address=145.79.227.49:4338 , piece_id=7YHEDJHWYYKEMOJOT7WDSYWMVCCRGGCH5XD3GRWSAQAT3T6XKLYA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.973444016 (504h56m14.844527051s) [noise=true , address=145.79.227.5:4222 , piece_id=ZORDZIWYYSJ72IWQOUG57K2XQJJWTQPXQ46PEFGJTBDXMCUG67OA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.974671563 (504h56m14.845754598s) [noise=true , address=145.79.227.11:4730 , piece_id=D35YMJIHH7ABNWVSAYXO2RC3SDE3GGWOAXWHQ2ERDEQXOINRHAGA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.986783416 (504h56m14.857866461s) [noise=true , address=145.79.227.46:4603 , piece_id=BGDTN5EXNU2EKMRRNDYFEHK75JITOPRTNW5FXDN66IGSSBFZIFJQ, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.988435422 (504h56m14.859518457s) [noise=true , address=145.79.227.51:4409 , piece_id=FPENA4MRRTBEPG2I6IE52NXEQMAHJCNRI4FE67ZXEHJOILJMPFCQ, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.992004167 (504h56m14.863087212s) [noise=true , address=145.79.227.37:4566 , piece_id=P26XQX6NSU2NLXXOW4MJWNB23KK5ZO7EVI3UOTIJGUTTT32GODHA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:15.004627965 (504h56m14.875711011s) [noise=true , address=145.79.227.13:4804 , piece_id=5CAQV5NZHKLXGXLJMKXR74FWW46SOYOPRVLK2PR5PMKRW7E5JGVQ, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:15.006485713 (504h56m14.877568748s) [noise=true , address=145.79.227.2:4114 , piece_id=7QGITYUBVGZHJGN6DVR3ZOSFIL4LMO5T6J3BNZEUNF366FHFMUUQ, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:15.225128197 (504h56m15.096211242s) [noise=true , address=45.91.116.12:4227 , piece_id=BMMRT4Z57MH4V3AVKLL5VBRZXU54PCSUPVVEEIITNL6MAECIWYVA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:15.236042749 (504h56m15.107125794s) [noise=true , address=45.91.116.8:4048 , piece_id=WX6WACST3J2JELW765CS4U3TCNREHRQTTUINYQGLW7WOM4RE72YA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:15.371498702 (504h56m15.242581737s) [noise=true , address=45.91.116.6:4973 , piece_id=A243H6Z5EPE73AR24WOE3JL4HCTUCW6MRAZZQOKOOJ27DKVOZPGA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:16.500662196 (504h56m16.371745231s) [noise=true , address=145.79.227.16:4911 , piece_id=D3FD463XSS4WWDSLHVMZCQICZMAM7OXNTS5JNVDRXBPQOHDSDRWA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:16.500699437 (504h56m16.371782472s) [seg_pos={0 0}, upload=443079]: Segment reached optimal threshold of 13 pieces. Dec 26 16:25:16.500923443 (504h56m16.372006488s) [noise=true , address=77.90.31.13:4629 , piece_id=E5DLCMAOGNWZ3NQZNL6YMYYKFBCWIHXTX4DUTY2OFREZNMDUFBRA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:16.500932931 (504h56m16.372015966s) [noise=true , address=77.90.31.13:4629 , piece_id=E5DLCMAOGNWZ3NQZNL6YMYYKFBCWIHXTX4DUTY2OFREZNMDUFBRA, seg_pos={0 0}, upload=443079]: Long tail cancellation due to optimal threshold Dec 26 16:25:16.500994638 (504h56m16.372077683s) [noise=true , address=45.91.116.10:4084 , piece_id=HRQGIXXVFG7GHGK3C7BWKG7JIOZKXTJDOPVNF22VBURLMKUDMKDA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:16.501009487 (504h56m16.372092522s) [noise=true , address=45.91.116.10:4084 , piece_id=HRQGIXXVFG7GHGK3C7BWKG7JIOZKXTJDOPVNF22VBURLMKUDMKDA, seg_pos={0 0}, upload=443079]: Long tail cancellation due to optimal threshold Dec 26 16:25:16.501111121 (504h56m16.372194176s) [noise=true , address=145.79.227.35:4483 , piece_id=P3WYWEADNCLVRAYBUKQ7DLHDGLFPCFIMZYAA5MXDX42X5G7TFPMA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:16.501121580 (504h56m16.372204615s) [noise=true , address=145.79.227.35:4483 , piece_id=P3WYWEADNCLVRAYBUKQ7DLHDGLFPCFIMZYAA5MXDX42X5G7TFPMA, seg_pos={0 0}, upload=443079]: Long tail cancellation due to optimal threshold Dec 26 16:25:16.501165484 (504h56m16.372248519s) [seg_pos={0 0}, upload=443079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:25:16.501257439 (504h56m16.372340474s) [upload=443079]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:16.501264282 (504h56m16.372347327s) [upload=443079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:16.795278376 (504h56m16.666361421s) [upload=443079]: Done uploading segments... Dec 26 16:25:16.795300788 (504h56m16.666383854s) [upload=443079]: Upload finished. err: <nil> Duration: 5.865753969s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [X....XX...XX.........X.............XXX...X...XX.....XX......] duration: 3.374661865s trace: https://jaeger.dev.storj.io/trace/7e43da40a4822a39 FAIL (2025-12-26 16:25:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.374698235s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 35.157875823s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:25:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 35.157928052s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...XX...................X.....X..XX..XXX.X...XX.X.XX.X....XX] duration: 30.003286404s trace: https://jaeger.dev.storj.io/trace/3f4ee18d6c20a87 FAIL (2025-12-26 16:25:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 16:25:12.724473263 (504h56m12.595556298s) [upload=443079]: Starting upload Dec 26 16:25:12.724540882 (504h56m12.595623917s) [upload=443079]: Done starting upload Dec 26 16:25:12.724757634 (504h56m12.595840669s) [upload=443079]: Uploading segments... Dec 26 16:25:12.724851774 (504h56m12.595934809s) [upload=443079]: Got next segment. Inline: false Dec 26 16:25:12.724881711 (504h56m12.595964756s) [upload=443079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:25:12.724885949 (504h56m12.595968994s) [upload=443079]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:25:12.967631409 (504h56m12.838714454s) [seg_pos={0 0}, upload=443079]: Begin upload segment... Dec 26 16:25:12.967672637 (504h56m12.838755682s) [seg_pos={0 0}, upload=443079]: Done begin upload segment. Dec 26 16:25:12.967717332 (504h56m12.838800367s) [noise=true , address=145.79.227.49:4338 , piece_id=7YHEDJHWYYKEMOJOT7WDSYWMVCCRGGCH5XD3GRWSAQAT3T6XKLYA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.967727943 (504h56m12.838810978s) [noise=true , address=145.79.227.13:4804 , piece_id=5CAQV5NZHKLXGXLJMKXR74FWW46SOYOPRVLK2PR5PMKRW7E5JGVQ, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.968427765 (504h56m12.83951081s) [noise=true , address=145.79.227.51:4409 , piece_id=FPENA4MRRTBEPG2I6IE52NXEQMAHJCNRI4FE67ZXEHJOILJMPFCQ, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.968554516 (504h56m12.839637561s) [noise=true , address=145.79.227.37:4566 , piece_id=P26XQX6NSU2NLXXOW4MJWNB23KK5ZO7EVI3UOTIJGUTTT32GODHA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.968891548 (504h56m12.839974583s) [noise=true , address=45.91.116.12:4227 , piece_id=BMMRT4Z57MH4V3AVKLL5VBRZXU54PCSUPVVEEIITNL6MAECIWYVA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.968939940 (504h56m12.840022985s) [noise=true , address=145.79.227.5:4222 , piece_id=ZORDZIWYYSJ72IWQOUG57K2XQJJWTQPXQ46PEFGJTBDXMCUG67OA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.969328460 (504h56m12.840411495s) [noise=true , address=145.79.227.46:4603 , piece_id=BGDTN5EXNU2EKMRRNDYFEHK75JITOPRTNW5FXDN66IGSSBFZIFJQ, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.969337808 (504h56m12.840420843s) [noise=true , address=145.79.227.16:4911 , piece_id=D3FD463XSS4WWDSLHVMZCQICZMAM7OXNTS5JNVDRXBPQOHDSDRWA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.969781403 (504h56m12.840864448s) [noise=true , address=145.79.227.32:4836 , piece_id=HUZJ3Z46K4IHBOUBLN7GHMYKD67QLJ5EWWNJGWXHO3B4WQO6KCLA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.969942158 (504h56m12.841025193s) [noise=true , address=77.90.31.13:4629 , piece_id=E5DLCMAOGNWZ3NQZNL6YMYYKFBCWIHXTX4DUTY2OFREZNMDUFBRA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.970172437 (504h56m12.841255472s) [noise=true , address=45.91.116.10:4084 , piece_id=HRQGIXXVFG7GHGK3C7BWKG7JIOZKXTJDOPVNF22VBURLMKUDMKDA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.970560686 (504h56m12.841643721s) [noise=true , address=45.91.116.6:4973 , piece_id=A243H6Z5EPE73AR24WOE3JL4HCTUCW6MRAZZQOKOOJ27DKVOZPGA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.970972701 (504h56m12.842055736s) [noise=true , address=45.91.116.8:4048 , piece_id=WX6WACST3J2JELW765CS4U3TCNREHRQTTUINYQGLW7WOM4RE72YA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.971342765 (504h56m12.84242581s) [noise=true , address=145.79.227.2:4114 , piece_id=7QGITYUBVGZHJGN6DVR3ZOSFIL4LMO5T6J3BNZEUNF366FHFMUUQ, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.971722348 (504h56m12.842805393s) [noise=true , address=145.79.227.11:4730 , piece_id=D35YMJIHH7ABNWVSAYXO2RC3SDE3GGWOAXWHQ2ERDEQXOINRHAGA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:12.972829446 (504h56m12.843912491s) [noise=true , address=145.79.227.35:4483 , piece_id=P3WYWEADNCLVRAYBUKQ7DLHDGLFPCFIMZYAA5MXDX42X5G7TFPMA, seg_pos={0 0}, upload=443079]: Uploading piece... Dec 26 16:25:14.675166383 (504h56m14.546249418s) [upload=443079]: Next returned nil segment Dec 26 16:25:14.675179578 (504h56m14.546262613s) [upload=443079]: Waiting for error group managing segments... Dec 26 16:25:14.965434624 (504h56m14.83651766s) [noise=true , address=145.79.227.32:4836 , piece_id=HUZJ3Z46K4IHBOUBLN7GHMYKD67QLJ5EWWNJGWXHO3B4WQO6KCLA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.965969002 (504h56m14.837052047s) [noise=true , address=145.79.227.49:4338 , piece_id=7YHEDJHWYYKEMOJOT7WDSYWMVCCRGGCH5XD3GRWSAQAT3T6XKLYA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.973444016 (504h56m14.844527051s) [noise=true , address=145.79.227.5:4222 , piece_id=ZORDZIWYYSJ72IWQOUG57K2XQJJWTQPXQ46PEFGJTBDXMCUG67OA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.974671563 (504h56m14.845754598s) [noise=true , address=145.79.227.11:4730 , piece_id=D35YMJIHH7ABNWVSAYXO2RC3SDE3GGWOAXWHQ2ERDEQXOINRHAGA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.986783416 (504h56m14.857866461s) [noise=true , address=145.79.227.46:4603 , piece_id=BGDTN5EXNU2EKMRRNDYFEHK75JITOPRTNW5FXDN66IGSSBFZIFJQ, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.988435422 (504h56m14.859518457s) [noise=true , address=145.79.227.51:4409 , piece_id=FPENA4MRRTBEPG2I6IE52NXEQMAHJCNRI4FE67ZXEHJOILJMPFCQ, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:14.992004167 (504h56m14.863087212s) [noise=true , address=145.79.227.37:4566 , piece_id=P26XQX6NSU2NLXXOW4MJWNB23KK5ZO7EVI3UOTIJGUTTT32GODHA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:15.004627965 (504h56m14.875711011s) [noise=true , address=145.79.227.13:4804 , piece_id=5CAQV5NZHKLXGXLJMKXR74FWW46SOYOPRVLK2PR5PMKRW7E5JGVQ, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:15.006485713 (504h56m14.877568748s) [noise=true , address=145.79.227.2:4114 , piece_id=7QGITYUBVGZHJGN6DVR3ZOSFIL4LMO5T6J3BNZEUNF366FHFMUUQ, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:15.225128197 (504h56m15.096211242s) [noise=true , address=45.91.116.12:4227 , piece_id=BMMRT4Z57MH4V3AVKLL5VBRZXU54PCSUPVVEEIITNL6MAECIWYVA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:15.236042749 (504h56m15.107125794s) [noise=true , address=45.91.116.8:4048 , piece_id=WX6WACST3J2JELW765CS4U3TCNREHRQTTUINYQGLW7WOM4RE72YA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:15.371498702 (504h56m15.242581737s) [noise=true , address=45.91.116.6:4973 , piece_id=A243H6Z5EPE73AR24WOE3JL4HCTUCW6MRAZZQOKOOJ27DKVOZPGA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:16.500662196 (504h56m16.371745231s) [noise=true , address=145.79.227.16:4911 , piece_id=D3FD463XSS4WWDSLHVMZCQICZMAM7OXNTS5JNVDRXBPQOHDSDRWA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: <nil> Dec 26 16:25:16.500699437 (504h56m16.371782472s) [seg_pos={0 0}, upload=443079]: Segment reached optimal threshold of 13 pieces. Dec 26 16:25:16.500923443 (504h56m16.372006488s) [noise=true , address=77.90.31.13:4629 , piece_id=E5DLCMAOGNWZ3NQZNL6YMYYKFBCWIHXTX4DUTY2OFREZNMDUFBRA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:16.500932931 (504h56m16.372015966s) [noise=true , address=77.90.31.13:4629 , piece_id=E5DLCMAOGNWZ3NQZNL6YMYYKFBCWIHXTX4DUTY2OFREZNMDUFBRA, seg_pos={0 0}, upload=443079]: Long tail cancellation due to optimal threshold Dec 26 16:25:16.500994638 (504h56m16.372077683s) [noise=true , address=45.91.116.10:4084 , piece_id=HRQGIXXVFG7GHGK3C7BWKG7JIOZKXTJDOPVNF22VBURLMKUDMKDA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:16.501009487 (504h56m16.372092522s) [noise=true , address=45.91.116.10:4084 , piece_id=HRQGIXXVFG7GHGK3C7BWKG7JIOZKXTJDOPVNF22VBURLMKUDMKDA, seg_pos={0 0}, upload=443079]: Long tail cancellation due to optimal threshold Dec 26 16:25:16.501111121 (504h56m16.372194176s) [noise=true , address=145.79.227.35:4483 , piece_id=P3WYWEADNCLVRAYBUKQ7DLHDGLFPCFIMZYAA5MXDX42X5G7TFPMA, seg_pos={0 0}, upload=443079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:16.501121580 (504h56m16.372204615s) [noise=true , address=145.79.227.35:4483 , piece_id=P3WYWEADNCLVRAYBUKQ7DLHDGLFPCFIMZYAA5MXDX42X5G7TFPMA, seg_pos={0 0}, upload=443079]: Long tail cancellation due to optimal threshold Dec 26 16:25:16.501165484 (504h56m16.372248519s) [seg_pos={0 0}, upload=443079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:25:16.501257439 (504h56m16.372340474s) [upload=443079]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:16.501264282 (504h56m16.372347327s) [upload=443079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:16.795278376 (504h56m16.666361421s) [upload=443079]: Done uploading segments... Dec 26 16:25:16.795300788 (504h56m16.666383854s) [upload=443079]: Upload finished. err: <nil> Duration: 30.00329896s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.X....XX...XX.........X.............XXX...X...XX.....XX.....] duration: 3.391319113s trace: https://jaeger.dev.storj.io/trace/7e43da40a4822a39 FAIL (2025-12-26 16:25:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.39134879s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....XX...................X.....X..XX..XXX.X...XX.X.XX.X....X] duration: 30.003447342s trace: https://jaeger.dev.storj.io/trace/d3174d9252538e FAIL (2025-12-26 16:25:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: JZOBUZELGBAA76Z4Q6XBMQQLLFWOUCYDPQK6P7GDUW6A5WW2OK2A) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: KB4RN7ZCTULKWOGK5EQROS7754SHY4PRK5SSNXNLKD5CYYCTQPPQ) context deadline exceeded ======LOGS====== Dec 26 16:24:42.492313894 (504h55m42.363396929s) [upload=443072]: Starting upload Dec 26 16:24:42.492366995 (504h55m42.36345004s) [upload=443072]: Done starting upload Dec 26 16:24:42.492590590 (504h55m42.363673636s) [upload=443072]: Uploading segments... Dec 26 16:24:42.492741128 (504h55m42.363824173s) [upload=443072]: Got next segment. Inline: false Dec 26 16:24:42.492791774 (504h55m42.363874819s) [upload=443072]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:24:42.492798927 (504h55m42.363881982s) [upload=443072]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:24:42.740843067 (504h55m42.611926112s) [seg_pos={0 0}, upload=443072]: Begin upload segment... Dec 26 16:24:42.740924713 (504h55m42.612007748s) [seg_pos={0 0}, upload=443072]: Done begin upload segment. Dec 26 16:24:42.740994786 (504h55m42.612077821s) [noise=true , address=145.79.227.49:4339 , piece_id=QFFU6NEBZUCZRL4CL7SEAXUUXZTHNYTJBJIMJHLA7XNQHK7237SQ, seg_pos={0 0}, upload=443072]: Uploading piece... Dec 26 16:24:42.741009333 (504h55m42.612092368s) [noise=true , address=145.79.227.4:4187 , piece_id=R3XP75QIKM7D23E2I7MMQTMO7ODVRY4BND5GWJLT26VQKV36PTKA, seg_pos={0 0}, upload=443072]: Uploading piece... Dec 26 16:24:42.741568578 (504h55m42.612651623s) [noise=true , address=77.90.31.11:4556 , piece_id=KB4RN7ZCTULKWOGK5EQROS7754SHY4PRK5SSNXNLKD5CYYCTQPPQ, seg_pos={0 0}, upload=443072]: Uploading piece... Dec 26 16:24:42.741743301 (504h55m42.612826336s) [noise=true , address=145.79.227.12:4767 , piece_id=Y3ZMB6FQGMKGWLJ3OPJWA2FJ25ECDHSW63O4ABM5PJPBKDKNLUMA, seg_pos={0 0}, upload=443072]: Uploading piece... Dec 26 16:24:42.742081675 (504h55m42.613164721s) [noise=true , address=145.79.227.52:4445 , piece_id=DORMWBQYDCQX5XPV3WTF4DKMD5X6GJR7DHIR7CSJ2EAAW5YY6CEQ, seg_pos={0 0}, upload=443072]: Uploading piece... Dec 26 16:24:42.742287938 (504h55m42.613370973s) [noise=true , address=145.79.227.36:4519 , piece_id=C2UFT4HJMYAFJTR4WHCPTO5ZVRFXGA7KVS7HQW3RH27N3ZX4OV5Q, seg_pos={0 0}, upload=443072]: Uploading piece... Dec 26 16:24:42.742340989 (504h55m42.613424034s) [noise=true , address=145.79.227.50:4375 , piece_id=43W5CJEM76FFAEJCNCOMLE6VET6HG6COCXJYRG2EGNKYXJ3JXQDA, seg_pos={0 0}, upload=443072]: Uploading piece... Dec 26 16:24:42.742674765 (504h55m42.61375781s) [noise=true , address=145.79.227.47:4267 , piece_id=XTATDL7R54QZCUBBNDLARQ4GLBX7IQMXWAVQXKKTJ7YTBMUYOBPQ, seg_pos={0 0}, upload=443072]: Uploading piece... Dec 26 16:24:42.742980227 (504h55m42.614063262s) [noise=true , address=145.79.227.38:4979 , piece_id=LJVOZZB4JGECEZ3U6N5GHBNF24L3GIGNKTYFW3PSAVOB5EQR255A, seg_pos={0 0}, upload=443072]: Uploading piece... Dec 26 16:24:42.743013099 (504h55m42.614096144s) [noise=true , address=45.91.116.7:4010 , piece_id=FIDJAHH5PTGWL3G62YQDRZYUG4YGLSYR6OTB2R4STJ6JWSLVQQDA, seg_pos={0 0}, upload=443072]: Uploading piece... Dec 26 16:24:42.743280949 (504h55m42.614363984s) [noise=true , address=45.91.116.10:4081 , piece_id=TYVIZSYCVX7UUDPEXG4TOBLAGHAUG4BKNDBTCAAUUWOS4MQAWRZA, seg_pos={0 0}, upload=443072]: Uploading piece... Dec 26 16:24:42.743479557 (504h55m42.614562592s) [noise=true , address=45.91.116.14:4192 , piece_id=JZOBUZELGBAA76Z4Q6XBMQQLLFWOUCYDPQK6P7GDUW6A5WW2OK2A, seg_pos={0 0}, upload=443072]: Uploading piece... Dec 26 16:24:42.743590017 (504h55m42.614673052s) [noise=true , address=145.79.227.22:4511 , piece_id=X5PCSC77BBW72LR3NOUIZKAQBSASMVWAA2VT7NE3YU6ZO67MA3LQ, seg_pos={0 0}, upload=443072]: Uploading piece... Dec 26 16:24:42.744012261 (504h55m42.615095306s) [noise=true , address=77.90.31.18:4808 , piece_id=R4KJIFJ44U4JI4JACJ6U7GRUAB42VT3IONIWB2NYDQFQ3MVOYSAA, seg_pos={0 0}, upload=443072]: Uploading piece... Dec 26 16:24:42.744478840 (504h55m42.615561885s) [noise=true , address=45.91.116.11:4120 , piece_id=JGI3MLMHF5EVN4LHDUVUTLNUMPYMXFL2UZNWFPYTJMB3M42LH7TQ, seg_pos={0 0}, upload=443072]: Uploading piece... Dec 26 16:24:42.744991426 (504h55m42.616074471s) [noise=true , address=145.79.227.40:4051 , piece_id=6GCMXCK45TIFE76MKNE2LO5GTSPOY5A6SM56CYTA76RYB2I3B5AQ, seg_pos={0 0}, upload=443072]: Uploading piece... Dec 26 16:24:44.460243755 (504h55m44.33132679s) [upload=443072]: Next returned nil segment Dec 26 16:24:44.460284463 (504h55m44.331367498s) [upload=443072]: Waiting for error group managing segments... Dec 26 16:24:44.739789210 (504h55m44.610872245s) [noise=true , address=145.79.227.49:4339 , piece_id=QFFU6NEBZUCZRL4CL7SEAXUUXZTHNYTJBJIMJHLA7XNQHK7237SQ, seg_pos={0 0}, upload=443072]: Done uploading piece. err: <nil> Dec 26 16:24:44.741931899 (504h55m44.613014944s) [noise=true , address=145.79.227.12:4767 , piece_id=Y3ZMB6FQGMKGWLJ3OPJWA2FJ25ECDHSW63O4ABM5PJPBKDKNLUMA, seg_pos={0 0}, upload=443072]: Done uploading piece. err: <nil> Dec 26 16:24:44.746197422 (504h55m44.617280477s) [noise=true , address=145.79.227.52:4445 , piece_id=DORMWBQYDCQX5XPV3WTF4DKMD5X6GJR7DHIR7CSJ2EAAW5YY6CEQ, seg_pos={0 0}, upload=443072]: Done uploading piece. err: <nil> Dec 26 16:24:44.747952704 (504h55m44.619035769s) [noise=true , address=145.79.227.50:4375 , piece_id=43W5CJEM76FFAEJCNCOMLE6VET6HG6COCXJYRG2EGNKYXJ3JXQDA, seg_pos={0 0}, upload=443072]: Done uploading piece. err: <nil> Dec 26 16:24:44.752620622 (504h55m44.623703667s) [noise=true , address=145.79.227.47:4267 , piece_id=XTATDL7R54QZCUBBNDLARQ4GLBX7IQMXWAVQXKKTJ7YTBMUYOBPQ, seg_pos={0 0}, upload=443072]: Done uploading piece. err: <nil> Dec 26 16:24:44.766545828 (504h55m44.637628863s) [noise=true , address=145.79.227.38:4979 , piece_id=LJVOZZB4JGECEZ3U6N5GHBNF24L3GIGNKTYFW3PSAVOB5EQR255A, seg_pos={0 0}, upload=443072]: Done uploading piece. err: <nil> Dec 26 16:24:44.767738530 (504h55m44.638821565s) [noise=true , address=145.79.227.36:4519 , piece_id=C2UFT4HJMYAFJTR4WHCPTO5ZVRFXGA7KVS7HQW3RH27N3ZX4OV5Q, seg_pos={0 0}, upload=443072]: Done uploading piece. err: <nil> Dec 26 16:24:44.786240010 (504h55m44.657323055s) [noise=true , address=145.79.227.4:4187 , piece_id=R3XP75QIKM7D23E2I7MMQTMO7ODVRY4BND5GWJLT26VQKV36PTKA, seg_pos={0 0}, upload=443072]: Done uploading piece. err: <nil> Dec 26 16:24:44.786769848 (504h55m44.657852884s) [noise=true , address=145.79.227.22:4511 , piece_id=X5PCSC77BBW72LR3NOUIZKAQBSASMVWAA2VT7NE3YU6ZO67MA3LQ, seg_pos={0 0}, upload=443072]: Done uploading piece. err: <nil> Dec 26 16:24:44.789383395 (504h55m44.66046643s) [noise=true , address=145.79.227.40:4051 , piece_id=6GCMXCK45TIFE76MKNE2LO5GTSPOY5A6SM56CYTA76RYB2I3B5AQ, seg_pos={0 0}, upload=443072]: Done uploading piece. err: <nil> Dec 26 16:24:44.963257993 (504h55m44.834341028s) [noise=true , address=77.90.31.11:4556 , piece_id=KB4RN7ZCTULKWOGK5EQROS7754SHY4PRK5SSNXNLKD5CYYCTQPPQ, seg_pos={0 0}, upload=443072]: Done uploading piece. err: <nil> Dec 26 16:24:45.845611758 (504h55m45.716694803s) [noise=true , address=45.91.116.7:4010 , piece_id=FIDJAHH5PTGWL3G62YQDRZYUG4YGLSYR6OTB2R4STJ6JWSLVQQDA, seg_pos={0 0}, upload=443072]: Done uploading piece. err: <nil> Dec 26 16:24:45.855935707 (504h55m45.727018752s) [noise=true , address=45.91.116.14:4192 , piece_id=JZOBUZELGBAA76Z4Q6XBMQQLLFWOUCYDPQK6P7GDUW6A5WW2OK2A, seg_pos={0 0}, upload=443072]: Done uploading piece. err: <nil> Dec 26 16:24:45.855959703 (504h55m45.727042748s) [seg_pos={0 0}, upload=443072]: Segment reached optimal threshold of 13 pieces. Dec 26 16:24:45.856184060 (504h55m45.727267095s) [noise=true , address=77.90.31.18:4808 , piece_id=R4KJIFJ44U4JI4JACJ6U7GRUAB42VT3IONIWB2NYDQFQ3MVOYSAA, seg_pos={0 0}, upload=443072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 26 16:24:45.856192486 (504h55m45.727275521s) [noise=true , address=77.90.31.18:4808 , piece_id=R4KJIFJ44U4JI4JACJ6U7GRUAB42VT3IONIWB2NYDQFQ3MVOYSAA, seg_pos={0 0}, upload=443072]: Long tail cancellation due to optimal threshold Dec 26 16:24:45.856219047 (504h55m45.727302082s) [noise=true , address=45.91.116.10:4081 , piece_id=TYVIZSYCVX7UUDPEXG4TOBLAGHAUG4BKNDBTCAAUUWOS4MQAWRZA, seg_pos={0 0}, upload=443072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 26 16:24:45.856229868 (504h55m45.727312913s) [noise=true , address=45.91.116.10:4081 , piece_id=TYVIZSYCVX7UUDPEXG4TOBLAGHAUG4BKNDBTCAAUUWOS4MQAWRZA, seg_pos={0 0}, upload=443072]: Long tail cancellation due to optimal threshold Dec 26 16:24:45.856287948 (504h55m45.727370983s) [noise=true , address=45.91.116.11:4120 , piece_id=JGI3MLMHF5EVN4LHDUVUTLNUMPYMXFL2UZNWFPYTJMB3M42LH7TQ, seg_pos={0 0}, upload=443072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 16:24:45.856294320 (504h55m45.727377365s) [noise=true , address=45.91.116.11:4120 , piece_id=JGI3MLMHF5EVN4LHDUVUTLNUMPYMXFL2UZNWFPYTJMB3M42LH7TQ, seg_pos={0 0}, upload=443072]: Long tail cancellation due to optimal threshold Dec 26 16:24:45.856311483 (504h55m45.727394528s) [seg_pos={0 0}, upload=443072]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:24:45.856392888 (504h55m45.727475923s) [upload=443072]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:24:45.856399911 (504h55m45.727482946s) [upload=443072]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:24:46.153475094 (504h55m46.024558139s) [upload=443072]: Done uploading segments... Dec 26 16:24:46.153498098 (504h55m46.024581143s) [upload=443072]: Upload finished. err: <nil> Duration: 30.003459134s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XX.X....X....X..X...X.......XX..........X.........X..X.....] duration: 40.998103917s trace: https://jaeger.dev.storj.io/trace/3e3dfe0984f19d4e FAIL (2025-12-26 16:25:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.466998847 (504h51m15.338081892s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467028614 (504h51m15.338111659s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467359594 (504h51m15.338442639s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467908379 (504h51m15.338991424s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.468520054 (504h51m15.339603089s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469055293 (504h51m15.340138328s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469497605 (504h51m15.34058064s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469973642 (504h51m15.341056687s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.470454397 (504h51m15.341537443s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471043630 (504h51m15.342126695s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471470552 (504h51m15.342553587s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471676735 (504h51m15.34275977s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472062900 (504h51m15.343145935s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472369224 (504h51m15.343452259s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472823849 (504h51m15.343906884s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:16.743410939 (504h51m16.614493984s) [upload=443005]: Next returned nil segment Dec 26 16:20:16.743428862 (504h51m16.614511907s) [upload=443005]: Waiting for error group managing segments... Dec 26 16:20:17.091086984 (504h51m16.962170029s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.389229539 (504h51m17.260312594s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.464846438 (504h51m17.335929483s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.542075705 (504h51m17.41315875s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.623422163 (504h51m17.494505208s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.665409184 (504h51m17.536492219s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.301995626 (504h51m18.173078671s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.311693532 (504h51m18.182776567s) [noise=true , address=145.79.227.34:4915 , piece_id=7ZBWNPUF54BEGIXTDRDI3J7HNEZ63MEXKUBMRZW6PFWWNCEAH3AA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.340201235 (504h51m18.21128428s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.379076200 (504h51m18.250159245s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.474362322 (504h51m18.345445367s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.555133063 (504h51m18.426216108s) [noise=true , address=77.90.31.17:4791 , piece_id=QFHBBE4B5DWZU7HTAARXW655MKX4YPKPOXVM77CH5DCPSLZ7UGNA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.707673447 (504h51m18.578756492s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734654664 (504h51m18.605737709s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734678970 (504h51m18.605762005s) [seg_pos={0 0}, upload=443005]: Segment reached optimal threshold of 14 pieces. Dec 26 16:20:18.734942853 (504h51m18.606025888s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.734952842 (504h51m18.606035887s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735555389 (504h51m18.606638434s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection Dec 26 16:20:18.735566701 (504h51m18.606649746s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735681349 (504h51m18.606764394s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735699113 (504h51m18.606782148s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735826605 (504h51m18.60690968s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735837737 (504h51m18.606920782s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735908392 (504h51m18.606991447s) [seg_pos={0 0}, upload=443005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:20:18.736024182 (504h51m18.607107227s) [upload=443005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:20:18.736031275 (504h51m18.60711431s) [upload=443005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:20:19.031573339 (504h51m18.902656384s) [upload=443005]: Done uploading segments... Dec 26 16:20:19.031602895 (504h51m18.90268594s) [upload=443005]: Upload finished. err: <nil> Duration: 40.99813235s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [..X....XX...XX.........X.............XXX...X...XX.....XX....] duration: 5.001385261s trace: https://jaeger.dev.storj.io/trace/7e43da40a4822a39 FAIL (2025-12-26 16:25:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001412804s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.XX.X....X....X..X...X.......XX..........X.........X..X....] duration: 38.020875955s trace: https://jaeger.dev.storj.io/trace/3e3dfe0984f19d4e FAIL (2025-12-26 16:24:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.466998847 (504h51m15.338081892s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467028614 (504h51m15.338111659s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467359594 (504h51m15.338442639s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467908379 (504h51m15.338991424s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.468520054 (504h51m15.339603089s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469055293 (504h51m15.340138328s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469497605 (504h51m15.34058064s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469973642 (504h51m15.341056687s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.470454397 (504h51m15.341537443s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471043630 (504h51m15.342126695s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471470552 (504h51m15.342553587s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471676735 (504h51m15.34275977s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472062900 (504h51m15.343145935s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472369224 (504h51m15.343452259s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472823849 (504h51m15.343906884s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:16.743410939 (504h51m16.614493984s) [upload=443005]: Next returned nil segment Dec 26 16:20:16.743428862 (504h51m16.614511907s) [upload=443005]: Waiting for error group managing segments... Dec 26 16:20:17.091086984 (504h51m16.962170029s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.389229539 (504h51m17.260312594s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.464846438 (504h51m17.335929483s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.542075705 (504h51m17.41315875s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.623422163 (504h51m17.494505208s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.665409184 (504h51m17.536492219s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.301995626 (504h51m18.173078671s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.311693532 (504h51m18.182776567s) [noise=true , address=145.79.227.34:4915 , piece_id=7ZBWNPUF54BEGIXTDRDI3J7HNEZ63MEXKUBMRZW6PFWWNCEAH3AA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.340201235 (504h51m18.21128428s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.379076200 (504h51m18.250159245s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.474362322 (504h51m18.345445367s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.555133063 (504h51m18.426216108s) [noise=true , address=77.90.31.17:4791 , piece_id=QFHBBE4B5DWZU7HTAARXW655MKX4YPKPOXVM77CH5DCPSLZ7UGNA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.707673447 (504h51m18.578756492s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734654664 (504h51m18.605737709s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734678970 (504h51m18.605762005s) [seg_pos={0 0}, upload=443005]: Segment reached optimal threshold of 14 pieces. Dec 26 16:20:18.734942853 (504h51m18.606025888s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.734952842 (504h51m18.606035887s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735555389 (504h51m18.606638434s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection Dec 26 16:20:18.735566701 (504h51m18.606649746s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735681349 (504h51m18.606764394s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735699113 (504h51m18.606782148s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735826605 (504h51m18.60690968s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735837737 (504h51m18.606920782s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735908392 (504h51m18.606991447s) [seg_pos={0 0}, upload=443005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:20:18.736024182 (504h51m18.607107227s) [upload=443005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:20:18.736031275 (504h51m18.60711431s) [upload=443005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:20:19.031573339 (504h51m18.902656384s) [upload=443005]: Done uploading segments... Dec 26 16:20:19.031602895 (504h51m18.90268594s) [upload=443005]: Upload finished. err: <nil> Duration: 38.020901424s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [...X....XX...XX.........X.............XXX...X...XX.....XX...] duration: 3.386195396s trace: https://jaeger.dev.storj.io/trace/7e43da40a4822a39 FAIL (2025-12-26 16:24:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.386288754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.....XX...................X.....X..XX..XXX.X...XX.X.XX.X....] duration: 7.118732089s trace: https://jaeger.dev.storj.io/trace/69c79e1728802acc FAIL (2025-12-26 16:23:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 26 16:21:21.334630884 (504h52m21.205713929s) [upload=443021]: Got next segment. Inline: false Dec 26 16:21:21.334700176 (504h52m21.205783211s) [upload=443021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:21:21.334707751 (504h52m21.205790786s) [upload=443021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:21:21.591847600 (504h52m21.462930635s) [seg_pos={0 0}, upload=443021]: Begin upload segment... Dec 26 16:21:21.591932832 (504h52m21.463015877s) [noise=true , address=145.79.227.48:4302 , piece_id=LBLDI6UPAANLADOBFRMRPCG4YHYBKUHENWFZ56T2HDGZ3CELD6FQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.595896960 (504h52m21.466979995s) [noise=true , address=145.79.227.15:4875 , piece_id=J34HAXDYJ6GPZH7VXV7ITLGMQ2JYC5ZDYSOTLVVNELS5C2JHDFMA, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.598190928 (504h52m21.469273974s) [noise=true , address=45.91.116.5:4300 , piece_id=SVNLJADSHVTUU472AVD3RED6FRUJKUUH6OKBWAECZTTHBY3FWHFQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.599944899 (504h52m21.471027944s) [noise=true , address=145.79.227.42:4122 , piece_id=M4HLMR726ZYOPL4MLJHHFRWG6CXILJM5BAY27KT7FKWD3HRFKFHQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.601290721 (504h52m21.472373766s) [seg_pos={0 0}, upload=443021]: Done begin upload segment. Dec 26 16:21:21.601349553 (504h52m21.472432588s) [noise=true , address=145.79.227.6:4257 , piece_id=MKXQSFR6SWJPBDREIKDNKUXJ4XUYE7WSQD4HKNB3NUZQJCVUOGIQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.602597941 (504h52m21.473680986s) [noise=true , address=145.79.227.29:4941 , piece_id=K65LJVCJQUWV6BQBRYQ3ZMRAILA5SZW7GHW7SJCCZI6FJ3DWBV6A, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.604132242 (504h52m21.475215277s) [noise=true , address=145.79.227.47:4266 , piece_id=4ELAIAMEY2ZHWX5I4RLKMCBEYYBN7SWZDZL62YS5M7IROEJEEIKA, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.605160159 (504h52m21.476243214s) [noise=true , address=45.91.116.4:4937 , piece_id=4TX3E7FQEIPHWNAZBA35BX3PQTKGC23SPTEBFMHV4QNRGSVLAVGQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.606536550 (504h52m21.477619585s) [noise=true , address=145.79.227.17:4971 , piece_id=CYJTPNZSYNBNSIFMDMBWHVODZMXQTONRVJDICB3KBTGNA2G3OYGA, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.607229610 (504h52m21.478312665s) [noise=true , address=145.79.227.13:4803 , piece_id=7OAQAPJQESTTS2HWWNL47PGBHN7CSE6D4MOGQNSLXKC5PWFDSTVQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.608110497 (504h52m21.479193542s) [noise=true , address=77.90.31.18:4808 , piece_id=XDCM4YDB43OVHWX34PYTIGF5LEVMPUIS6XH2DEGAXC6QHNOJTI3Q, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.609448986 (504h52m21.480532031s) [noise=true , address=145.79.227.39:4015 , piece_id=5DST4NJ4AQMX2PNXUWBBSX3WGN2U5AAXW5KI4XXGBVVSTQFSMRGA, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.609544788 (504h52m21.480627843s) [noise=true , address=145.79.227.41:4087 , piece_id=HMS5JFKAGPLWTQMHVCNUZSDT47572PV7TXH3RFZSASTQG7RM5ONQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.610640515 (504h52m21.48172355s) [noise=true , address=145.79.227.45:4229 , piece_id=63G2EK37WNJWRFNZYVZQRA62BVRWSH3DIA7CPDLMWFBWWNNX2QNQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.611727645 (504h52m21.48281069s) [noise=true , address=45.91.116.11:4120 , piece_id=YV7EDQRKSV2WJO6KHUWY7575BCII67DJMSHZSLHWJQ3IGF65AQLQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.612875010 (504h52m21.483958045s) [noise=true , address=45.91.116.10:4081 , piece_id=3JFFCL5NDX7ZJ2LPFUXXGT2LMUA5M7J4FMZGMJHJCSI3K5SWSNDA, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:23.220103954 (504h52m23.091186999s) [upload=443021]: Next returned nil segment Dec 26 16:21:23.220116057 (504h52m23.091199092s) [upload=443021]: Waiting for error group managing segments... Dec 26 16:21:23.601397364 (504h52m23.472480399s) [noise=true , address=145.79.227.15:4875 , piece_id=J34HAXDYJ6GPZH7VXV7ITLGMQ2JYC5ZDYSOTLVVNELS5C2JHDFMA, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.605238828 (504h52m23.476321873s) [noise=true , address=145.79.227.42:4122 , piece_id=M4HLMR726ZYOPL4MLJHHFRWG6CXILJM5BAY27KT7FKWD3HRFKFHQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.615995431 (504h52m23.487078466s) [noise=true , address=145.79.227.48:4302 , piece_id=LBLDI6UPAANLADOBFRMRPCG4YHYBKUHENWFZ56T2HDGZ3CELD6FQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.616510261 (504h52m23.487593296s) [noise=true , address=145.79.227.47:4266 , piece_id=4ELAIAMEY2ZHWX5I4RLKMCBEYYBN7SWZDZL62YS5M7IROEJEEIKA, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.620088113 (504h52m23.491171148s) [noise=true , address=145.79.227.41:4087 , piece_id=HMS5JFKAGPLWTQMHVCNUZSDT47572PV7TXH3RFZSASTQG7RM5ONQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.624887863 (504h52m23.495970898s) [noise=true , address=145.79.227.17:4971 , piece_id=CYJTPNZSYNBNSIFMDMBWHVODZMXQTONRVJDICB3KBTGNA2G3OYGA, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.632784950 (504h52m23.503867995s) [noise=true , address=145.79.227.39:4015 , piece_id=5DST4NJ4AQMX2PNXUWBBSX3WGN2U5AAXW5KI4XXGBVVSTQFSMRGA, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.637987698 (504h52m23.509070743s) [noise=true , address=145.79.227.45:4229 , piece_id=63G2EK37WNJWRFNZYVZQRA62BVRWSH3DIA7CPDLMWFBWWNNX2QNQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.639022678 (504h52m23.510105723s) [noise=true , address=145.79.227.13:4803 , piece_id=7OAQAPJQESTTS2HWWNL47PGBHN7CSE6D4MOGQNSLXKC5PWFDSTVQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.840165660 (504h52m23.711248695s) [noise=true , address=45.91.116.5:4300 , piece_id=SVNLJADSHVTUU472AVD3RED6FRUJKUUH6OKBWAECZTTHBY3FWHFQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.974003410 (504h52m23.845086445s) [noise=true , address=45.91.116.4:4937 , piece_id=4TX3E7FQEIPHWNAZBA35BX3PQTKGC23SPTEBFMHV4QNRGSVLAVGQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:24.126817415 (504h52m23.99790046s) [noise=true , address=45.91.116.10:4081 , piece_id=3JFFCL5NDX7ZJ2LPFUXXGT2LMUA5M7J4FMZGMJHJCSI3K5SWSNDA, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:24.305965214 (504h52m24.177048259s) [noise=true , address=77.90.31.18:4808 , piece_id=XDCM4YDB43OVHWX34PYTIGF5LEVMPUIS6XH2DEGAXC6QHNOJTI3Q, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:24.305993628 (504h52m24.177076673s) [seg_pos={0 0}, upload=443021]: Segment reached optimal threshold of 13 pieces. Dec 26 16:21:24.306259954 (504h52m24.177342989s) [noise=true , address=145.79.227.29:4941 , piece_id=K65LJVCJQUWV6BQBRYQ3ZMRAILA5SZW7GHW7SJCCZI6FJ3DWBV6A, seg_pos={0 0}, upload=443021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 16:21:24.306267979 (504h52m24.177351024s) [noise=true , address=145.79.227.29:4941 , piece_id=K65LJVCJQUWV6BQBRYQ3ZMRAILA5SZW7GHW7SJCCZI6FJ3DWBV6A, seg_pos={0 0}, upload=443021]: Long tail cancellation due to optimal threshold Dec 26 16:21:24.306388830 (504h52m24.177471875s) [noise=true , address=45.91.116.11:4120 , piece_id=YV7EDQRKSV2WJO6KHUWY7575BCII67DJMSHZSLHWJQ3IGF65AQLQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:44776->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:44776->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44776->45.91.116.11:4120: use of closed network connection Dec 26 16:21:24.306397616 (504h52m24.177480652s) [noise=true , address=45.91.116.11:4120 , piece_id=YV7EDQRKSV2WJO6KHUWY7575BCII67DJMSHZSLHWJQ3IGF65AQLQ, seg_pos={0 0}, upload=443021]: Long tail cancellation due to optimal threshold Dec 26 16:21:24.306435168 (504h52m24.177518203s) [noise=true , address=145.79.227.6:4257 , piece_id=MKXQSFR6SWJPBDREIKDNKUXJ4XUYE7WSQD4HKNB3NUZQJCVUOGIQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:40536->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:40536->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40536->145.79.227.6:4257: use of closed network connection Dec 26 16:21:24.306441169 (504h52m24.177524214s) [noise=true , address=145.79.227.6:4257 , piece_id=MKXQSFR6SWJPBDREIKDNKUXJ4XUYE7WSQD4HKNB3NUZQJCVUOGIQ, seg_pos={0 0}, upload=443021]: Long tail cancellation due to optimal threshold Dec 26 16:21:24.306533075 (504h52m24.17761611s) [seg_pos={0 0}, upload=443021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:21:24.306625471 (504h52m24.177708506s) [upload=443021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:21:24.306630500 (504h52m24.177713535s) [upload=443021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:21:24.608089450 (504h52m24.479172495s) [upload=443021]: Done uploading segments... Dec 26 16:21:24.608113426 (504h52m24.479196471s) [upload=443021]: Upload finished. err: <nil> Duration: 7.118800408s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......XX...................X.....X..XX..XXX.X...XX.X.XX.X...] duration: 6.250764419s trace: https://jaeger.dev.storj.io/trace/69c79e1728802acc FAIL (2025-12-26 16:23:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 26 16:21:21.334630884 (504h52m21.205713929s) [upload=443021]: Got next segment. Inline: false Dec 26 16:21:21.334700176 (504h52m21.205783211s) [upload=443021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:21:21.334707751 (504h52m21.205790786s) [upload=443021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:21:21.591847600 (504h52m21.462930635s) [seg_pos={0 0}, upload=443021]: Begin upload segment... Dec 26 16:21:21.591932832 (504h52m21.463015877s) [noise=true , address=145.79.227.48:4302 , piece_id=LBLDI6UPAANLADOBFRMRPCG4YHYBKUHENWFZ56T2HDGZ3CELD6FQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.595896960 (504h52m21.466979995s) [noise=true , address=145.79.227.15:4875 , piece_id=J34HAXDYJ6GPZH7VXV7ITLGMQ2JYC5ZDYSOTLVVNELS5C2JHDFMA, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.598190928 (504h52m21.469273974s) [noise=true , address=45.91.116.5:4300 , piece_id=SVNLJADSHVTUU472AVD3RED6FRUJKUUH6OKBWAECZTTHBY3FWHFQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.599944899 (504h52m21.471027944s) [noise=true , address=145.79.227.42:4122 , piece_id=M4HLMR726ZYOPL4MLJHHFRWG6CXILJM5BAY27KT7FKWD3HRFKFHQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.601290721 (504h52m21.472373766s) [seg_pos={0 0}, upload=443021]: Done begin upload segment. Dec 26 16:21:21.601349553 (504h52m21.472432588s) [noise=true , address=145.79.227.6:4257 , piece_id=MKXQSFR6SWJPBDREIKDNKUXJ4XUYE7WSQD4HKNB3NUZQJCVUOGIQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.602597941 (504h52m21.473680986s) [noise=true , address=145.79.227.29:4941 , piece_id=K65LJVCJQUWV6BQBRYQ3ZMRAILA5SZW7GHW7SJCCZI6FJ3DWBV6A, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.604132242 (504h52m21.475215277s) [noise=true , address=145.79.227.47:4266 , piece_id=4ELAIAMEY2ZHWX5I4RLKMCBEYYBN7SWZDZL62YS5M7IROEJEEIKA, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.605160159 (504h52m21.476243214s) [noise=true , address=45.91.116.4:4937 , piece_id=4TX3E7FQEIPHWNAZBA35BX3PQTKGC23SPTEBFMHV4QNRGSVLAVGQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.606536550 (504h52m21.477619585s) [noise=true , address=145.79.227.17:4971 , piece_id=CYJTPNZSYNBNSIFMDMBWHVODZMXQTONRVJDICB3KBTGNA2G3OYGA, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.607229610 (504h52m21.478312665s) [noise=true , address=145.79.227.13:4803 , piece_id=7OAQAPJQESTTS2HWWNL47PGBHN7CSE6D4MOGQNSLXKC5PWFDSTVQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.608110497 (504h52m21.479193542s) [noise=true , address=77.90.31.18:4808 , piece_id=XDCM4YDB43OVHWX34PYTIGF5LEVMPUIS6XH2DEGAXC6QHNOJTI3Q, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.609448986 (504h52m21.480532031s) [noise=true , address=145.79.227.39:4015 , piece_id=5DST4NJ4AQMX2PNXUWBBSX3WGN2U5AAXW5KI4XXGBVVSTQFSMRGA, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.609544788 (504h52m21.480627843s) [noise=true , address=145.79.227.41:4087 , piece_id=HMS5JFKAGPLWTQMHVCNUZSDT47572PV7TXH3RFZSASTQG7RM5ONQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.610640515 (504h52m21.48172355s) [noise=true , address=145.79.227.45:4229 , piece_id=63G2EK37WNJWRFNZYVZQRA62BVRWSH3DIA7CPDLMWFBWWNNX2QNQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.611727645 (504h52m21.48281069s) [noise=true , address=45.91.116.11:4120 , piece_id=YV7EDQRKSV2WJO6KHUWY7575BCII67DJMSHZSLHWJQ3IGF65AQLQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.612875010 (504h52m21.483958045s) [noise=true , address=45.91.116.10:4081 , piece_id=3JFFCL5NDX7ZJ2LPFUXXGT2LMUA5M7J4FMZGMJHJCSI3K5SWSNDA, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:23.220103954 (504h52m23.091186999s) [upload=443021]: Next returned nil segment Dec 26 16:21:23.220116057 (504h52m23.091199092s) [upload=443021]: Waiting for error group managing segments... Dec 26 16:21:23.601397364 (504h52m23.472480399s) [noise=true , address=145.79.227.15:4875 , piece_id=J34HAXDYJ6GPZH7VXV7ITLGMQ2JYC5ZDYSOTLVVNELS5C2JHDFMA, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.605238828 (504h52m23.476321873s) [noise=true , address=145.79.227.42:4122 , piece_id=M4HLMR726ZYOPL4MLJHHFRWG6CXILJM5BAY27KT7FKWD3HRFKFHQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.615995431 (504h52m23.487078466s) [noise=true , address=145.79.227.48:4302 , piece_id=LBLDI6UPAANLADOBFRMRPCG4YHYBKUHENWFZ56T2HDGZ3CELD6FQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.616510261 (504h52m23.487593296s) [noise=true , address=145.79.227.47:4266 , piece_id=4ELAIAMEY2ZHWX5I4RLKMCBEYYBN7SWZDZL62YS5M7IROEJEEIKA, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.620088113 (504h52m23.491171148s) [noise=true , address=145.79.227.41:4087 , piece_id=HMS5JFKAGPLWTQMHVCNUZSDT47572PV7TXH3RFZSASTQG7RM5ONQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.624887863 (504h52m23.495970898s) [noise=true , address=145.79.227.17:4971 , piece_id=CYJTPNZSYNBNSIFMDMBWHVODZMXQTONRVJDICB3KBTGNA2G3OYGA, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.632784950 (504h52m23.503867995s) [noise=true , address=145.79.227.39:4015 , piece_id=5DST4NJ4AQMX2PNXUWBBSX3WGN2U5AAXW5KI4XXGBVVSTQFSMRGA, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.637987698 (504h52m23.509070743s) [noise=true , address=145.79.227.45:4229 , piece_id=63G2EK37WNJWRFNZYVZQRA62BVRWSH3DIA7CPDLMWFBWWNNX2QNQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.639022678 (504h52m23.510105723s) [noise=true , address=145.79.227.13:4803 , piece_id=7OAQAPJQESTTS2HWWNL47PGBHN7CSE6D4MOGQNSLXKC5PWFDSTVQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.840165660 (504h52m23.711248695s) [noise=true , address=45.91.116.5:4300 , piece_id=SVNLJADSHVTUU472AVD3RED6FRUJKUUH6OKBWAECZTTHBY3FWHFQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.974003410 (504h52m23.845086445s) [noise=true , address=45.91.116.4:4937 , piece_id=4TX3E7FQEIPHWNAZBA35BX3PQTKGC23SPTEBFMHV4QNRGSVLAVGQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:24.126817415 (504h52m23.99790046s) [noise=true , address=45.91.116.10:4081 , piece_id=3JFFCL5NDX7ZJ2LPFUXXGT2LMUA5M7J4FMZGMJHJCSI3K5SWSNDA, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:24.305965214 (504h52m24.177048259s) [noise=true , address=77.90.31.18:4808 , piece_id=XDCM4YDB43OVHWX34PYTIGF5LEVMPUIS6XH2DEGAXC6QHNOJTI3Q, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:24.305993628 (504h52m24.177076673s) [seg_pos={0 0}, upload=443021]: Segment reached optimal threshold of 13 pieces. Dec 26 16:21:24.306259954 (504h52m24.177342989s) [noise=true , address=145.79.227.29:4941 , piece_id=K65LJVCJQUWV6BQBRYQ3ZMRAILA5SZW7GHW7SJCCZI6FJ3DWBV6A, seg_pos={0 0}, upload=443021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 16:21:24.306267979 (504h52m24.177351024s) [noise=true , address=145.79.227.29:4941 , piece_id=K65LJVCJQUWV6BQBRYQ3ZMRAILA5SZW7GHW7SJCCZI6FJ3DWBV6A, seg_pos={0 0}, upload=443021]: Long tail cancellation due to optimal threshold Dec 26 16:21:24.306388830 (504h52m24.177471875s) [noise=true , address=45.91.116.11:4120 , piece_id=YV7EDQRKSV2WJO6KHUWY7575BCII67DJMSHZSLHWJQ3IGF65AQLQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:44776->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:44776->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44776->45.91.116.11:4120: use of closed network connection Dec 26 16:21:24.306397616 (504h52m24.177480652s) [noise=true , address=45.91.116.11:4120 , piece_id=YV7EDQRKSV2WJO6KHUWY7575BCII67DJMSHZSLHWJQ3IGF65AQLQ, seg_pos={0 0}, upload=443021]: Long tail cancellation due to optimal threshold Dec 26 16:21:24.306435168 (504h52m24.177518203s) [noise=true , address=145.79.227.6:4257 , piece_id=MKXQSFR6SWJPBDREIKDNKUXJ4XUYE7WSQD4HKNB3NUZQJCVUOGIQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:40536->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:40536->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40536->145.79.227.6:4257: use of closed network connection Dec 26 16:21:24.306441169 (504h52m24.177524214s) [noise=true , address=145.79.227.6:4257 , piece_id=MKXQSFR6SWJPBDREIKDNKUXJ4XUYE7WSQD4HKNB3NUZQJCVUOGIQ, seg_pos={0 0}, upload=443021]: Long tail cancellation due to optimal threshold Dec 26 16:21:24.306533075 (504h52m24.17761611s) [seg_pos={0 0}, upload=443021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:21:24.306625471 (504h52m24.177708506s) [upload=443021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:21:24.306630500 (504h52m24.177713535s) [upload=443021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:21:24.608089450 (504h52m24.479172495s) [upload=443021]: Done uploading segments... Dec 26 16:21:24.608113426 (504h52m24.479196471s) [upload=443021]: Upload finished. err: <nil> Duration: 6.25078135s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X......XX...................X.....X..XX..XXX.X...XX.X.XX.X..] duration: 24.485394355s trace: https://jaeger.dev.storj.io/trace/69c79e1728802acc FAIL (2025-12-26 16:23:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 26 16:21:21.334630884 (504h52m21.205713929s) [upload=443021]: Got next segment. Inline: false Dec 26 16:21:21.334700176 (504h52m21.205783211s) [upload=443021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:21:21.334707751 (504h52m21.205790786s) [upload=443021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:21:21.591847600 (504h52m21.462930635s) [seg_pos={0 0}, upload=443021]: Begin upload segment... Dec 26 16:21:21.591932832 (504h52m21.463015877s) [noise=true , address=145.79.227.48:4302 , piece_id=LBLDI6UPAANLADOBFRMRPCG4YHYBKUHENWFZ56T2HDGZ3CELD6FQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.595896960 (504h52m21.466979995s) [noise=true , address=145.79.227.15:4875 , piece_id=J34HAXDYJ6GPZH7VXV7ITLGMQ2JYC5ZDYSOTLVVNELS5C2JHDFMA, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.598190928 (504h52m21.469273974s) [noise=true , address=45.91.116.5:4300 , piece_id=SVNLJADSHVTUU472AVD3RED6FRUJKUUH6OKBWAECZTTHBY3FWHFQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.599944899 (504h52m21.471027944s) [noise=true , address=145.79.227.42:4122 , piece_id=M4HLMR726ZYOPL4MLJHHFRWG6CXILJM5BAY27KT7FKWD3HRFKFHQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.601290721 (504h52m21.472373766s) [seg_pos={0 0}, upload=443021]: Done begin upload segment. Dec 26 16:21:21.601349553 (504h52m21.472432588s) [noise=true , address=145.79.227.6:4257 , piece_id=MKXQSFR6SWJPBDREIKDNKUXJ4XUYE7WSQD4HKNB3NUZQJCVUOGIQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.602597941 (504h52m21.473680986s) [noise=true , address=145.79.227.29:4941 , piece_id=K65LJVCJQUWV6BQBRYQ3ZMRAILA5SZW7GHW7SJCCZI6FJ3DWBV6A, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.604132242 (504h52m21.475215277s) [noise=true , address=145.79.227.47:4266 , piece_id=4ELAIAMEY2ZHWX5I4RLKMCBEYYBN7SWZDZL62YS5M7IROEJEEIKA, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.605160159 (504h52m21.476243214s) [noise=true , address=45.91.116.4:4937 , piece_id=4TX3E7FQEIPHWNAZBA35BX3PQTKGC23SPTEBFMHV4QNRGSVLAVGQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.606536550 (504h52m21.477619585s) [noise=true , address=145.79.227.17:4971 , piece_id=CYJTPNZSYNBNSIFMDMBWHVODZMXQTONRVJDICB3KBTGNA2G3OYGA, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.607229610 (504h52m21.478312665s) [noise=true , address=145.79.227.13:4803 , piece_id=7OAQAPJQESTTS2HWWNL47PGBHN7CSE6D4MOGQNSLXKC5PWFDSTVQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.608110497 (504h52m21.479193542s) [noise=true , address=77.90.31.18:4808 , piece_id=XDCM4YDB43OVHWX34PYTIGF5LEVMPUIS6XH2DEGAXC6QHNOJTI3Q, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.609448986 (504h52m21.480532031s) [noise=true , address=145.79.227.39:4015 , piece_id=5DST4NJ4AQMX2PNXUWBBSX3WGN2U5AAXW5KI4XXGBVVSTQFSMRGA, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.609544788 (504h52m21.480627843s) [noise=true , address=145.79.227.41:4087 , piece_id=HMS5JFKAGPLWTQMHVCNUZSDT47572PV7TXH3RFZSASTQG7RM5ONQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.610640515 (504h52m21.48172355s) [noise=true , address=145.79.227.45:4229 , piece_id=63G2EK37WNJWRFNZYVZQRA62BVRWSH3DIA7CPDLMWFBWWNNX2QNQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.611727645 (504h52m21.48281069s) [noise=true , address=45.91.116.11:4120 , piece_id=YV7EDQRKSV2WJO6KHUWY7575BCII67DJMSHZSLHWJQ3IGF65AQLQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.612875010 (504h52m21.483958045s) [noise=true , address=45.91.116.10:4081 , piece_id=3JFFCL5NDX7ZJ2LPFUXXGT2LMUA5M7J4FMZGMJHJCSI3K5SWSNDA, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:23.220103954 (504h52m23.091186999s) [upload=443021]: Next returned nil segment Dec 26 16:21:23.220116057 (504h52m23.091199092s) [upload=443021]: Waiting for error group managing segments... Dec 26 16:21:23.601397364 (504h52m23.472480399s) [noise=true , address=145.79.227.15:4875 , piece_id=J34HAXDYJ6GPZH7VXV7ITLGMQ2JYC5ZDYSOTLVVNELS5C2JHDFMA, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.605238828 (504h52m23.476321873s) [noise=true , address=145.79.227.42:4122 , piece_id=M4HLMR726ZYOPL4MLJHHFRWG6CXILJM5BAY27KT7FKWD3HRFKFHQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.615995431 (504h52m23.487078466s) [noise=true , address=145.79.227.48:4302 , piece_id=LBLDI6UPAANLADOBFRMRPCG4YHYBKUHENWFZ56T2HDGZ3CELD6FQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.616510261 (504h52m23.487593296s) [noise=true , address=145.79.227.47:4266 , piece_id=4ELAIAMEY2ZHWX5I4RLKMCBEYYBN7SWZDZL62YS5M7IROEJEEIKA, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.620088113 (504h52m23.491171148s) [noise=true , address=145.79.227.41:4087 , piece_id=HMS5JFKAGPLWTQMHVCNUZSDT47572PV7TXH3RFZSASTQG7RM5ONQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.624887863 (504h52m23.495970898s) [noise=true , address=145.79.227.17:4971 , piece_id=CYJTPNZSYNBNSIFMDMBWHVODZMXQTONRVJDICB3KBTGNA2G3OYGA, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.632784950 (504h52m23.503867995s) [noise=true , address=145.79.227.39:4015 , piece_id=5DST4NJ4AQMX2PNXUWBBSX3WGN2U5AAXW5KI4XXGBVVSTQFSMRGA, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.637987698 (504h52m23.509070743s) [noise=true , address=145.79.227.45:4229 , piece_id=63G2EK37WNJWRFNZYVZQRA62BVRWSH3DIA7CPDLMWFBWWNNX2QNQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.639022678 (504h52m23.510105723s) [noise=true , address=145.79.227.13:4803 , piece_id=7OAQAPJQESTTS2HWWNL47PGBHN7CSE6D4MOGQNSLXKC5PWFDSTVQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.840165660 (504h52m23.711248695s) [noise=true , address=45.91.116.5:4300 , piece_id=SVNLJADSHVTUU472AVD3RED6FRUJKUUH6OKBWAECZTTHBY3FWHFQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.974003410 (504h52m23.845086445s) [noise=true , address=45.91.116.4:4937 , piece_id=4TX3E7FQEIPHWNAZBA35BX3PQTKGC23SPTEBFMHV4QNRGSVLAVGQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:24.126817415 (504h52m23.99790046s) [noise=true , address=45.91.116.10:4081 , piece_id=3JFFCL5NDX7ZJ2LPFUXXGT2LMUA5M7J4FMZGMJHJCSI3K5SWSNDA, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:24.305965214 (504h52m24.177048259s) [noise=true , address=77.90.31.18:4808 , piece_id=XDCM4YDB43OVHWX34PYTIGF5LEVMPUIS6XH2DEGAXC6QHNOJTI3Q, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:24.305993628 (504h52m24.177076673s) [seg_pos={0 0}, upload=443021]: Segment reached optimal threshold of 13 pieces. Dec 26 16:21:24.306259954 (504h52m24.177342989s) [noise=true , address=145.79.227.29:4941 , piece_id=K65LJVCJQUWV6BQBRYQ3ZMRAILA5SZW7GHW7SJCCZI6FJ3DWBV6A, seg_pos={0 0}, upload=443021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 16:21:24.306267979 (504h52m24.177351024s) [noise=true , address=145.79.227.29:4941 , piece_id=K65LJVCJQUWV6BQBRYQ3ZMRAILA5SZW7GHW7SJCCZI6FJ3DWBV6A, seg_pos={0 0}, upload=443021]: Long tail cancellation due to optimal threshold Dec 26 16:21:24.306388830 (504h52m24.177471875s) [noise=true , address=45.91.116.11:4120 , piece_id=YV7EDQRKSV2WJO6KHUWY7575BCII67DJMSHZSLHWJQ3IGF65AQLQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:44776->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:44776->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44776->45.91.116.11:4120: use of closed network connection Dec 26 16:21:24.306397616 (504h52m24.177480652s) [noise=true , address=45.91.116.11:4120 , piece_id=YV7EDQRKSV2WJO6KHUWY7575BCII67DJMSHZSLHWJQ3IGF65AQLQ, seg_pos={0 0}, upload=443021]: Long tail cancellation due to optimal threshold Dec 26 16:21:24.306435168 (504h52m24.177518203s) [noise=true , address=145.79.227.6:4257 , piece_id=MKXQSFR6SWJPBDREIKDNKUXJ4XUYE7WSQD4HKNB3NUZQJCVUOGIQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:40536->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:40536->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40536->145.79.227.6:4257: use of closed network connection Dec 26 16:21:24.306441169 (504h52m24.177524214s) [noise=true , address=145.79.227.6:4257 , piece_id=MKXQSFR6SWJPBDREIKDNKUXJ4XUYE7WSQD4HKNB3NUZQJCVUOGIQ, seg_pos={0 0}, upload=443021]: Long tail cancellation due to optimal threshold Dec 26 16:21:24.306533075 (504h52m24.17761611s) [seg_pos={0 0}, upload=443021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:21:24.306625471 (504h52m24.177708506s) [upload=443021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:21:24.306630500 (504h52m24.177713535s) [upload=443021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:21:24.608089450 (504h52m24.479172495s) [upload=443021]: Done uploading segments... Dec 26 16:21:24.608113426 (504h52m24.479196471s) [upload=443021]: Upload finished. err: <nil> Duration: 24.485427307s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.XX.X....X....X..X...X.......XX..........X.........X..X...] duration: 15.209907228s trace: https://jaeger.dev.storj.io/trace/3e3dfe0984f19d4e FAIL (2025-12-26 16:23:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.466998847 (504h51m15.338081892s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467028614 (504h51m15.338111659s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467359594 (504h51m15.338442639s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467908379 (504h51m15.338991424s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.468520054 (504h51m15.339603089s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469055293 (504h51m15.340138328s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469497605 (504h51m15.34058064s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469973642 (504h51m15.341056687s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.470454397 (504h51m15.341537443s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471043630 (504h51m15.342126695s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471470552 (504h51m15.342553587s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471676735 (504h51m15.34275977s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472062900 (504h51m15.343145935s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472369224 (504h51m15.343452259s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472823849 (504h51m15.343906884s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:16.743410939 (504h51m16.614493984s) [upload=443005]: Next returned nil segment Dec 26 16:20:16.743428862 (504h51m16.614511907s) [upload=443005]: Waiting for error group managing segments... Dec 26 16:20:17.091086984 (504h51m16.962170029s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.389229539 (504h51m17.260312594s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.464846438 (504h51m17.335929483s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.542075705 (504h51m17.41315875s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.623422163 (504h51m17.494505208s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.665409184 (504h51m17.536492219s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.301995626 (504h51m18.173078671s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.311693532 (504h51m18.182776567s) [noise=true , address=145.79.227.34:4915 , piece_id=7ZBWNPUF54BEGIXTDRDI3J7HNEZ63MEXKUBMRZW6PFWWNCEAH3AA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.340201235 (504h51m18.21128428s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.379076200 (504h51m18.250159245s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.474362322 (504h51m18.345445367s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.555133063 (504h51m18.426216108s) [noise=true , address=77.90.31.17:4791 , piece_id=QFHBBE4B5DWZU7HTAARXW655MKX4YPKPOXVM77CH5DCPSLZ7UGNA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.707673447 (504h51m18.578756492s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734654664 (504h51m18.605737709s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734678970 (504h51m18.605762005s) [seg_pos={0 0}, upload=443005]: Segment reached optimal threshold of 14 pieces. Dec 26 16:20:18.734942853 (504h51m18.606025888s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.734952842 (504h51m18.606035887s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735555389 (504h51m18.606638434s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection Dec 26 16:20:18.735566701 (504h51m18.606649746s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735681349 (504h51m18.606764394s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735699113 (504h51m18.606782148s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735826605 (504h51m18.60690968s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735837737 (504h51m18.606920782s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735908392 (504h51m18.606991447s) [seg_pos={0 0}, upload=443005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:20:18.736024182 (504h51m18.607107227s) [upload=443005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:20:18.736031275 (504h51m18.60711431s) [upload=443005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:20:19.031573339 (504h51m18.902656384s) [upload=443005]: Done uploading segments... Dec 26 16:20:19.031602895 (504h51m18.90268594s) [upload=443005]: Upload finished. err: <nil> Duration: 15.20993466s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [X...X....XX...XX.........X.............XXX...X...XX.....XX..] duration: 5.001003985s trace: https://jaeger.dev.storj.io/trace/7e43da40a4822a39 FAIL (2025-12-26 16:23:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001030595s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.XX.X....X....X..X...X.......XX..........X.........X..X..] duration: 34.744134386s trace: https://jaeger.dev.storj.io/trace/3e3dfe0984f19d4e FAIL (2025-12-26 16:22:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.466998847 (504h51m15.338081892s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467028614 (504h51m15.338111659s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467359594 (504h51m15.338442639s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.467908379 (504h51m15.338991424s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.468520054 (504h51m15.339603089s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469055293 (504h51m15.340138328s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469497605 (504h51m15.34058064s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.469973642 (504h51m15.341056687s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.470454397 (504h51m15.341537443s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471043630 (504h51m15.342126695s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471470552 (504h51m15.342553587s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.471676735 (504h51m15.34275977s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472062900 (504h51m15.343145935s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472369224 (504h51m15.343452259s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:15.472823849 (504h51m15.343906884s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Uploading piece... Dec 26 16:20:16.743410939 (504h51m16.614493984s) [upload=443005]: Next returned nil segment Dec 26 16:20:16.743428862 (504h51m16.614511907s) [upload=443005]: Waiting for error group managing segments... Dec 26 16:20:17.091086984 (504h51m16.962170029s) [noise=true , address=64.49.13.138:4487 , piece_id=2ZMXGWZKPYAKUVI6TJOT3JDMALNWFHJQJTM6K2UAOCNYTKXYMXGQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.389229539 (504h51m17.260312594s) [noise=true , address=145.79.227.50:4377 , piece_id=RCLVMFTAYKGEUR7XQC5OASJ5VKIK2M5HM4RO6JVSMAHWLT7B6D3A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.464846438 (504h51m17.335929483s) [noise=true , address=145.79.227.19:4088 , piece_id=OX45UAH7VHTIXCTBV2D6DBV3JLCFO5B4IGIZATS6SHTS3CYDFRXA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.542075705 (504h51m17.41315875s) [noise=true , address=77.90.31.18:4839 , piece_id=5PNHJDHIZPHKKHS3HUIZRXLPFH2CIJLMJ34KXEJVS62IN4RWNBCA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.623422163 (504h51m17.494505208s) [noise=true , address=77.90.31.15:4729 , piece_id=RTFYFZJGDXX6KVJHDCDK54LFYMRLJZT2SMBDL57NMRZJPEKNSKPQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:17.665409184 (504h51m17.536492219s) [noise=true , address=45.91.116.9:4275 , piece_id=7NHFACLW6JAWAILMQUIHGABFTR4CJDRV3WBNOURZDU7YVIZ25Q2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.301995626 (504h51m18.173078671s) [noise=true , address=145.79.227.44:4201 , piece_id=H4HC65ZJ5P2JU6TAMBNZ6VLMYRQH42J4Y3AQLZ4FAKAWWK55DKUQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.311693532 (504h51m18.182776567s) [noise=true , address=145.79.227.34:4915 , piece_id=7ZBWNPUF54BEGIXTDRDI3J7HNEZ63MEXKUBMRZW6PFWWNCEAH3AA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.340201235 (504h51m18.21128428s) [noise=true , address=145.79.227.34:4929 , piece_id=B6PY5LAIY72ZQFEXZBL6QJUIJ22O3XDVVUYQR52ZHIQGYXXFSQ4Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.379076200 (504h51m18.250159245s) [noise=true , address=77.90.31.16:4739 , piece_id=VKZ4WASV4ZTB5RJLQCSQ44FHAXUF5KGPOZACQFMNLCOP5RYU66MQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.474362322 (504h51m18.345445367s) [noise=true , address=145.79.227.33:4903 , piece_id=EBG2334VIGTLYQ6FTQ6ICRO5HKGVRXKMFKIVTZFATBWXZAFTH23A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.555133063 (504h51m18.426216108s) [noise=true , address=77.90.31.17:4791 , piece_id=QFHBBE4B5DWZU7HTAARXW655MKX4YPKPOXVM77CH5DCPSLZ7UGNA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.707673447 (504h51m18.578756492s) [noise=true , address=77.90.31.13:4649 , piece_id=ZKWOBNES7AH64PBTN4N4BNA4RHSRLMRQPSYA5SGG3CO6YYRIQMIA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734654664 (504h51m18.605737709s) [noise=true , address=77.90.31.11:4564 , piece_id=OBFPJYHZIAB4OTQH2B6BAQU4Q6P4FTLYK37YAY5KTPOHTT7A3O2A, seg_pos={0 0}, upload=443005]: Done uploading piece. err: <nil> Dec 26 16:20:18.734678970 (504h51m18.605762005s) [seg_pos={0 0}, upload=443005]: Segment reached optimal threshold of 14 pieces. Dec 26 16:20:18.734942853 (504h51m18.606025888s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.734952842 (504h51m18.606035887s) [noise=true , address=77.90.31.10:4532 , piece_id=CE4CKXDTVLATMYEFYM3Y7X7PEOSPXGVGDAOTK6NS5V35TCZYRT6Q, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735555389 (504h51m18.606638434s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50774->64.49.13.169:4709: use of closed network connection Dec 26 16:20:18.735566701 (504h51m18.606649746s) [noise=true , address=64.49.13.169:4709 , piece_id=BYA6V2FOAQZSWMS4CDNMFYFE7ZVVE3KYXVUBM227PWJHLJ2DYLAQ, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735681349 (504h51m18.606764394s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Xz49MqUbhiKpnPKri94CdfaNZdvBjarfMFYbW9uL9YJXW3W6n): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735699113 (504h51m18.606782148s) [noise=true , address=45.91.116.14:4210 , piece_id=LFWSGMHQRBR4NRINE5FQF43ID2ISHPUFSCQRRB3H2EP43757CCTA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735826605 (504h51m18.60690968s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu): protocol: expected piece hash; context canceled; EOF Dec 26 16:20:18.735837737 (504h51m18.606920782s) [noise=true , address=45.91.116.7:4033 , piece_id=YXOFWO2TUF6KRAFQXUEENQ3X7LGTCK7V2AZAVZVZJRDEB7SUL7IA, seg_pos={0 0}, upload=443005]: Long tail cancellation due to optimal threshold Dec 26 16:20:18.735908392 (504h51m18.606991447s) [seg_pos={0 0}, upload=443005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:20:18.736024182 (504h51m18.607107227s) [upload=443005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:20:18.736031275 (504h51m18.60711431s) [upload=443005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:20:19.031573339 (504h51m18.902656384s) [upload=443005]: Done uploading segments... Dec 26 16:20:19.031602895 (504h51m18.90268594s) [upload=443005]: Upload finished. err: <nil> Duration: 34.744153763s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X......XX...................X.....X..XX..XXX.X...XX.X.XX.X.] duration: 13.478245793s trace: https://jaeger.dev.storj.io/trace/69c79e1728802acc FAIL (2025-12-26 16:22:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 26 16:21:21.334630884 (504h52m21.205713929s) [upload=443021]: Got next segment. Inline: false Dec 26 16:21:21.334700176 (504h52m21.205783211s) [upload=443021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:21:21.334707751 (504h52m21.205790786s) [upload=443021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:21:21.591847600 (504h52m21.462930635s) [seg_pos={0 0}, upload=443021]: Begin upload segment... Dec 26 16:21:21.591932832 (504h52m21.463015877s) [noise=true , address=145.79.227.48:4302 , piece_id=LBLDI6UPAANLADOBFRMRPCG4YHYBKUHENWFZ56T2HDGZ3CELD6FQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.595896960 (504h52m21.466979995s) [noise=true , address=145.79.227.15:4875 , piece_id=J34HAXDYJ6GPZH7VXV7ITLGMQ2JYC5ZDYSOTLVVNELS5C2JHDFMA, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.598190928 (504h52m21.469273974s) [noise=true , address=45.91.116.5:4300 , piece_id=SVNLJADSHVTUU472AVD3RED6FRUJKUUH6OKBWAECZTTHBY3FWHFQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.599944899 (504h52m21.471027944s) [noise=true , address=145.79.227.42:4122 , piece_id=M4HLMR726ZYOPL4MLJHHFRWG6CXILJM5BAY27KT7FKWD3HRFKFHQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.601290721 (504h52m21.472373766s) [seg_pos={0 0}, upload=443021]: Done begin upload segment. Dec 26 16:21:21.601349553 (504h52m21.472432588s) [noise=true , address=145.79.227.6:4257 , piece_id=MKXQSFR6SWJPBDREIKDNKUXJ4XUYE7WSQD4HKNB3NUZQJCVUOGIQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.602597941 (504h52m21.473680986s) [noise=true , address=145.79.227.29:4941 , piece_id=K65LJVCJQUWV6BQBRYQ3ZMRAILA5SZW7GHW7SJCCZI6FJ3DWBV6A, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.604132242 (504h52m21.475215277s) [noise=true , address=145.79.227.47:4266 , piece_id=4ELAIAMEY2ZHWX5I4RLKMCBEYYBN7SWZDZL62YS5M7IROEJEEIKA, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.605160159 (504h52m21.476243214s) [noise=true , address=45.91.116.4:4937 , piece_id=4TX3E7FQEIPHWNAZBA35BX3PQTKGC23SPTEBFMHV4QNRGSVLAVGQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.606536550 (504h52m21.477619585s) [noise=true , address=145.79.227.17:4971 , piece_id=CYJTPNZSYNBNSIFMDMBWHVODZMXQTONRVJDICB3KBTGNA2G3OYGA, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.607229610 (504h52m21.478312665s) [noise=true , address=145.79.227.13:4803 , piece_id=7OAQAPJQESTTS2HWWNL47PGBHN7CSE6D4MOGQNSLXKC5PWFDSTVQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.608110497 (504h52m21.479193542s) [noise=true , address=77.90.31.18:4808 , piece_id=XDCM4YDB43OVHWX34PYTIGF5LEVMPUIS6XH2DEGAXC6QHNOJTI3Q, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.609448986 (504h52m21.480532031s) [noise=true , address=145.79.227.39:4015 , piece_id=5DST4NJ4AQMX2PNXUWBBSX3WGN2U5AAXW5KI4XXGBVVSTQFSMRGA, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.609544788 (504h52m21.480627843s) [noise=true , address=145.79.227.41:4087 , piece_id=HMS5JFKAGPLWTQMHVCNUZSDT47572PV7TXH3RFZSASTQG7RM5ONQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.610640515 (504h52m21.48172355s) [noise=true , address=145.79.227.45:4229 , piece_id=63G2EK37WNJWRFNZYVZQRA62BVRWSH3DIA7CPDLMWFBWWNNX2QNQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.611727645 (504h52m21.48281069s) [noise=true , address=45.91.116.11:4120 , piece_id=YV7EDQRKSV2WJO6KHUWY7575BCII67DJMSHZSLHWJQ3IGF65AQLQ, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:21.612875010 (504h52m21.483958045s) [noise=true , address=45.91.116.10:4081 , piece_id=3JFFCL5NDX7ZJ2LPFUXXGT2LMUA5M7J4FMZGMJHJCSI3K5SWSNDA, seg_pos={0 0}, upload=443021]: Uploading piece... Dec 26 16:21:23.220103954 (504h52m23.091186999s) [upload=443021]: Next returned nil segment Dec 26 16:21:23.220116057 (504h52m23.091199092s) [upload=443021]: Waiting for error group managing segments... Dec 26 16:21:23.601397364 (504h52m23.472480399s) [noise=true , address=145.79.227.15:4875 , piece_id=J34HAXDYJ6GPZH7VXV7ITLGMQ2JYC5ZDYSOTLVVNELS5C2JHDFMA, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.605238828 (504h52m23.476321873s) [noise=true , address=145.79.227.42:4122 , piece_id=M4HLMR726ZYOPL4MLJHHFRWG6CXILJM5BAY27KT7FKWD3HRFKFHQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.615995431 (504h52m23.487078466s) [noise=true , address=145.79.227.48:4302 , piece_id=LBLDI6UPAANLADOBFRMRPCG4YHYBKUHENWFZ56T2HDGZ3CELD6FQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.616510261 (504h52m23.487593296s) [noise=true , address=145.79.227.47:4266 , piece_id=4ELAIAMEY2ZHWX5I4RLKMCBEYYBN7SWZDZL62YS5M7IROEJEEIKA, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.620088113 (504h52m23.491171148s) [noise=true , address=145.79.227.41:4087 , piece_id=HMS5JFKAGPLWTQMHVCNUZSDT47572PV7TXH3RFZSASTQG7RM5ONQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.624887863 (504h52m23.495970898s) [noise=true , address=145.79.227.17:4971 , piece_id=CYJTPNZSYNBNSIFMDMBWHVODZMXQTONRVJDICB3KBTGNA2G3OYGA, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.632784950 (504h52m23.503867995s) [noise=true , address=145.79.227.39:4015 , piece_id=5DST4NJ4AQMX2PNXUWBBSX3WGN2U5AAXW5KI4XXGBVVSTQFSMRGA, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.637987698 (504h52m23.509070743s) [noise=true , address=145.79.227.45:4229 , piece_id=63G2EK37WNJWRFNZYVZQRA62BVRWSH3DIA7CPDLMWFBWWNNX2QNQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.639022678 (504h52m23.510105723s) [noise=true , address=145.79.227.13:4803 , piece_id=7OAQAPJQESTTS2HWWNL47PGBHN7CSE6D4MOGQNSLXKC5PWFDSTVQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.840165660 (504h52m23.711248695s) [noise=true , address=45.91.116.5:4300 , piece_id=SVNLJADSHVTUU472AVD3RED6FRUJKUUH6OKBWAECZTTHBY3FWHFQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:23.974003410 (504h52m23.845086445s) [noise=true , address=45.91.116.4:4937 , piece_id=4TX3E7FQEIPHWNAZBA35BX3PQTKGC23SPTEBFMHV4QNRGSVLAVGQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:24.126817415 (504h52m23.99790046s) [noise=true , address=45.91.116.10:4081 , piece_id=3JFFCL5NDX7ZJ2LPFUXXGT2LMUA5M7J4FMZGMJHJCSI3K5SWSNDA, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:24.305965214 (504h52m24.177048259s) [noise=true , address=77.90.31.18:4808 , piece_id=XDCM4YDB43OVHWX34PYTIGF5LEVMPUIS6XH2DEGAXC6QHNOJTI3Q, seg_pos={0 0}, upload=443021]: Done uploading piece. err: <nil> Dec 26 16:21:24.305993628 (504h52m24.177076673s) [seg_pos={0 0}, upload=443021]: Segment reached optimal threshold of 13 pieces. Dec 26 16:21:24.306259954 (504h52m24.177342989s) [noise=true , address=145.79.227.29:4941 , piece_id=K65LJVCJQUWV6BQBRYQ3ZMRAILA5SZW7GHW7SJCCZI6FJ3DWBV6A, seg_pos={0 0}, upload=443021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 16:21:24.306267979 (504h52m24.177351024s) [noise=true , address=145.79.227.29:4941 , piece_id=K65LJVCJQUWV6BQBRYQ3ZMRAILA5SZW7GHW7SJCCZI6FJ3DWBV6A, seg_pos={0 0}, upload=443021]: Long tail cancellation due to optimal threshold Dec 26 16:21:24.306388830 (504h52m24.177471875s) [noise=true , address=45.91.116.11:4120 , piece_id=YV7EDQRKSV2WJO6KHUWY7575BCII67DJMSHZSLHWJQ3IGF65AQLQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:44776->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:44776->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44776->45.91.116.11:4120: use of closed network connection Dec 26 16:21:24.306397616 (504h52m24.177480652s) [noise=true , address=45.91.116.11:4120 , piece_id=YV7EDQRKSV2WJO6KHUWY7575BCII67DJMSHZSLHWJQ3IGF65AQLQ, seg_pos={0 0}, upload=443021]: Long tail cancellation due to optimal threshold Dec 26 16:21:24.306435168 (504h52m24.177518203s) [noise=true , address=145.79.227.6:4257 , piece_id=MKXQSFR6SWJPBDREIKDNKUXJ4XUYE7WSQD4HKNB3NUZQJCVUOGIQ, seg_pos={0 0}, upload=443021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:40536->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:40536->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40536->145.79.227.6:4257: use of closed network connection Dec 26 16:21:24.306441169 (504h52m24.177524214s) [noise=true , address=145.79.227.6:4257 , piece_id=MKXQSFR6SWJPBDREIKDNKUXJ4XUYE7WSQD4HKNB3NUZQJCVUOGIQ, seg_pos={0 0}, upload=443021]: Long tail cancellation due to optimal threshold Dec 26 16:21:24.306533075 (504h52m24.17761611s) [seg_pos={0 0}, upload=443021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:21:24.306625471 (504h52m24.177708506s) [upload=443021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:21:24.306630500 (504h52m24.177713535s) [upload=443021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:21:24.608089450 (504h52m24.479172495s) [upload=443021]: Done uploading segments... Dec 26 16:21:24.608113426 (504h52m24.479196471s) [upload=443021]: Upload finished. err: <nil> Duration: 13.478275129s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)