Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.02114042s trace: https://jaeger.dev.storj.io/trace/21f9fb0e12c4fe9 FAIL (2025-12-08 19:50:13 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.021167131s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................X...........X.................] duration: 5.940730952s trace: https://jaeger.dev.storj.io/trace/2c6b8a8384caec89 FAIL (2025-12-08 19:50:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.940760057s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.475928835s trace: https://jaeger.dev.storj.io/trace/3d28e5ace62e0e01 FAIL (2025-12-08 19:50:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== OK2NZ4HXKQHC5MTTRY2YLP7YYS7SBJUU7M365LU56VLVAUJQ, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.293932973 (76h13m38.165016008s) [noise=true , address=81.171.14.70:4030 , piece_id=JFXGK7KJLPI5LJXGCADL5GYQZVKOSJMTO2VYHGIS4NWU44HNORVQ, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.294413788 (76h13m38.165496823s) [noise=true , address=81.171.14.70:4036 , piece_id=ET46ZX2EMELT4OASYNU4WR4DNITLS2VA4FP422OVQXASV6TWU3MA, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.294875628 (76h13m38.165958663s) [noise=true , address=185.167.204.206:4045 , piece_id=Z4IGGM74BYWYDOC5M6C6NK574ZOBBFZ4M7DFTR7LUDR7DKD2LUAA, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.301577459 (76h13m38.172660524s) [noise=true , address=185.167.204.205:4018 , piece_id=4SCEDL7KSFFSBPWAEGKETTTWFDGN5NID4ZCFRVJ6QZ2JTXVHS2XQ, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.304616707 (76h13m38.175699752s) [noise=true , address=185.167.204.205:4022 , piece_id=QVHNV4M6BJIYQSLFFM6PHZ4NO2Z4F7S2UFL3UNOIBVTI6PBLX7UQ, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.307486942 (76h13m38.178569977s) [noise=true , address=185.167.204.206:4066 , piece_id=BONIZ563CJFAW7ADZ6FDZAEMT6CJESN7Z7BMTBCQCO2WE3INL2RQ, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.311116633 (76h13m38.182199668s) [noise=true , address=185.167.204.205:4014 , piece_id=CJZ6K3EONWY3ZBAEFQ2GBBZ5PSYVHOMCP2EFKKERUG4QW27CSN2Q, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.314576141 (76h13m38.185659187s) [noise=true , address=185.167.204.206:4057 , piece_id=X5Y7PZGFIQ5KE6PHQN5YOHUM773U6NKOGF5DPWPG76YXRPMWLMHA, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.318186856 (76h13m38.189269911s) [noise=true , address=81.171.14.70:4033 , piece_id=IBR7PYBXDE3L5UELKVXP5WDVZI72VRJVRWJ7MDOHXHQB24ZUVV3A, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.320152179 (76h13m38.191235214s) [noise=true , address=81.171.14.70:4048 , piece_id=D2N5VBUAMWTK3754BEKSK2OYRIFXNYUCDMOZIG3NJTZG5URN4ULA, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:39.134825843 (76h13m39.005908878s) [upload=66666]: Next returned nil segment Dec 8 19:42:39.134842836 (76h13m39.005925881s) [upload=66666]: Waiting for error group managing segments... Dec 8 19:42:39.519924371 (76h13m39.391007406s) [noise=true , address=185.167.204.205:4029 , piece_id=BAEIL5V3IHBHWYB77MIDBNGX356OXUXAXU4KKP7I7I5SYMC6QKEQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.520407791 (76h13m39.391490826s) [noise=true , address=185.167.204.206:4058 , piece_id=DIDUUF4SQ2XFG5N3T3PAFYDKGF3YDYXYAJHZ4BFFWBIB5J47ZPOA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.520919345 (76h13m39.39200239s) [noise=true , address=185.167.204.206:4059 , piece_id=ZXOANAVMA33NV37MSQUDEU2QVRWQQ3AHAGOQ4J4YOYWDYWLWNVIQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.524350188 (76h13m39.395433233s) [noise=true , address=185.167.204.205:4023 , piece_id=JQGURY62AF5IEARL2K3C3XGW6S57UZO6QR345JGVB2IN46OZNXTQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.528022881 (76h13m39.399105916s) [noise=true , address=185.167.204.206:4060 , piece_id=IPMGIYN5R4UZSNXFUG7KDQL5D6LCJCPLLPJCCERJ52BT7ASMMIHQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.535814137 (76h13m39.406897182s) [noise=true , address=185.167.204.206:4051 , piece_id=DOMDNAWAWOV5NS5UFFMZWYRVKZFXIUEU7ZBTVT5SRKFQLQRGDZKQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.541502700 (76h13m39.412585735s) [noise=true , address=185.167.204.206:4069 , piece_id=6NDEHGDMCACFBJEKOJDWDEGD3CSZDMSXEDQBOPBG5FVLQJMC4QYQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.541942957 (76h13m39.413025993s) [noise=true , address=185.167.204.206:4066 , piece_id=BONIZ563CJFAW7ADZ6FDZAEMT6CJESN7Z7BMTBCQCO2WE3INL2RQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.542708876 (76h13m39.413791961s) [noise=true , address=185.167.204.206:4046 , piece_id=KSPWZJAUZPXIBGHIWXTRXIBOPYF42ZBRWQL2XRCHSUIC35NCUFDQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.543549516 (76h13m39.414632602s) [noise=true , address=185.167.204.205:4016 , piece_id=3E3GOK2NZ4HXKQHC5MTTRY2YLP7YYS7SBJUU7M365LU56VLVAUJQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.546598362 (76h13m39.417681397s) [noise=true , address=185.167.204.206:4045 , piece_id=Z4IGGM74BYWYDOC5M6C6NK574ZOBBFZ4M7DFTR7LUDR7DKD2LUAA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.548463354 (76h13m39.419546399s) [noise=true , address=185.167.204.205:4022 , piece_id=QVHNV4M6BJIYQSLFFM6PHZ4NO2Z4F7S2UFL3UNOIBVTI6PBLX7UQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.549008301 (76h13m39.420091346s) [noise=true , address=185.167.204.206:4047 , piece_id=TWTCTSIARGE2AOCLMF6YO7RLOL7YSZN5D7PDCNAXOX26S6KWC4ZA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.552460625 (76h13m39.42354367s) [noise=true , address=185.167.204.206:4040 , piece_id=VS4ZLJJWZUPUBAIH6CC4W7VHOLMZWYKDWWAOOAIPCG7WP3VJEHYA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.552710752 (76h13m39.423793797s) [noise=true , address=185.167.204.206:4057 , piece_id=X5Y7PZGFIQ5KE6PHQN5YOHUM773U6NKOGF5DPWPG76YXRPMWLMHA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.553051219 (76h13m39.424134264s) [noise=true , address=185.167.204.205:4018 , piece_id=4SCEDL7KSFFSBPWAEGKETTTWFDGN5NID4ZCFRVJ6QZ2JTXVHS2XQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.557215800 (76h13m39.428298835s) [noise=true , address=185.167.204.205:4014 , piece_id=CJZ6K3EONWY3ZBAEFQ2GBBZ5PSYVHOMCP2EFKKERUG4QW27CSN2Q, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.092656704 (76h13m39.963739739s) [noise=true , address=81.171.14.70:4044 , piece_id=NPX7C4PJNRXJFFFWS72F4NTXNWBECFDTIVSNIQEV5N4UAYKV3ZCQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.133122829 (76h13m40.004205874s) [noise=true , address=81.171.14.70:4030 , piece_id=JFXGK7KJLPI5LJXGCADL5GYQZVKOSJMTO2VYHGIS4NWU44HNORVQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.135113711 (76h13m40.006196746s) [noise=true , address=81.171.14.70:4039 , piece_id=MXY6PIXD5CBG2ZDQK3QJZBIXUTJURV57UOZLC7QQCAYZGTJ2UHEQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.157260182 (76h13m40.028343217s) [noise=true , address=81.171.14.70:4048 , piece_id=D2N5VBUAMWTK3754BEKSK2OYRIFXNYUCDMOZIG3NJTZG5URN4ULA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.158303388 (76h13m40.029386423s) [noise=true , address=81.171.14.70:4036 , piece_id=ET46ZX2EMELT4OASYNU4WR4DNITLS2VA4FP422OVQXASV6TWU3MA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.539176087 (76h13m40.410259122s) [noise=true , address=81.171.14.70:4040 , piece_id=YIXCA6XG3O2DB3QM2R2O7IEWKH2JIA6TBOB46EA5SOAYIEBXKVKA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.547733031 (76h13m40.418816076s) [noise=true , address=81.171.14.70:4033 , piece_id=IBR7PYBXDE3L5UELKVXP5WDVZI72VRJVRWJ7MDOHXHQB24ZUVV3A, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.566289897 (76h13m40.437372942s) [noise=true , address=81.171.14.70:4047 , piece_id=Q2RTM6SANOOKVSRN2XGXFJ6LN5GSFR24IAGSU6JO3AOMZYHJZ4DQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.628077722 (76h13m40.499160757s) [noise=true , address=81.171.14.70:4042 , piece_id=7UBOZ3TM6U6KLXKV7VOKZNEY6ABWB5VDKGDV6OSZSAXHEKLRNZ4A, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.628096548 (76h13m40.499179583s) [seg_pos={0 0}, upload=66666]: Segment reached optimal threshold of 26 pieces. Dec 8 19:42:40.628363045 (76h13m40.499446111s) [noise=true , address=89.149.201.147:4017 , piece_id=QLG2MVERXCEFCXBBJBWYZARUNKS6VPOUUSXSUJOGX2G5CCTH6BGQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ): protocol: expected piece hash; context canceled; EOF Dec 8 19:42:40.628373044 (76h13m40.499456079s) [noise=true , address=89.149.201.147:4017 , piece_id=QLG2MVERXCEFCXBBJBWYZARUNKS6VPOUUSXSUJOGX2G5CCTH6BGQ, seg_pos={0 0}, upload=66666]: Long tail cancellation due to optimal threshold Dec 8 19:42:40.628734754 (76h13m40.499817789s) [noise=true , address=89.149.201.147:4001 , piece_id=D65FANROOLI47T6542V4YMZ5HJ6YG5BG6IS2H27KYIR3P4WNH2YQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNB8ZUfm8aR6jWBvWvVx6w7zZgGvbw4kuUCqMHipRWmU8o5R8): protocol: expected piece hash; context canceled; EOF Dec 8 19:42:40.628744101 (76h13m40.499827146s) [noise=true , address=89.149.201.147:4001 , piece_id=D65FANROOLI47T6542V4YMZ5HJ6YG5BG6IS2H27KYIR3P4WNH2YQ, seg_pos={0 0}, upload=66666]: Long tail cancellation due to optimal threshold Dec 8 19:42:40.628767596 (76h13m40.499850641s) [seg_pos={0 0}, upload=66666]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 19:42:40.628886502 (76h13m40.499969547s) [upload=66666]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 19:42:40.628893004 (76h13m40.49997606s) [upload=66666]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 19:48:22.295722109 (76h19m22.166805154s) [upload=66666]: Done uploading segments... Dec 8 19:48:22.295853299 (76h19m22.166936354s) [upload=66666]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.475955044s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................X...........X................] duration: 5.831048916s trace: https://jaeger.dev.storj.io/trace/2c6b8a8384caec89 FAIL (2025-12-08 19:49:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.831078904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.975132491s trace: https://jaeger.dev.storj.io/trace/30f9f47120cf6974 FAIL (2025-12-08 19:49:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 19:10:37.433384005 (75h41m37.30446704s) [upload=66189]: Starting upload Dec 8 19:10:37.433449450 (75h41m37.304532495s) [upload=66189]: Done starting upload Dec 8 19:10:37.433643290 (75h41m37.304726335s) [upload=66189]: Uploading segments... Dec 8 19:10:37.433675882 (75h41m37.304758917s) [upload=66189]: Got next segment. Inline: false Dec 8 19:10:37.433704186 (75h41m37.304787221s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 19:10:37.433708875 (75h41m37.30479191s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 19:10:37.679613305 (75h41m37.55069634s) [seg_pos={0 0}, upload=66189]: Begin upload segment... Dec 8 19:10:37.679683469 (75h41m37.550766514s) [seg_pos={0 0}, upload=66189]: Done begin upload segment. Dec 8 19:10:37.679752580 (75h41m37.550835625s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.679774421 (75h41m37.550857456s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.680422305 (75h41m37.55150534s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681032428 (75h41m37.552115463s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681404696 (75h41m37.552487741s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681922101 (75h41m37.553005136s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682032361 (75h41m37.553115396s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682588229 (75h41m37.553671274s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682965478 (75h41m37.554048523s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683881782 (75h41m37.554964817s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683958880 (75h41m37.555041915s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.684982759 (75h41m37.556065804s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687280685 (75h41m37.5583638s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687699612 (75h41m37.558782658s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688055951 (75h41m37.559139006s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688417931 (75h41m37.559500966s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:38.705641184 (75h41m38.576724219s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 19:10:39.291878765 (75h41m39.16296181s) [upload=66189]: Next returned nil segment Dec 8 19:10:39.291893282 (75h41m39.162976317s) [upload=66189]: Waiting for error group managing segments... Dec 8 19:10:39.671673458 (75h41m39.542756523s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.672368722 (75h41m39.543451767s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.674201522 (75h41m39.545284557s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.679966128 (75h41m39.551049163s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.684661019 (75h41m39.555744054s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.688164300 (75h41m39.559247335s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.689301024 (75h41m39.56038406s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699377182 (75h41m39.570460227s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699944802 (75h41m39.571027837s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.701579305 (75h41m39.57266234s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.702082162 (75h41m39.573165197s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.858397206 (75h41m39.729480251s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919886866 (75h41m39.790969901s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919914609 (75h41m39.790997644s) [seg_pos={0 0}, upload=66189]: Segment reached optimal threshold of 13 pieces. Dec 8 19:10:39.920157301 (75h41m39.791240346s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection Dec 8 19:10:39.920166508 (75h41m39.791249553s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920380536 (75h41m39.791463571s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 19:10:39.920389733 (75h41m39.791472788s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920416865 (75h41m39.79149991s) [seg_pos={0 0}, upload=66189]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 19:10:39.920506556 (75h41m39.791589601s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 19:10:39.920513880 (75h41m39.791596925s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 19:10:40.218384935 (75h41m40.08946797s) [upload=66189]: Done uploading segments... Dec 8 19:10:40.218431954 (75h41m40.089514999s) [upload=66189]: Upload finished. err: <nil> Duration: 5.975162737s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.819761825s trace: https://jaeger.dev.storj.io/trace/6bf00a8c3609c4c7 FAIL (2025-12-08 19:49:42 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.819792895s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.238122379s trace: https://jaeger.dev.storj.io/trace/3d28e5ace62e0e01 FAIL (2025-12-08 19:49:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== OK2NZ4HXKQHC5MTTRY2YLP7YYS7SBJUU7M365LU56VLVAUJQ, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.293932973 (76h13m38.165016008s) [noise=true , address=81.171.14.70:4030 , piece_id=JFXGK7KJLPI5LJXGCADL5GYQZVKOSJMTO2VYHGIS4NWU44HNORVQ, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.294413788 (76h13m38.165496823s) [noise=true , address=81.171.14.70:4036 , piece_id=ET46ZX2EMELT4OASYNU4WR4DNITLS2VA4FP422OVQXASV6TWU3MA, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.294875628 (76h13m38.165958663s) [noise=true , address=185.167.204.206:4045 , piece_id=Z4IGGM74BYWYDOC5M6C6NK574ZOBBFZ4M7DFTR7LUDR7DKD2LUAA, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.301577459 (76h13m38.172660524s) [noise=true , address=185.167.204.205:4018 , piece_id=4SCEDL7KSFFSBPWAEGKETTTWFDGN5NID4ZCFRVJ6QZ2JTXVHS2XQ, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.304616707 (76h13m38.175699752s) [noise=true , address=185.167.204.205:4022 , piece_id=QVHNV4M6BJIYQSLFFM6PHZ4NO2Z4F7S2UFL3UNOIBVTI6PBLX7UQ, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.307486942 (76h13m38.178569977s) [noise=true , address=185.167.204.206:4066 , piece_id=BONIZ563CJFAW7ADZ6FDZAEMT6CJESN7Z7BMTBCQCO2WE3INL2RQ, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.311116633 (76h13m38.182199668s) [noise=true , address=185.167.204.205:4014 , piece_id=CJZ6K3EONWY3ZBAEFQ2GBBZ5PSYVHOMCP2EFKKERUG4QW27CSN2Q, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.314576141 (76h13m38.185659187s) [noise=true , address=185.167.204.206:4057 , piece_id=X5Y7PZGFIQ5KE6PHQN5YOHUM773U6NKOGF5DPWPG76YXRPMWLMHA, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.318186856 (76h13m38.189269911s) [noise=true , address=81.171.14.70:4033 , piece_id=IBR7PYBXDE3L5UELKVXP5WDVZI72VRJVRWJ7MDOHXHQB24ZUVV3A, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.320152179 (76h13m38.191235214s) [noise=true , address=81.171.14.70:4048 , piece_id=D2N5VBUAMWTK3754BEKSK2OYRIFXNYUCDMOZIG3NJTZG5URN4ULA, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:39.134825843 (76h13m39.005908878s) [upload=66666]: Next returned nil segment Dec 8 19:42:39.134842836 (76h13m39.005925881s) [upload=66666]: Waiting for error group managing segments... Dec 8 19:42:39.519924371 (76h13m39.391007406s) [noise=true , address=185.167.204.205:4029 , piece_id=BAEIL5V3IHBHWYB77MIDBNGX356OXUXAXU4KKP7I7I5SYMC6QKEQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.520407791 (76h13m39.391490826s) [noise=true , address=185.167.204.206:4058 , piece_id=DIDUUF4SQ2XFG5N3T3PAFYDKGF3YDYXYAJHZ4BFFWBIB5J47ZPOA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.520919345 (76h13m39.39200239s) [noise=true , address=185.167.204.206:4059 , piece_id=ZXOANAVMA33NV37MSQUDEU2QVRWQQ3AHAGOQ4J4YOYWDYWLWNVIQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.524350188 (76h13m39.395433233s) [noise=true , address=185.167.204.205:4023 , piece_id=JQGURY62AF5IEARL2K3C3XGW6S57UZO6QR345JGVB2IN46OZNXTQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.528022881 (76h13m39.399105916s) [noise=true , address=185.167.204.206:4060 , piece_id=IPMGIYN5R4UZSNXFUG7KDQL5D6LCJCPLLPJCCERJ52BT7ASMMIHQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.535814137 (76h13m39.406897182s) [noise=true , address=185.167.204.206:4051 , piece_id=DOMDNAWAWOV5NS5UFFMZWYRVKZFXIUEU7ZBTVT5SRKFQLQRGDZKQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.541502700 (76h13m39.412585735s) [noise=true , address=185.167.204.206:4069 , piece_id=6NDEHGDMCACFBJEKOJDWDEGD3CSZDMSXEDQBOPBG5FVLQJMC4QYQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.541942957 (76h13m39.413025993s) [noise=true , address=185.167.204.206:4066 , piece_id=BONIZ563CJFAW7ADZ6FDZAEMT6CJESN7Z7BMTBCQCO2WE3INL2RQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.542708876 (76h13m39.413791961s) [noise=true , address=185.167.204.206:4046 , piece_id=KSPWZJAUZPXIBGHIWXTRXIBOPYF42ZBRWQL2XRCHSUIC35NCUFDQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.543549516 (76h13m39.414632602s) [noise=true , address=185.167.204.205:4016 , piece_id=3E3GOK2NZ4HXKQHC5MTTRY2YLP7YYS7SBJUU7M365LU56VLVAUJQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.546598362 (76h13m39.417681397s) [noise=true , address=185.167.204.206:4045 , piece_id=Z4IGGM74BYWYDOC5M6C6NK574ZOBBFZ4M7DFTR7LUDR7DKD2LUAA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.548463354 (76h13m39.419546399s) [noise=true , address=185.167.204.205:4022 , piece_id=QVHNV4M6BJIYQSLFFM6PHZ4NO2Z4F7S2UFL3UNOIBVTI6PBLX7UQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.549008301 (76h13m39.420091346s) [noise=true , address=185.167.204.206:4047 , piece_id=TWTCTSIARGE2AOCLMF6YO7RLOL7YSZN5D7PDCNAXOX26S6KWC4ZA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.552460625 (76h13m39.42354367s) [noise=true , address=185.167.204.206:4040 , piece_id=VS4ZLJJWZUPUBAIH6CC4W7VHOLMZWYKDWWAOOAIPCG7WP3VJEHYA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.552710752 (76h13m39.423793797s) [noise=true , address=185.167.204.206:4057 , piece_id=X5Y7PZGFIQ5KE6PHQN5YOHUM773U6NKOGF5DPWPG76YXRPMWLMHA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.553051219 (76h13m39.424134264s) [noise=true , address=185.167.204.205:4018 , piece_id=4SCEDL7KSFFSBPWAEGKETTTWFDGN5NID4ZCFRVJ6QZ2JTXVHS2XQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.557215800 (76h13m39.428298835s) [noise=true , address=185.167.204.205:4014 , piece_id=CJZ6K3EONWY3ZBAEFQ2GBBZ5PSYVHOMCP2EFKKERUG4QW27CSN2Q, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.092656704 (76h13m39.963739739s) [noise=true , address=81.171.14.70:4044 , piece_id=NPX7C4PJNRXJFFFWS72F4NTXNWBECFDTIVSNIQEV5N4UAYKV3ZCQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.133122829 (76h13m40.004205874s) [noise=true , address=81.171.14.70:4030 , piece_id=JFXGK7KJLPI5LJXGCADL5GYQZVKOSJMTO2VYHGIS4NWU44HNORVQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.135113711 (76h13m40.006196746s) [noise=true , address=81.171.14.70:4039 , piece_id=MXY6PIXD5CBG2ZDQK3QJZBIXUTJURV57UOZLC7QQCAYZGTJ2UHEQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.157260182 (76h13m40.028343217s) [noise=true , address=81.171.14.70:4048 , piece_id=D2N5VBUAMWTK3754BEKSK2OYRIFXNYUCDMOZIG3NJTZG5URN4ULA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.158303388 (76h13m40.029386423s) [noise=true , address=81.171.14.70:4036 , piece_id=ET46ZX2EMELT4OASYNU4WR4DNITLS2VA4FP422OVQXASV6TWU3MA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.539176087 (76h13m40.410259122s) [noise=true , address=81.171.14.70:4040 , piece_id=YIXCA6XG3O2DB3QM2R2O7IEWKH2JIA6TBOB46EA5SOAYIEBXKVKA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.547733031 (76h13m40.418816076s) [noise=true , address=81.171.14.70:4033 , piece_id=IBR7PYBXDE3L5UELKVXP5WDVZI72VRJVRWJ7MDOHXHQB24ZUVV3A, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.566289897 (76h13m40.437372942s) [noise=true , address=81.171.14.70:4047 , piece_id=Q2RTM6SANOOKVSRN2XGXFJ6LN5GSFR24IAGSU6JO3AOMZYHJZ4DQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.628077722 (76h13m40.499160757s) [noise=true , address=81.171.14.70:4042 , piece_id=7UBOZ3TM6U6KLXKV7VOKZNEY6ABWB5VDKGDV6OSZSAXHEKLRNZ4A, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.628096548 (76h13m40.499179583s) [seg_pos={0 0}, upload=66666]: Segment reached optimal threshold of 26 pieces. Dec 8 19:42:40.628363045 (76h13m40.499446111s) [noise=true , address=89.149.201.147:4017 , piece_id=QLG2MVERXCEFCXBBJBWYZARUNKS6VPOUUSXSUJOGX2G5CCTH6BGQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ): protocol: expected piece hash; context canceled; EOF Dec 8 19:42:40.628373044 (76h13m40.499456079s) [noise=true , address=89.149.201.147:4017 , piece_id=QLG2MVERXCEFCXBBJBWYZARUNKS6VPOUUSXSUJOGX2G5CCTH6BGQ, seg_pos={0 0}, upload=66666]: Long tail cancellation due to optimal threshold Dec 8 19:42:40.628734754 (76h13m40.499817789s) [noise=true , address=89.149.201.147:4001 , piece_id=D65FANROOLI47T6542V4YMZ5HJ6YG5BG6IS2H27KYIR3P4WNH2YQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNB8ZUfm8aR6jWBvWvVx6w7zZgGvbw4kuUCqMHipRWmU8o5R8): protocol: expected piece hash; context canceled; EOF Dec 8 19:42:40.628744101 (76h13m40.499827146s) [noise=true , address=89.149.201.147:4001 , piece_id=D65FANROOLI47T6542V4YMZ5HJ6YG5BG6IS2H27KYIR3P4WNH2YQ, seg_pos={0 0}, upload=66666]: Long tail cancellation due to optimal threshold Dec 8 19:42:40.628767596 (76h13m40.499850641s) [seg_pos={0 0}, upload=66666]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 19:42:40.628886502 (76h13m40.499969547s) [upload=66666]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 19:42:40.628893004 (76h13m40.49997606s) [upload=66666]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 19:48:22.295722109 (76h19m22.166805154s) [upload=66666]: Done uploading segments... Dec 8 19:48:22.295853299 (76h19m22.166936354s) [upload=66666]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.238148308s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 3.394295163s trace: https://jaeger.dev.storj.io/trace/624d2b5b2aee8843 FAIL (2025-12-08 19:49:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.394341902s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.014651102s trace: https://jaeger.dev.storj.io/trace/21f9fb0e12c4fe9 FAIL (2025-12-08 19:49:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.01467617s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 476.90904ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/5e91473084188cf7 FAIL (2025-12-08 19:49:19 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 476.938396ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.429914439s trace: https://jaeger.dev.storj.io/trace/624d2b5b2aee8843 FAIL (2025-12-08 19:49:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.429943215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.919019648s trace: https://jaeger.dev.storj.io/trace/6bf00a8c3609c4c7 FAIL (2025-12-08 19:49:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.919048972s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.795337544s trace: https://jaeger.dev.storj.io/trace/30f9f47120cf6974 FAIL (2025-12-08 19:48:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 19:10:37.433384005 (75h41m37.30446704s) [upload=66189]: Starting upload Dec 8 19:10:37.433449450 (75h41m37.304532495s) [upload=66189]: Done starting upload Dec 8 19:10:37.433643290 (75h41m37.304726335s) [upload=66189]: Uploading segments... Dec 8 19:10:37.433675882 (75h41m37.304758917s) [upload=66189]: Got next segment. Inline: false Dec 8 19:10:37.433704186 (75h41m37.304787221s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 19:10:37.433708875 (75h41m37.30479191s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 19:10:37.679613305 (75h41m37.55069634s) [seg_pos={0 0}, upload=66189]: Begin upload segment... Dec 8 19:10:37.679683469 (75h41m37.550766514s) [seg_pos={0 0}, upload=66189]: Done begin upload segment. Dec 8 19:10:37.679752580 (75h41m37.550835625s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.679774421 (75h41m37.550857456s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.680422305 (75h41m37.55150534s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681032428 (75h41m37.552115463s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681404696 (75h41m37.552487741s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681922101 (75h41m37.553005136s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682032361 (75h41m37.553115396s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682588229 (75h41m37.553671274s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682965478 (75h41m37.554048523s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683881782 (75h41m37.554964817s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683958880 (75h41m37.555041915s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.684982759 (75h41m37.556065804s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687280685 (75h41m37.5583638s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687699612 (75h41m37.558782658s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688055951 (75h41m37.559139006s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688417931 (75h41m37.559500966s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:38.705641184 (75h41m38.576724219s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 19:10:39.291878765 (75h41m39.16296181s) [upload=66189]: Next returned nil segment Dec 8 19:10:39.291893282 (75h41m39.162976317s) [upload=66189]: Waiting for error group managing segments... Dec 8 19:10:39.671673458 (75h41m39.542756523s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.672368722 (75h41m39.543451767s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.674201522 (75h41m39.545284557s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.679966128 (75h41m39.551049163s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.684661019 (75h41m39.555744054s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.688164300 (75h41m39.559247335s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.689301024 (75h41m39.56038406s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699377182 (75h41m39.570460227s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699944802 (75h41m39.571027837s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.701579305 (75h41m39.57266234s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.702082162 (75h41m39.573165197s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.858397206 (75h41m39.729480251s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919886866 (75h41m39.790969901s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919914609 (75h41m39.790997644s) [seg_pos={0 0}, upload=66189]: Segment reached optimal threshold of 13 pieces. Dec 8 19:10:39.920157301 (75h41m39.791240346s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection Dec 8 19:10:39.920166508 (75h41m39.791249553s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920380536 (75h41m39.791463571s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 19:10:39.920389733 (75h41m39.791472788s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920416865 (75h41m39.79149991s) [seg_pos={0 0}, upload=66189]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 19:10:39.920506556 (75h41m39.791589601s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 19:10:39.920513880 (75h41m39.791596925s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 19:10:40.218384935 (75h41m40.08946797s) [upload=66189]: Done uploading segments... Dec 8 19:10:40.218431954 (75h41m40.089514999s) [upload=66189]: Upload finished. err: <nil> Duration: 5.795367351s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................X...........X...............] duration: 5.570065225s trace: https://jaeger.dev.storj.io/trace/2c6b8a8384caec89 FAIL (2025-12-08 19:48:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.570090394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.019565012s trace: https://jaeger.dev.storj.io/trace/21f9fb0e12c4fe9 FAIL (2025-12-08 19:48:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.019595581s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.871901256s trace: https://jaeger.dev.storj.io/trace/3d28e5ace62e0e01 FAIL (2025-12-08 19:48:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== OK2NZ4HXKQHC5MTTRY2YLP7YYS7SBJUU7M365LU56VLVAUJQ, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.293932973 (76h13m38.165016008s) [noise=true , address=81.171.14.70:4030 , piece_id=JFXGK7KJLPI5LJXGCADL5GYQZVKOSJMTO2VYHGIS4NWU44HNORVQ, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.294413788 (76h13m38.165496823s) [noise=true , address=81.171.14.70:4036 , piece_id=ET46ZX2EMELT4OASYNU4WR4DNITLS2VA4FP422OVQXASV6TWU3MA, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.294875628 (76h13m38.165958663s) [noise=true , address=185.167.204.206:4045 , piece_id=Z4IGGM74BYWYDOC5M6C6NK574ZOBBFZ4M7DFTR7LUDR7DKD2LUAA, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.301577459 (76h13m38.172660524s) [noise=true , address=185.167.204.205:4018 , piece_id=4SCEDL7KSFFSBPWAEGKETTTWFDGN5NID4ZCFRVJ6QZ2JTXVHS2XQ, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.304616707 (76h13m38.175699752s) [noise=true , address=185.167.204.205:4022 , piece_id=QVHNV4M6BJIYQSLFFM6PHZ4NO2Z4F7S2UFL3UNOIBVTI6PBLX7UQ, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.307486942 (76h13m38.178569977s) [noise=true , address=185.167.204.206:4066 , piece_id=BONIZ563CJFAW7ADZ6FDZAEMT6CJESN7Z7BMTBCQCO2WE3INL2RQ, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.311116633 (76h13m38.182199668s) [noise=true , address=185.167.204.205:4014 , piece_id=CJZ6K3EONWY3ZBAEFQ2GBBZ5PSYVHOMCP2EFKKERUG4QW27CSN2Q, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.314576141 (76h13m38.185659187s) [noise=true , address=185.167.204.206:4057 , piece_id=X5Y7PZGFIQ5KE6PHQN5YOHUM773U6NKOGF5DPWPG76YXRPMWLMHA, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.318186856 (76h13m38.189269911s) [noise=true , address=81.171.14.70:4033 , piece_id=IBR7PYBXDE3L5UELKVXP5WDVZI72VRJVRWJ7MDOHXHQB24ZUVV3A, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.320152179 (76h13m38.191235214s) [noise=true , address=81.171.14.70:4048 , piece_id=D2N5VBUAMWTK3754BEKSK2OYRIFXNYUCDMOZIG3NJTZG5URN4ULA, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:39.134825843 (76h13m39.005908878s) [upload=66666]: Next returned nil segment Dec 8 19:42:39.134842836 (76h13m39.005925881s) [upload=66666]: Waiting for error group managing segments... Dec 8 19:42:39.519924371 (76h13m39.391007406s) [noise=true , address=185.167.204.205:4029 , piece_id=BAEIL5V3IHBHWYB77MIDBNGX356OXUXAXU4KKP7I7I5SYMC6QKEQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.520407791 (76h13m39.391490826s) [noise=true , address=185.167.204.206:4058 , piece_id=DIDUUF4SQ2XFG5N3T3PAFYDKGF3YDYXYAJHZ4BFFWBIB5J47ZPOA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.520919345 (76h13m39.39200239s) [noise=true , address=185.167.204.206:4059 , piece_id=ZXOANAVMA33NV37MSQUDEU2QVRWQQ3AHAGOQ4J4YOYWDYWLWNVIQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.524350188 (76h13m39.395433233s) [noise=true , address=185.167.204.205:4023 , piece_id=JQGURY62AF5IEARL2K3C3XGW6S57UZO6QR345JGVB2IN46OZNXTQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.528022881 (76h13m39.399105916s) [noise=true , address=185.167.204.206:4060 , piece_id=IPMGIYN5R4UZSNXFUG7KDQL5D6LCJCPLLPJCCERJ52BT7ASMMIHQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.535814137 (76h13m39.406897182s) [noise=true , address=185.167.204.206:4051 , piece_id=DOMDNAWAWOV5NS5UFFMZWYRVKZFXIUEU7ZBTVT5SRKFQLQRGDZKQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.541502700 (76h13m39.412585735s) [noise=true , address=185.167.204.206:4069 , piece_id=6NDEHGDMCACFBJEKOJDWDEGD3CSZDMSXEDQBOPBG5FVLQJMC4QYQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.541942957 (76h13m39.413025993s) [noise=true , address=185.167.204.206:4066 , piece_id=BONIZ563CJFAW7ADZ6FDZAEMT6CJESN7Z7BMTBCQCO2WE3INL2RQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.542708876 (76h13m39.413791961s) [noise=true , address=185.167.204.206:4046 , piece_id=KSPWZJAUZPXIBGHIWXTRXIBOPYF42ZBRWQL2XRCHSUIC35NCUFDQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.543549516 (76h13m39.414632602s) [noise=true , address=185.167.204.205:4016 , piece_id=3E3GOK2NZ4HXKQHC5MTTRY2YLP7YYS7SBJUU7M365LU56VLVAUJQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.546598362 (76h13m39.417681397s) [noise=true , address=185.167.204.206:4045 , piece_id=Z4IGGM74BYWYDOC5M6C6NK574ZOBBFZ4M7DFTR7LUDR7DKD2LUAA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.548463354 (76h13m39.419546399s) [noise=true , address=185.167.204.205:4022 , piece_id=QVHNV4M6BJIYQSLFFM6PHZ4NO2Z4F7S2UFL3UNOIBVTI6PBLX7UQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.549008301 (76h13m39.420091346s) [noise=true , address=185.167.204.206:4047 , piece_id=TWTCTSIARGE2AOCLMF6YO7RLOL7YSZN5D7PDCNAXOX26S6KWC4ZA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.552460625 (76h13m39.42354367s) [noise=true , address=185.167.204.206:4040 , piece_id=VS4ZLJJWZUPUBAIH6CC4W7VHOLMZWYKDWWAOOAIPCG7WP3VJEHYA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.552710752 (76h13m39.423793797s) [noise=true , address=185.167.204.206:4057 , piece_id=X5Y7PZGFIQ5KE6PHQN5YOHUM773U6NKOGF5DPWPG76YXRPMWLMHA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.553051219 (76h13m39.424134264s) [noise=true , address=185.167.204.205:4018 , piece_id=4SCEDL7KSFFSBPWAEGKETTTWFDGN5NID4ZCFRVJ6QZ2JTXVHS2XQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.557215800 (76h13m39.428298835s) [noise=true , address=185.167.204.205:4014 , piece_id=CJZ6K3EONWY3ZBAEFQ2GBBZ5PSYVHOMCP2EFKKERUG4QW27CSN2Q, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.092656704 (76h13m39.963739739s) [noise=true , address=81.171.14.70:4044 , piece_id=NPX7C4PJNRXJFFFWS72F4NTXNWBECFDTIVSNIQEV5N4UAYKV3ZCQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.133122829 (76h13m40.004205874s) [noise=true , address=81.171.14.70:4030 , piece_id=JFXGK7KJLPI5LJXGCADL5GYQZVKOSJMTO2VYHGIS4NWU44HNORVQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.135113711 (76h13m40.006196746s) [noise=true , address=81.171.14.70:4039 , piece_id=MXY6PIXD5CBG2ZDQK3QJZBIXUTJURV57UOZLC7QQCAYZGTJ2UHEQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.157260182 (76h13m40.028343217s) [noise=true , address=81.171.14.70:4048 , piece_id=D2N5VBUAMWTK3754BEKSK2OYRIFXNYUCDMOZIG3NJTZG5URN4ULA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.158303388 (76h13m40.029386423s) [noise=true , address=81.171.14.70:4036 , piece_id=ET46ZX2EMELT4OASYNU4WR4DNITLS2VA4FP422OVQXASV6TWU3MA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.539176087 (76h13m40.410259122s) [noise=true , address=81.171.14.70:4040 , piece_id=YIXCA6XG3O2DB3QM2R2O7IEWKH2JIA6TBOB46EA5SOAYIEBXKVKA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.547733031 (76h13m40.418816076s) [noise=true , address=81.171.14.70:4033 , piece_id=IBR7PYBXDE3L5UELKVXP5WDVZI72VRJVRWJ7MDOHXHQB24ZUVV3A, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.566289897 (76h13m40.437372942s) [noise=true , address=81.171.14.70:4047 , piece_id=Q2RTM6SANOOKVSRN2XGXFJ6LN5GSFR24IAGSU6JO3AOMZYHJZ4DQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.628077722 (76h13m40.499160757s) [noise=true , address=81.171.14.70:4042 , piece_id=7UBOZ3TM6U6KLXKV7VOKZNEY6ABWB5VDKGDV6OSZSAXHEKLRNZ4A, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.628096548 (76h13m40.499179583s) [seg_pos={0 0}, upload=66666]: Segment reached optimal threshold of 26 pieces. Dec 8 19:42:40.628363045 (76h13m40.499446111s) [noise=true , address=89.149.201.147:4017 , piece_id=QLG2MVERXCEFCXBBJBWYZARUNKS6VPOUUSXSUJOGX2G5CCTH6BGQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ): protocol: expected piece hash; context canceled; EOF Dec 8 19:42:40.628373044 (76h13m40.499456079s) [noise=true , address=89.149.201.147:4017 , piece_id=QLG2MVERXCEFCXBBJBWYZARUNKS6VPOUUSXSUJOGX2G5CCTH6BGQ, seg_pos={0 0}, upload=66666]: Long tail cancellation due to optimal threshold Dec 8 19:42:40.628734754 (76h13m40.499817789s) [noise=true , address=89.149.201.147:4001 , piece_id=D65FANROOLI47T6542V4YMZ5HJ6YG5BG6IS2H27KYIR3P4WNH2YQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNB8ZUfm8aR6jWBvWvVx6w7zZgGvbw4kuUCqMHipRWmU8o5R8): protocol: expected piece hash; context canceled; EOF Dec 8 19:42:40.628744101 (76h13m40.499827146s) [noise=true , address=89.149.201.147:4001 , piece_id=D65FANROOLI47T6542V4YMZ5HJ6YG5BG6IS2H27KYIR3P4WNH2YQ, seg_pos={0 0}, upload=66666]: Long tail cancellation due to optimal threshold Dec 8 19:42:40.628767596 (76h13m40.499850641s) [seg_pos={0 0}, upload=66666]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 19:42:40.628886502 (76h13m40.499969547s) [upload=66666]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 19:42:40.628893004 (76h13m40.49997606s) [upload=66666]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 19:48:22.295722109 (76h19m22.166805154s) [upload=66666]: Done uploading segments... Dec 8 19:48:22.295853299 (76h19m22.166936354s) [upload=66666]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.871918068s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 5m45.743381809s trace: https://jaeger.dev.storj.io/trace/3d28e5ace62e0e01 FAIL (2025-12-08 19:48:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== OK2NZ4HXKQHC5MTTRY2YLP7YYS7SBJUU7M365LU56VLVAUJQ, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.293932973 (76h13m38.165016008s) [noise=true , address=81.171.14.70:4030 , piece_id=JFXGK7KJLPI5LJXGCADL5GYQZVKOSJMTO2VYHGIS4NWU44HNORVQ, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.294413788 (76h13m38.165496823s) [noise=true , address=81.171.14.70:4036 , piece_id=ET46ZX2EMELT4OASYNU4WR4DNITLS2VA4FP422OVQXASV6TWU3MA, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.294875628 (76h13m38.165958663s) [noise=true , address=185.167.204.206:4045 , piece_id=Z4IGGM74BYWYDOC5M6C6NK574ZOBBFZ4M7DFTR7LUDR7DKD2LUAA, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.301577459 (76h13m38.172660524s) [noise=true , address=185.167.204.205:4018 , piece_id=4SCEDL7KSFFSBPWAEGKETTTWFDGN5NID4ZCFRVJ6QZ2JTXVHS2XQ, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.304616707 (76h13m38.175699752s) [noise=true , address=185.167.204.205:4022 , piece_id=QVHNV4M6BJIYQSLFFM6PHZ4NO2Z4F7S2UFL3UNOIBVTI6PBLX7UQ, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.307486942 (76h13m38.178569977s) [noise=true , address=185.167.204.206:4066 , piece_id=BONIZ563CJFAW7ADZ6FDZAEMT6CJESN7Z7BMTBCQCO2WE3INL2RQ, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.311116633 (76h13m38.182199668s) [noise=true , address=185.167.204.205:4014 , piece_id=CJZ6K3EONWY3ZBAEFQ2GBBZ5PSYVHOMCP2EFKKERUG4QW27CSN2Q, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.314576141 (76h13m38.185659187s) [noise=true , address=185.167.204.206:4057 , piece_id=X5Y7PZGFIQ5KE6PHQN5YOHUM773U6NKOGF5DPWPG76YXRPMWLMHA, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.318186856 (76h13m38.189269911s) [noise=true , address=81.171.14.70:4033 , piece_id=IBR7PYBXDE3L5UELKVXP5WDVZI72VRJVRWJ7MDOHXHQB24ZUVV3A, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:38.320152179 (76h13m38.191235214s) [noise=true , address=81.171.14.70:4048 , piece_id=D2N5VBUAMWTK3754BEKSK2OYRIFXNYUCDMOZIG3NJTZG5URN4ULA, seg_pos={0 0}, upload=66666]: Uploading piece... Dec 8 19:42:39.134825843 (76h13m39.005908878s) [upload=66666]: Next returned nil segment Dec 8 19:42:39.134842836 (76h13m39.005925881s) [upload=66666]: Waiting for error group managing segments... Dec 8 19:42:39.519924371 (76h13m39.391007406s) [noise=true , address=185.167.204.205:4029 , piece_id=BAEIL5V3IHBHWYB77MIDBNGX356OXUXAXU4KKP7I7I5SYMC6QKEQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.520407791 (76h13m39.391490826s) [noise=true , address=185.167.204.206:4058 , piece_id=DIDUUF4SQ2XFG5N3T3PAFYDKGF3YDYXYAJHZ4BFFWBIB5J47ZPOA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.520919345 (76h13m39.39200239s) [noise=true , address=185.167.204.206:4059 , piece_id=ZXOANAVMA33NV37MSQUDEU2QVRWQQ3AHAGOQ4J4YOYWDYWLWNVIQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.524350188 (76h13m39.395433233s) [noise=true , address=185.167.204.205:4023 , piece_id=JQGURY62AF5IEARL2K3C3XGW6S57UZO6QR345JGVB2IN46OZNXTQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.528022881 (76h13m39.399105916s) [noise=true , address=185.167.204.206:4060 , piece_id=IPMGIYN5R4UZSNXFUG7KDQL5D6LCJCPLLPJCCERJ52BT7ASMMIHQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.535814137 (76h13m39.406897182s) [noise=true , address=185.167.204.206:4051 , piece_id=DOMDNAWAWOV5NS5UFFMZWYRVKZFXIUEU7ZBTVT5SRKFQLQRGDZKQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.541502700 (76h13m39.412585735s) [noise=true , address=185.167.204.206:4069 , piece_id=6NDEHGDMCACFBJEKOJDWDEGD3CSZDMSXEDQBOPBG5FVLQJMC4QYQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.541942957 (76h13m39.413025993s) [noise=true , address=185.167.204.206:4066 , piece_id=BONIZ563CJFAW7ADZ6FDZAEMT6CJESN7Z7BMTBCQCO2WE3INL2RQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.542708876 (76h13m39.413791961s) [noise=true , address=185.167.204.206:4046 , piece_id=KSPWZJAUZPXIBGHIWXTRXIBOPYF42ZBRWQL2XRCHSUIC35NCUFDQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.543549516 (76h13m39.414632602s) [noise=true , address=185.167.204.205:4016 , piece_id=3E3GOK2NZ4HXKQHC5MTTRY2YLP7YYS7SBJUU7M365LU56VLVAUJQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.546598362 (76h13m39.417681397s) [noise=true , address=185.167.204.206:4045 , piece_id=Z4IGGM74BYWYDOC5M6C6NK574ZOBBFZ4M7DFTR7LUDR7DKD2LUAA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.548463354 (76h13m39.419546399s) [noise=true , address=185.167.204.205:4022 , piece_id=QVHNV4M6BJIYQSLFFM6PHZ4NO2Z4F7S2UFL3UNOIBVTI6PBLX7UQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.549008301 (76h13m39.420091346s) [noise=true , address=185.167.204.206:4047 , piece_id=TWTCTSIARGE2AOCLMF6YO7RLOL7YSZN5D7PDCNAXOX26S6KWC4ZA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.552460625 (76h13m39.42354367s) [noise=true , address=185.167.204.206:4040 , piece_id=VS4ZLJJWZUPUBAIH6CC4W7VHOLMZWYKDWWAOOAIPCG7WP3VJEHYA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.552710752 (76h13m39.423793797s) [noise=true , address=185.167.204.206:4057 , piece_id=X5Y7PZGFIQ5KE6PHQN5YOHUM773U6NKOGF5DPWPG76YXRPMWLMHA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.553051219 (76h13m39.424134264s) [noise=true , address=185.167.204.205:4018 , piece_id=4SCEDL7KSFFSBPWAEGKETTTWFDGN5NID4ZCFRVJ6QZ2JTXVHS2XQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:39.557215800 (76h13m39.428298835s) [noise=true , address=185.167.204.205:4014 , piece_id=CJZ6K3EONWY3ZBAEFQ2GBBZ5PSYVHOMCP2EFKKERUG4QW27CSN2Q, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.092656704 (76h13m39.963739739s) [noise=true , address=81.171.14.70:4044 , piece_id=NPX7C4PJNRXJFFFWS72F4NTXNWBECFDTIVSNIQEV5N4UAYKV3ZCQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.133122829 (76h13m40.004205874s) [noise=true , address=81.171.14.70:4030 , piece_id=JFXGK7KJLPI5LJXGCADL5GYQZVKOSJMTO2VYHGIS4NWU44HNORVQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.135113711 (76h13m40.006196746s) [noise=true , address=81.171.14.70:4039 , piece_id=MXY6PIXD5CBG2ZDQK3QJZBIXUTJURV57UOZLC7QQCAYZGTJ2UHEQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.157260182 (76h13m40.028343217s) [noise=true , address=81.171.14.70:4048 , piece_id=D2N5VBUAMWTK3754BEKSK2OYRIFXNYUCDMOZIG3NJTZG5URN4ULA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.158303388 (76h13m40.029386423s) [noise=true , address=81.171.14.70:4036 , piece_id=ET46ZX2EMELT4OASYNU4WR4DNITLS2VA4FP422OVQXASV6TWU3MA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.539176087 (76h13m40.410259122s) [noise=true , address=81.171.14.70:4040 , piece_id=YIXCA6XG3O2DB3QM2R2O7IEWKH2JIA6TBOB46EA5SOAYIEBXKVKA, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.547733031 (76h13m40.418816076s) [noise=true , address=81.171.14.70:4033 , piece_id=IBR7PYBXDE3L5UELKVXP5WDVZI72VRJVRWJ7MDOHXHQB24ZUVV3A, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.566289897 (76h13m40.437372942s) [noise=true , address=81.171.14.70:4047 , piece_id=Q2RTM6SANOOKVSRN2XGXFJ6LN5GSFR24IAGSU6JO3AOMZYHJZ4DQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.628077722 (76h13m40.499160757s) [noise=true , address=81.171.14.70:4042 , piece_id=7UBOZ3TM6U6KLXKV7VOKZNEY6ABWB5VDKGDV6OSZSAXHEKLRNZ4A, seg_pos={0 0}, upload=66666]: Done uploading piece. err: <nil> Dec 8 19:42:40.628096548 (76h13m40.499179583s) [seg_pos={0 0}, upload=66666]: Segment reached optimal threshold of 26 pieces. Dec 8 19:42:40.628363045 (76h13m40.499446111s) [noise=true , address=89.149.201.147:4017 , piece_id=QLG2MVERXCEFCXBBJBWYZARUNKS6VPOUUSXSUJOGX2G5CCTH6BGQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1foiaFSv2TTCkBBCnVYJ6xRbAJ2YYkhnKKmtZJY31ZAzFMTWQQ): protocol: expected piece hash; context canceled; EOF Dec 8 19:42:40.628373044 (76h13m40.499456079s) [noise=true , address=89.149.201.147:4017 , piece_id=QLG2MVERXCEFCXBBJBWYZARUNKS6VPOUUSXSUJOGX2G5CCTH6BGQ, seg_pos={0 0}, upload=66666]: Long tail cancellation due to optimal threshold Dec 8 19:42:40.628734754 (76h13m40.499817789s) [noise=true , address=89.149.201.147:4001 , piece_id=D65FANROOLI47T6542V4YMZ5HJ6YG5BG6IS2H27KYIR3P4WNH2YQ, seg_pos={0 0}, upload=66666]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sNB8ZUfm8aR6jWBvWvVx6w7zZgGvbw4kuUCqMHipRWmU8o5R8): protocol: expected piece hash; context canceled; EOF Dec 8 19:42:40.628744101 (76h13m40.499827146s) [noise=true , address=89.149.201.147:4001 , piece_id=D65FANROOLI47T6542V4YMZ5HJ6YG5BG6IS2H27KYIR3P4WNH2YQ, seg_pos={0 0}, upload=66666]: Long tail cancellation due to optimal threshold Dec 8 19:42:40.628767596 (76h13m40.499850641s) [seg_pos={0 0}, upload=66666]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 19:42:40.628886502 (76h13m40.499969547s) [upload=66666]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 19:42:40.628893004 (76h13m40.49997606s) [upload=66666]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 19:48:22.295722109 (76h19m22.166805154s) [upload=66666]: Done uploading segments... Dec 8 19:48:22.295853299 (76h19m22.166936354s) [upload=66666]: Upload finished. err: metaclient: context deadline exceeded Duration: 5m45.743411756s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.503037256s trace: https://jaeger.dev.storj.io/trace/624d2b5b2aee8843 FAIL (2025-12-08 19:48:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.503094898s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................X...........X..............] duration: 5.808826161s trace: https://jaeger.dev.storj.io/trace/2c6b8a8384caec89 FAIL (2025-12-08 19:48:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.808854715s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 475.646005ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/5e91473084188cf7 FAIL (2025-12-08 19:48:06 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 475.665482ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.01362844s trace: https://jaeger.dev.storj.io/trace/6bf00a8c3609c4c7 FAIL (2025-12-08 19:47:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.013657536s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.733115472s trace: https://jaeger.dev.storj.io/trace/30f9f47120cf6974 FAIL (2025-12-08 19:47:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 19:10:37.433384005 (75h41m37.30446704s) [upload=66189]: Starting upload Dec 8 19:10:37.433449450 (75h41m37.304532495s) [upload=66189]: Done starting upload Dec 8 19:10:37.433643290 (75h41m37.304726335s) [upload=66189]: Uploading segments... Dec 8 19:10:37.433675882 (75h41m37.304758917s) [upload=66189]: Got next segment. Inline: false Dec 8 19:10:37.433704186 (75h41m37.304787221s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 19:10:37.433708875 (75h41m37.30479191s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 19:10:37.679613305 (75h41m37.55069634s) [seg_pos={0 0}, upload=66189]: Begin upload segment... Dec 8 19:10:37.679683469 (75h41m37.550766514s) [seg_pos={0 0}, upload=66189]: Done begin upload segment. Dec 8 19:10:37.679752580 (75h41m37.550835625s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.679774421 (75h41m37.550857456s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.680422305 (75h41m37.55150534s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681032428 (75h41m37.552115463s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681404696 (75h41m37.552487741s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681922101 (75h41m37.553005136s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682032361 (75h41m37.553115396s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682588229 (75h41m37.553671274s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682965478 (75h41m37.554048523s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683881782 (75h41m37.554964817s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683958880 (75h41m37.555041915s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.684982759 (75h41m37.556065804s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687280685 (75h41m37.5583638s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687699612 (75h41m37.558782658s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688055951 (75h41m37.559139006s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688417931 (75h41m37.559500966s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:38.705641184 (75h41m38.576724219s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 19:10:39.291878765 (75h41m39.16296181s) [upload=66189]: Next returned nil segment Dec 8 19:10:39.291893282 (75h41m39.162976317s) [upload=66189]: Waiting for error group managing segments... Dec 8 19:10:39.671673458 (75h41m39.542756523s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.672368722 (75h41m39.543451767s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.674201522 (75h41m39.545284557s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.679966128 (75h41m39.551049163s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.684661019 (75h41m39.555744054s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.688164300 (75h41m39.559247335s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.689301024 (75h41m39.56038406s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699377182 (75h41m39.570460227s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699944802 (75h41m39.571027837s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.701579305 (75h41m39.57266234s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.702082162 (75h41m39.573165197s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.858397206 (75h41m39.729480251s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919886866 (75h41m39.790969901s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919914609 (75h41m39.790997644s) [seg_pos={0 0}, upload=66189]: Segment reached optimal threshold of 13 pieces. Dec 8 19:10:39.920157301 (75h41m39.791240346s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection Dec 8 19:10:39.920166508 (75h41m39.791249553s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920380536 (75h41m39.791463571s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 19:10:39.920389733 (75h41m39.791472788s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920416865 (75h41m39.79149991s) [seg_pos={0 0}, upload=66189]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 19:10:39.920506556 (75h41m39.791589601s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 19:10:39.920513880 (75h41m39.791596925s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 19:10:40.218384935 (75h41m40.08946797s) [upload=66189]: Done uploading segments... Dec 8 19:10:40.218431954 (75h41m40.089514999s) [upload=66189]: Upload finished. err: <nil> Duration: 5.733142274s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 470.526507ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/5e91473084188cf7 FAIL (2025-12-08 19:47:35 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 470.552034ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.063823138s trace: https://jaeger.dev.storj.io/trace/21f9fb0e12c4fe9 FAIL (2025-12-08 19:47:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.063854878s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.51408859s trace: https://jaeger.dev.storj.io/trace/624d2b5b2aee8843 FAIL (2025-12-08 19:47:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.51411501s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 468.173817ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/5e91473084188cf7 FAIL (2025-12-08 19:46:57 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 468.201119ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................X...........X.............] duration: 5.809960851s trace: https://jaeger.dev.storj.io/trace/2c6b8a8384caec89 FAIL (2025-12-08 19:46:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.809987261s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.825112903s trace: https://jaeger.dev.storj.io/trace/6bf00a8c3609c4c7 FAIL (2025-12-08 19:46:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.825139214s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.717196413s trace: https://jaeger.dev.storj.io/trace/30f9f47120cf6974 FAIL (2025-12-08 19:46:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 19:10:37.433384005 (75h41m37.30446704s) [upload=66189]: Starting upload Dec 8 19:10:37.433449450 (75h41m37.304532495s) [upload=66189]: Done starting upload Dec 8 19:10:37.433643290 (75h41m37.304726335s) [upload=66189]: Uploading segments... Dec 8 19:10:37.433675882 (75h41m37.304758917s) [upload=66189]: Got next segment. Inline: false Dec 8 19:10:37.433704186 (75h41m37.304787221s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 19:10:37.433708875 (75h41m37.30479191s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 19:10:37.679613305 (75h41m37.55069634s) [seg_pos={0 0}, upload=66189]: Begin upload segment... Dec 8 19:10:37.679683469 (75h41m37.550766514s) [seg_pos={0 0}, upload=66189]: Done begin upload segment. Dec 8 19:10:37.679752580 (75h41m37.550835625s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.679774421 (75h41m37.550857456s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.680422305 (75h41m37.55150534s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681032428 (75h41m37.552115463s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681404696 (75h41m37.552487741s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681922101 (75h41m37.553005136s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682032361 (75h41m37.553115396s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682588229 (75h41m37.553671274s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682965478 (75h41m37.554048523s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683881782 (75h41m37.554964817s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683958880 (75h41m37.555041915s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.684982759 (75h41m37.556065804s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687280685 (75h41m37.5583638s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687699612 (75h41m37.558782658s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688055951 (75h41m37.559139006s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688417931 (75h41m37.559500966s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:38.705641184 (75h41m38.576724219s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 19:10:39.291878765 (75h41m39.16296181s) [upload=66189]: Next returned nil segment Dec 8 19:10:39.291893282 (75h41m39.162976317s) [upload=66189]: Waiting for error group managing segments... Dec 8 19:10:39.671673458 (75h41m39.542756523s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.672368722 (75h41m39.543451767s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.674201522 (75h41m39.545284557s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.679966128 (75h41m39.551049163s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.684661019 (75h41m39.555744054s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.688164300 (75h41m39.559247335s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.689301024 (75h41m39.56038406s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699377182 (75h41m39.570460227s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699944802 (75h41m39.571027837s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.701579305 (75h41m39.57266234s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.702082162 (75h41m39.573165197s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.858397206 (75h41m39.729480251s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919886866 (75h41m39.790969901s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919914609 (75h41m39.790997644s) [seg_pos={0 0}, upload=66189]: Segment reached optimal threshold of 13 pieces. Dec 8 19:10:39.920157301 (75h41m39.791240346s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection Dec 8 19:10:39.920166508 (75h41m39.791249553s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920380536 (75h41m39.791463571s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 19:10:39.920389733 (75h41m39.791472788s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920416865 (75h41m39.79149991s) [seg_pos={0 0}, upload=66189]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 19:10:39.920506556 (75h41m39.791589601s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 19:10:39.920513880 (75h41m39.791596925s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 19:10:40.218384935 (75h41m40.08946797s) [upload=66189]: Done uploading segments... Dec 8 19:10:40.218431954 (75h41m40.089514999s) [upload=66189]: Upload finished. err: <nil> Duration: 5.717223394s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 4.333962737s trace: https://jaeger.dev.storj.io/trace/624d2b5b2aee8843 FAIL (2025-12-08 19:46:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.333996061s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.022659955s trace: https://jaeger.dev.storj.io/trace/21f9fb0e12c4fe9 FAIL (2025-12-08 19:46:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.02268367s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 466.634475ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/5e91473084188cf7 FAIL (2025-12-08 19:46:13 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 466.661877ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 10.476266549s trace: https://jaeger.dev.storj.io/trace/30f9f47120cf6974 FAIL (2025-12-08 19:45:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 19:10:37.433384005 (75h41m37.30446704s) [upload=66189]: Starting upload Dec 8 19:10:37.433449450 (75h41m37.304532495s) [upload=66189]: Done starting upload Dec 8 19:10:37.433643290 (75h41m37.304726335s) [upload=66189]: Uploading segments... Dec 8 19:10:37.433675882 (75h41m37.304758917s) [upload=66189]: Got next segment. Inline: false Dec 8 19:10:37.433704186 (75h41m37.304787221s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 19:10:37.433708875 (75h41m37.30479191s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 19:10:37.679613305 (75h41m37.55069634s) [seg_pos={0 0}, upload=66189]: Begin upload segment... Dec 8 19:10:37.679683469 (75h41m37.550766514s) [seg_pos={0 0}, upload=66189]: Done begin upload segment. Dec 8 19:10:37.679752580 (75h41m37.550835625s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.679774421 (75h41m37.550857456s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.680422305 (75h41m37.55150534s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681032428 (75h41m37.552115463s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681404696 (75h41m37.552487741s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681922101 (75h41m37.553005136s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682032361 (75h41m37.553115396s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682588229 (75h41m37.553671274s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682965478 (75h41m37.554048523s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683881782 (75h41m37.554964817s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683958880 (75h41m37.555041915s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.684982759 (75h41m37.556065804s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687280685 (75h41m37.5583638s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687699612 (75h41m37.558782658s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688055951 (75h41m37.559139006s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688417931 (75h41m37.559500966s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:38.705641184 (75h41m38.576724219s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 19:10:39.291878765 (75h41m39.16296181s) [upload=66189]: Next returned nil segment Dec 8 19:10:39.291893282 (75h41m39.162976317s) [upload=66189]: Waiting for error group managing segments... Dec 8 19:10:39.671673458 (75h41m39.542756523s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.672368722 (75h41m39.543451767s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.674201522 (75h41m39.545284557s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.679966128 (75h41m39.551049163s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.684661019 (75h41m39.555744054s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.688164300 (75h41m39.559247335s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.689301024 (75h41m39.56038406s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699377182 (75h41m39.570460227s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699944802 (75h41m39.571027837s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.701579305 (75h41m39.57266234s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.702082162 (75h41m39.573165197s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.858397206 (75h41m39.729480251s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919886866 (75h41m39.790969901s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919914609 (75h41m39.790997644s) [seg_pos={0 0}, upload=66189]: Segment reached optimal threshold of 13 pieces. Dec 8 19:10:39.920157301 (75h41m39.791240346s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection Dec 8 19:10:39.920166508 (75h41m39.791249553s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920380536 (75h41m39.791463571s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 19:10:39.920389733 (75h41m39.791472788s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920416865 (75h41m39.79149991s) [seg_pos={0 0}, upload=66189]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 19:10:39.920506556 (75h41m39.791589601s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 19:10:39.920513880 (75h41m39.791596925s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 19:10:40.218384935 (75h41m40.08946797s) [upload=66189]: Done uploading segments... Dec 8 19:10:40.218431954 (75h41m40.089514999s) [upload=66189]: Upload finished. err: <nil> Duration: 10.476288931s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.046280311s trace: https://jaeger.dev.storj.io/trace/21f9fb0e12c4fe9 FAIL (2025-12-08 19:45:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.046308114s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................X...........X............] duration: 5.927343779s trace: https://jaeger.dev.storj.io/trace/2c6b8a8384caec89 FAIL (2025-12-08 19:45:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.927436095s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.829195549s trace: https://jaeger.dev.storj.io/trace/6bf00a8c3609c4c7 FAIL (2025-12-08 19:45:46 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.829242156s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X.....................................X.....................] duration: 3.534965254s trace: https://jaeger.dev.storj.io/trace/624d2b5b2aee8843 FAIL (2025-12-08 19:45:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.534990161s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 469.769214ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/5e91473084188cf7 FAIL (2025-12-08 19:45:32 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 469.781517ms core satellite eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.314147795s remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/5e91473084188cf7 FAIL (2025-12-08 19:45:31 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 1m0.314172752s core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [X...................................X...........X...........] duration: 6.538407174s trace: https://jaeger.dev.storj.io/trace/2c6b8a8384caec89 FAIL (2025-12-08 19:45:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.53843724s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X.....................................X....................] duration: 3.487208785s trace: https://jaeger.dev.storj.io/trace/624d2b5b2aee8843 FAIL (2025-12-08 19:45:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.487238692s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.064349722s trace: https://jaeger.dev.storj.io/trace/6bf00a8c3609c4c7 FAIL (2025-12-08 19:45:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.064376553s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.861677009s trace: https://jaeger.dev.storj.io/trace/30f9f47120cf6974 FAIL (2025-12-08 19:44:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 19:10:37.433384005 (75h41m37.30446704s) [upload=66189]: Starting upload Dec 8 19:10:37.433449450 (75h41m37.304532495s) [upload=66189]: Done starting upload Dec 8 19:10:37.433643290 (75h41m37.304726335s) [upload=66189]: Uploading segments... Dec 8 19:10:37.433675882 (75h41m37.304758917s) [upload=66189]: Got next segment. Inline: false Dec 8 19:10:37.433704186 (75h41m37.304787221s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 19:10:37.433708875 (75h41m37.30479191s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 19:10:37.679613305 (75h41m37.55069634s) [seg_pos={0 0}, upload=66189]: Begin upload segment... Dec 8 19:10:37.679683469 (75h41m37.550766514s) [seg_pos={0 0}, upload=66189]: Done begin upload segment. Dec 8 19:10:37.679752580 (75h41m37.550835625s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.679774421 (75h41m37.550857456s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.680422305 (75h41m37.55150534s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681032428 (75h41m37.552115463s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681404696 (75h41m37.552487741s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681922101 (75h41m37.553005136s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682032361 (75h41m37.553115396s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682588229 (75h41m37.553671274s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682965478 (75h41m37.554048523s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683881782 (75h41m37.554964817s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683958880 (75h41m37.555041915s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.684982759 (75h41m37.556065804s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687280685 (75h41m37.5583638s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687699612 (75h41m37.558782658s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688055951 (75h41m37.559139006s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688417931 (75h41m37.559500966s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:38.705641184 (75h41m38.576724219s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 19:10:39.291878765 (75h41m39.16296181s) [upload=66189]: Next returned nil segment Dec 8 19:10:39.291893282 (75h41m39.162976317s) [upload=66189]: Waiting for error group managing segments... Dec 8 19:10:39.671673458 (75h41m39.542756523s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.672368722 (75h41m39.543451767s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.674201522 (75h41m39.545284557s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.679966128 (75h41m39.551049163s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.684661019 (75h41m39.555744054s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.688164300 (75h41m39.559247335s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.689301024 (75h41m39.56038406s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699377182 (75h41m39.570460227s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699944802 (75h41m39.571027837s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.701579305 (75h41m39.57266234s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.702082162 (75h41m39.573165197s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.858397206 (75h41m39.729480251s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919886866 (75h41m39.790969901s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919914609 (75h41m39.790997644s) [seg_pos={0 0}, upload=66189]: Segment reached optimal threshold of 13 pieces. Dec 8 19:10:39.920157301 (75h41m39.791240346s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection Dec 8 19:10:39.920166508 (75h41m39.791249553s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920380536 (75h41m39.791463571s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 19:10:39.920389733 (75h41m39.791472788s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920416865 (75h41m39.79149991s) [seg_pos={0 0}, upload=66189]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 19:10:39.920506556 (75h41m39.791589601s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 19:10:39.920513880 (75h41m39.791596925s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 19:10:40.218384935 (75h41m40.08946797s) [upload=66189]: Done uploading segments... Dec 8 19:10:40.218431954 (75h41m40.089514999s) [upload=66189]: Upload finished. err: <nil> Duration: 5.861695413s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.99434521s trace: https://jaeger.dev.storj.io/trace/21f9fb0e12c4fe9 FAIL (2025-12-08 19:44:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.994368974s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [XX...................................X...........X..........] duration: 5.778040609s trace: https://jaeger.dev.storj.io/trace/2c6b8a8384caec89 FAIL (2025-12-08 19:44:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.778068383s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.882621871s trace: https://jaeger.dev.storj.io/trace/6bf00a8c3609c4c7 FAIL (2025-12-08 19:44:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.88265281s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..X.....................................X...................] duration: 4.446825187s trace: https://jaeger.dev.storj.io/trace/624d2b5b2aee8843 FAIL (2025-12-08 19:44:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.446851117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.02029495s trace: https://jaeger.dev.storj.io/trace/21f9fb0e12c4fe9 FAIL (2025-12-08 19:44:15 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.020348932s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.XX...................................X...........X.........] duration: 5.778183752s trace: https://jaeger.dev.storj.io/trace/2c6b8a8384caec89 FAIL (2025-12-08 19:43:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.778195935s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.713154906s trace: https://jaeger.dev.storj.io/trace/30f9f47120cf6974 FAIL (2025-12-08 19:43:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 19:10:37.433384005 (75h41m37.30446704s) [upload=66189]: Starting upload Dec 8 19:10:37.433449450 (75h41m37.304532495s) [upload=66189]: Done starting upload Dec 8 19:10:37.433643290 (75h41m37.304726335s) [upload=66189]: Uploading segments... Dec 8 19:10:37.433675882 (75h41m37.304758917s) [upload=66189]: Got next segment. Inline: false Dec 8 19:10:37.433704186 (75h41m37.304787221s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 19:10:37.433708875 (75h41m37.30479191s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 19:10:37.679613305 (75h41m37.55069634s) [seg_pos={0 0}, upload=66189]: Begin upload segment... Dec 8 19:10:37.679683469 (75h41m37.550766514s) [seg_pos={0 0}, upload=66189]: Done begin upload segment. Dec 8 19:10:37.679752580 (75h41m37.550835625s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.679774421 (75h41m37.550857456s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.680422305 (75h41m37.55150534s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681032428 (75h41m37.552115463s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681404696 (75h41m37.552487741s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.681922101 (75h41m37.553005136s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682032361 (75h41m37.553115396s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682588229 (75h41m37.553671274s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.682965478 (75h41m37.554048523s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683881782 (75h41m37.554964817s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.683958880 (75h41m37.555041915s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.684982759 (75h41m37.556065804s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687280685 (75h41m37.5583638s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.687699612 (75h41m37.558782658s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688055951 (75h41m37.559139006s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:37.688417931 (75h41m37.559500966s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Uploading piece... Dec 8 19:10:38.705641184 (75h41m38.576724219s) [noise=true , address=145.79.227.30:4763 , piece_id=MT7FEEXZPB7OJKKCQ4G5W6MKJ6NQERZQJ2WUD7RMWLJIRN7WLW3Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 8 19:10:39.291878765 (75h41m39.16296181s) [upload=66189]: Next returned nil segment Dec 8 19:10:39.291893282 (75h41m39.162976317s) [upload=66189]: Waiting for error group managing segments... Dec 8 19:10:39.671673458 (75h41m39.542756523s) [noise=true , address=145.79.227.27:4689 , piece_id=RUIGTMRJ32YAT4RYBT5FE3PRQ75NPDSBH3Y3GSJF6LSK774QKLHA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.672368722 (75h41m39.543451767s) [noise=true , address=145.79.227.34:4906 , piece_id=Z4C6UOKXFJDTN4EHOCNBZUCUKAYY5JUHXOMRUHT5XUMISBHJHNCA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.674201522 (75h41m39.545284557s) [noise=true , address=145.79.227.23:4546 , piece_id=H4RPQAJZ4233DE6NW56IY4UPSOCBOB2TUB2DPUKAL2I6RVHAG3XA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.679966128 (75h41m39.551049163s) [noise=true , address=145.79.227.42:4122 , piece_id=KPW72WFNAATIDZLDHSYILRP4YVUPLU3VPHVTSL4YNEDP6UOZGJNA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.684661019 (75h41m39.555744054s) [noise=true , address=145.79.227.43:4157 , piece_id=UO72RB7IKYUVQ5FWPBBEQNPLYRJF6VIMM3EUFYLT7DOBEDJ3S5DA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.688164300 (75h41m39.559247335s) [noise=true , address=145.79.227.38:4978 , piece_id=BMAF2PYEUIATK6ZZYWYQEVEW3TXYNKR3EKMQES2AS3ZUSFIIRP5Q, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.689301024 (75h41m39.56038406s) [noise=true , address=145.79.227.40:4050 , piece_id=PBY76CABRCG7BRXAMT3QGTS3KHNVQI52PR6FZGLTDQVXTJ3JETPQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699377182 (75h41m39.570460227s) [noise=true , address=145.79.227.28:4727 , piece_id=725UOLTBXPJOUSZAJ5HEC7WYJODPMTRFJG27IGGA4QDDTT2Y5WUQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.699944802 (75h41m39.571027837s) [noise=true , address=145.79.227.31:4799 , piece_id=WZCYG2DEEDZYVY6Z6GZ7CEPNIPVRZVYPRTDLYCD3USAPJJGSEZRA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.701579305 (75h41m39.57266234s) [noise=true , address=145.79.227.21:4475 , piece_id=GVHX6XLPCLMRCD5KYITFUXSNIPIZLPWZL5BPMJVYJ3A5SC3VKH2A, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.702082162 (75h41m39.573165197s) [noise=true , address=145.79.227.41:4085 , piece_id=QJBEV6UTTK2DQR2E3Y4LVK5UM254KVKXZQAVBSVXAORRCAVWGEGQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.858397206 (75h41m39.729480251s) [noise=true , address=77.90.31.11:4555 , piece_id=5JQHBTQKJKGEPJD2OP2NNBUIQBHR6AGDFULNSJGBDD5XI27VVFBA, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919886866 (75h41m39.790969901s) [noise=true , address=77.90.31.19:4842 , piece_id=KQRY2YJUDOUKYKQVGCT45SI6DOFNIYAB5YLBR3MQE3BIYAUZ2VWQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: <nil> Dec 8 19:10:39.919914609 (75h41m39.790997644s) [seg_pos={0 0}, upload=66189]: Segment reached optimal threshold of 13 pieces. Dec 8 19:10:39.920157301 (75h41m39.791240346s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51016->77.90.31.13:4627: use of closed network connection Dec 8 19:10:39.920166508 (75h41m39.791249553s) [noise=true , address=77.90.31.13:4627 , piece_id=FIEOJJW4R35QJ5QRZMKAZSHLI2C5NCVNXF43RKW43SJC7QWNAJXQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920380536 (75h41m39.791463571s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 19:10:39.920389733 (75h41m39.791472788s) [noise=true , address=145.79.227.20:4438 , piece_id=7ZZELR7MK4FMRSIVOKHILFR3KVOXAEAXMPEUG36PGQ4DOIJR3BKQ, seg_pos={0 0}, upload=66189]: Long tail cancellation due to optimal threshold Dec 8 19:10:39.920416865 (75h41m39.79149991s) [seg_pos={0 0}, upload=66189]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 19:10:39.920506556 (75h41m39.791589601s) [upload=66189]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 19:10:39.920513880 (75h41m39.791596925s) [upload=66189]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 19:10:40.218384935 (75h41m40.08946797s) [upload=66189]: Done uploading segments... Dec 8 19:10:40.218431954 (75h41m40.089514999s) [upload=66189]: Upload finished. err: <nil> Duration: 5.713181416s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)