Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.......X..XX..X.X.X..........X....X.......X.....X..XXX....X] duration: 30.004420412s trace: https://jaeger.dev.storj.io/trace/129f59a3714917d6 FAIL (2025-12-30 11:28:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 11:27:46.440240238 (595h58m46.311323273s) [upload=522433]: Starting upload Dec 30 11:27:46.440318116 (595h58m46.311401151s) [upload=522433]: Done starting upload Dec 30 11:27:46.440390404 (595h58m46.311473439s) [upload=522433]: Uploading segments... Dec 30 11:27:46.440584774 (595h58m46.311667809s) [upload=522433]: Got next segment. Inline: false Dec 30 11:27:46.440622746 (595h58m46.311705781s) [upload=522433]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:27:46.440628357 (595h58m46.311711392s) [upload=522433]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:27:46.683842646 (595h58m46.554925681s) [seg_pos={0 0}, upload=522433]: Begin upload segment... Dec 30 11:27:46.683891881 (595h58m46.554974916s) [seg_pos={0 0}, upload=522433]: Done begin upload segment. Dec 30 11:27:46.683924463 (595h58m46.555007498s) [noise=true , address=145.79.227.39:4014 , piece_id=LVY3UOQYQ22G75JI4H4PNHVA5U7LAQXM4M5NKVALPNYOBBOOXQ7A, seg_pos={0 0}, upload=522433]: Uploading piece... Dec 30 11:27:46.684055852 (595h58m46.555138887s) [noise=true , address=145.79.227.37:4567 , piece_id=BYDYK6RTVDPRH5NAXWNQCHRTBGOQMSYBQC2DCHQGDKCMM3NDYHWQ, seg_pos={0 0}, upload=522433]: Uploading piece... Dec 30 11:27:46.684466324 (595h58m46.555549359s) [noise=true , address=145.79.227.51:4410 , piece_id=EEALDUZKD5CU3P335S4ULNBHVOA5QJTFGJH5XFZ7AMLIEXCSVJOA, seg_pos={0 0}, upload=522433]: Uploading piece... Dec 30 11:27:46.684993838 (595h58m46.556076873s) [noise=true , address=77.90.31.18:4805 , piece_id=GOK4JX5QWQU6IDHR56PRT7DH3UFQI54RRB5ONIVYI7F5V5SC4SUA, seg_pos={0 0}, upload=522433]: Uploading piece... Dec 30 11:27:46.685047791 (595h58m46.556130836s) [noise=true , address=45.91.116.5:4299 , piece_id=UNOG65HH6MFU2RT3J6XCICZUK54ES67H32BD35HLYBXL2YURLELA, seg_pos={0 0}, upload=522433]: Uploading piece... Dec 30 11:27:46.685749978 (595h58m46.556833013s) [noise=true , address=145.79.227.45:4229 , piece_id=TBMEMEBU44ZX3XUHHNUOQWHPCHUVMJ224AQ5PH6HP5QADYAHHYEQ, seg_pos={0 0}, upload=522433]: Uploading piece... Dec 30 11:27:46.685789233 (595h58m46.556872268s) [noise=true , address=145.79.227.41:4085 , piece_id=YDQQ4DP5DRUOGTVG25NBO2DJOBXXVJOEAKSN774KMUYNSEJCQ6JQ, seg_pos={0 0}, upload=522433]: Uploading piece... Dec 30 11:27:46.686694186 (595h58m46.557777221s) [noise=true , address=45.91.116.8:4047 , piece_id=JUBVOE5A47MLONLPHS7IS5MUZG75AVB7B6OQIFFVCNPDVSTYI7QA, seg_pos={0 0}, upload=522433]: Uploading piece... Dec 30 11:27:46.686755744 (595h58m46.557838779s) [noise=true , address=145.79.227.3:4150 , piece_id=N4AX7D6AAZLNPJODHYMGJ44YYGBMB24ZLJRVI54IBWAS5HSLEW3Q, seg_pos={0 0}, upload=522433]: Uploading piece... Dec 30 11:27:46.687429546 (595h58m46.558512591s) [noise=true , address=145.79.227.10:4401 , piece_id=JXZEV4K37HVTVKURAFS7M2S4KDXP32K2BA3X3Y2L74O5F55XVAQQ, seg_pos={0 0}, upload=522433]: Uploading piece... Dec 30 11:27:46.687456919 (595h58m46.558539954s) [noise=true , address=145.79.227.19:4077 , piece_id=WLTEI2JMV2U6NRGYODSKZDJDETOLSS2BGW5CJH5IFKUDHAGYSTFQ, seg_pos={0 0}, upload=522433]: Uploading piece... Dec 30 11:27:46.688037474 (595h58m46.559120519s) [noise=true , address=145.79.227.34:4905 , piece_id=DON3R2JDOY5MVILXOTK3OYCP7UL6ZZAYG6DMXB7PTVNVHHDPVOMA, seg_pos={0 0}, upload=522433]: Uploading piece... Dec 30 11:27:46.688385697 (595h58m46.559468732s) [noise=true , address=45.91.116.9:4262 , piece_id=35HJWTMVOCERJ27OFMUEXZ6SX73GSQ4TYCW7EC5TGJT4QM5WNNKQ, seg_pos={0 0}, upload=522433]: Uploading piece... Dec 30 11:27:46.689944586 (595h58m46.561027631s) [noise=true , address=145.79.227.50:4373 , piece_id=7PPRM27TPD7DDC5ACNLWH7EOBAIYFRD23WY7FDVE7FVDPOSJ2LKA, seg_pos={0 0}, upload=522433]: Uploading piece... Dec 30 11:27:46.690433747 (595h58m46.561516803s) [noise=true , address=145.79.227.8:4331 , piece_id=RBYQZDBOTTXFFABFDJN4XUPGUPVXGEEASJQTFYWPWDNSSOYTTPAQ, seg_pos={0 0}, upload=522433]: Uploading piece... Dec 30 11:27:46.690756902 (595h58m46.561839938s) [noise=true , address=77.90.31.21:4877 , piece_id=IIMQX663WROV35EJQPNHVYKMAMYNXQS4HGKHLF3XQGC2HTOOM4HA, seg_pos={0 0}, upload=522433]: Uploading piece... Dec 30 11:27:48.304125345 (595h58m48.17520838s) [upload=522433]: Next returned nil segment Dec 30 11:27:48.304138780 (595h58m48.175221815s) [upload=522433]: Waiting for error group managing segments... Dec 30 11:27:48.675618648 (595h58m48.546701693s) [noise=true , address=145.79.227.10:4401 , piece_id=JXZEV4K37HVTVKURAFS7M2S4KDXP32K2BA3X3Y2L74O5F55XVAQQ, seg_pos={0 0}, upload=522433]: Done uploading piece. err: <nil> Dec 30 11:27:48.675859277 (595h58m48.546942312s) [noise=true , address=145.79.227.41:4085 , piece_id=YDQQ4DP5DRUOGTVG25NBO2DJOBXXVJOEAKSN774KMUYNSEJCQ6JQ, seg_pos={0 0}, upload=522433]: Done uploading piece. err: <nil> Dec 30 11:27:48.679095568 (595h58m48.550178623s) [noise=true , address=145.79.227.51:4410 , piece_id=EEALDUZKD5CU3P335S4ULNBHVOA5QJTFGJH5XFZ7AMLIEXCSVJOA, seg_pos={0 0}, upload=522433]: Done uploading piece. err: <nil> Dec 30 11:27:48.679716491 (595h58m48.550799536s) [noise=true , address=145.79.227.8:4331 , piece_id=RBYQZDBOTTXFFABFDJN4XUPGUPVXGEEASJQTFYWPWDNSSOYTTPAQ, seg_pos={0 0}, upload=522433]: Done uploading piece. err: <nil> Dec 30 11:27:48.698649222 (595h58m48.569732257s) [noise=true , address=145.79.227.3:4150 , piece_id=N4AX7D6AAZLNPJODHYMGJ44YYGBMB24ZLJRVI54IBWAS5HSLEW3Q, seg_pos={0 0}, upload=522433]: Done uploading piece. err: <nil> Dec 30 11:27:48.705909708 (595h58m48.576992753s) [noise=true , address=145.79.227.45:4229 , piece_id=TBMEMEBU44ZX3XUHHNUOQWHPCHUVMJ224AQ5PH6HP5QADYAHHYEQ, seg_pos={0 0}, upload=522433]: Done uploading piece. err: <nil> Dec 30 11:27:48.709463453 (595h58m48.580546488s) [noise=true , address=145.79.227.39:4014 , piece_id=LVY3UOQYQ22G75JI4H4PNHVA5U7LAQXM4M5NKVALPNYOBBOOXQ7A, seg_pos={0 0}, upload=522433]: Done uploading piece. err: <nil> Dec 30 11:27:48.757117385 (595h58m48.62820043s) [noise=true , address=145.79.227.50:4373 , piece_id=7PPRM27TPD7DDC5ACNLWH7EOBAIYFRD23WY7FDVE7FVDPOSJ2LKA, seg_pos={0 0}, upload=522433]: Done uploading piece. err: <nil> Dec 30 11:27:48.765155761 (595h58m48.636238796s) [noise=true , address=145.79.227.19:4077 , piece_id=WLTEI2JMV2U6NRGYODSKZDJDETOLSS2BGW5CJH5IFKUDHAGYSTFQ, seg_pos={0 0}, upload=522433]: Done uploading piece. err: <nil> Dec 30 11:27:48.772021034 (595h58m48.643104069s) [noise=true , address=145.79.227.37:4567 , piece_id=BYDYK6RTVDPRH5NAXWNQCHRTBGOQMSYBQC2DCHQGDKCMM3NDYHWQ, seg_pos={0 0}, upload=522433]: Done uploading piece. err: <nil> Dec 30 11:27:48.938401378 (595h58m48.809484413s) [noise=true , address=77.90.31.21:4877 , piece_id=IIMQX663WROV35EJQPNHVYKMAMYNXQS4HGKHLF3XQGC2HTOOM4HA, seg_pos={0 0}, upload=522433]: Done uploading piece. err: <nil> Dec 30 11:27:49.245346882 (595h58m49.116429917s) [noise=true , address=45.91.116.8:4047 , piece_id=JUBVOE5A47MLONLPHS7IS5MUZG75AVB7B6OQIFFVCNPDVSTYI7QA, seg_pos={0 0}, upload=522433]: Done uploading piece. err: <nil> Dec 30 11:27:49.645935507 (595h58m49.517018542s) [noise=true , address=77.90.31.18:4805 , piece_id=GOK4JX5QWQU6IDHR56PRT7DH3UFQI54RRB5ONIVYI7F5V5SC4SUA, seg_pos={0 0}, upload=522433]: Done uploading piece. err: <nil> Dec 30 11:27:49.645956317 (595h58m49.517039352s) [seg_pos={0 0}, upload=522433]: Segment reached optimal threshold of 13 pieces. Dec 30 11:27:49.646263942 (595h58m49.517346977s) [noise=true , address=45.91.116.9:4262 , piece_id=35HJWTMVOCERJ27OFMUEXZ6SX73GSQ4TYCW7EC5TGJT4QM5WNNKQ, seg_pos={0 0}, upload=522433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 30 11:27:49.646272088 (595h58m49.517355133s) [noise=true , address=45.91.116.9:4262 , piece_id=35HJWTMVOCERJ27OFMUEXZ6SX73GSQ4TYCW7EC5TGJT4QM5WNNKQ, seg_pos={0 0}, upload=522433]: Long tail cancellation due to optimal threshold Dec 30 11:27:49.646477600 (595h58m49.517560645s) [noise=true , address=45.91.116.5:4299 , piece_id=UNOG65HH6MFU2RT3J6XCICZUK54ES67H32BD35HLYBXL2YURLELA, seg_pos={0 0}, upload=522433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 30 11:27:49.646505191 (595h58m49.517588237s) [noise=true , address=45.91.116.5:4299 , piece_id=UNOG65HH6MFU2RT3J6XCICZUK54ES67H32BD35HLYBXL2YURLELA, seg_pos={0 0}, upload=522433]: Long tail cancellation due to optimal threshold Dec 30 11:27:49.646604851 (595h58m49.517687896s) [noise=true , address=145.79.227.34:4905 , piece_id=DON3R2JDOY5MVILXOTK3OYCP7UL6ZZAYG6DMXB7PTVNVHHDPVOMA, seg_pos={0 0}, upload=522433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 11:27:49.646611925 (595h58m49.51769496s) [noise=true , address=145.79.227.34:4905 , piece_id=DON3R2JDOY5MVILXOTK3OYCP7UL6ZZAYG6DMXB7PTVNVHHDPVOMA, seg_pos={0 0}, upload=522433]: Long tail cancellation due to optimal threshold Dec 30 11:27:49.646630871 (595h58m49.517713906s) [seg_pos={0 0}, upload=522433]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:27:49.646714680 (595h58m49.517797715s) [upload=522433]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:27:49.646719720 (595h58m49.517802755s) [upload=522433]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:27:49.936894881 (595h58m49.807977926s) [upload=522433]: Done uploading segments... Dec 30 11:27:49.936919077 (595h58m49.808002112s) [upload=522433]: Upload finished. err: <nil> Duration: 30.004444729s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X.................X..........X...X...........X....X.] duration: 46.183613129s trace: https://jaeger.dev.storj.io/trace/51943f4f02f0fbef FAIL (2025-12-30 11:27:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=522411]: Uploading piece... Dec 30 11:26:04.713753765 (595h57m4.58483681s) [noise=true , address=193.149.164.143:4697 , piece_id=KK6T67EEQX7F6J2E5NQANUBYUHDXZ2LDWSD2K4RMRUL2VTEFX3RA, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.713866410 (595h57m4.584949445s) [noise=true , address=64.49.13.169:4708 , piece_id=E7HPPLI24SYOM2POIZJPKFDWNC3CVA3FQ4DMATKB6M7277SOCI6A, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.714391520 (595h57m4.585474555s) [noise=true , address=77.90.31.18:4817 , piece_id=ZOEZ5AFC3C2VG6ARKT62SNLG4J2NA2MFBSATAKTTVBAH55I7GUSQ, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.714693134 (595h57m4.585776169s) [noise=true , address=145.79.227.29:4961 , piece_id=JCJ4OYJKNI5UEHSHTZEJ4ZSVMXJJDEME4ZES6RTXEASIJB56WRZQ, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.714791531 (595h57m4.585874566s) [noise=true , address=77.90.31.15:4710 , piece_id=P45UIV77TIYJJE4NKMKFU7DS75OVLXQ6ARNH52H3HSKXTO6FEXUA, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.715141928 (595h57m4.586224963s) [noise=true , address=145.79.227.30:4784 , piece_id=ZL3HJBEYR36TLMRZI22QMZQRUYDKME3BZZLG7J3L2HXT6UCXB2VA, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.715160304 (595h57m4.586243339s) [noise=true , address=145.79.227.40:4062 , piece_id=TL3XD7TH57B7AUXCAFXSXA424X2YO332QN6FXWGEWS6N3KNBMHOQ, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.715475855 (595h57m4.58655889s) [noise=true , address=145.79.227.12:4790 , piece_id=RL4XZCKHUMLXRUCPKVR6ZNPPMOGEXW332ITCZJVHHLBZ765OOBWQ, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.715638173 (595h57m4.586721218s) [noise=true , address=64.49.13.27:4965 , piece_id=24SDWK4JBDZFHMVPNYEKPF3RJ4WERNSZY226RQHR4EC44ZDNHCGA, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.715906915 (595h57m4.58698995s) [noise=true , address=45.91.116.6:4001 , piece_id=LTUOVU6C624NQYYZCIE5XJAJQINSHZQRB6D7KVAZPJZD4SV46X4A, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.716128207 (595h57m4.587211503s) [noise=true , address=145.79.227.43:4173 , piece_id=JLEI2NQBGOZEPTSNPBOANZM45MYSACNWEGJBGASPXN776W5JSUKQ, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.716292970 (595h57m4.587376015s) [noise=true , address=145.79.227.12:4781 , piece_id=NIZKPSQXJ46NUI7N6OUXGJGXMBOFF4CVWK4QJDN4MC4DIEN2BMPQ, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.716552004 (595h57m4.587635039s) [noise=true , address=145.79.227.17:4978 , piece_id=VKYFNTMMDPMUSIXIFMSPP42J54LW2ZQK32RF3BRJN6FVJUVDDLVQ, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.716731205 (595h57m4.58781424s) [noise=true , address=145.79.227.15:4902 , piece_id=X4NSOO736TN55MABZDFUW7EJXTKYOSHOBQRGDKVURTJST2PWWVPQ, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.716897352 (595h57m4.587980387s) [noise=true , address=145.79.227.35:4513 , piece_id=5QUQI6CPMVQRUYN3EY6PFDBVHQSXUIJGTBDTYFTBZ3S4HXNO2ARQ, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.717314065 (595h57m4.5883971s) [noise=true , address=145.79.227.33:4894 , piece_id=UNHQSHNBYVTNZTLE5XJCHR4BE4E2I53FY47DPPYM2SLRP272OPHQ, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.717782266 (595h57m4.588865301s) [noise=true , address=145.79.227.44:4222 , piece_id=I7P6DLKF6MXJYGHZS4WARPG5HHFOAD2C2OLVI3RCE2GCWXM5F6VA, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:05.725730011 (595h57m5.596813056s) [noise=true , address=64.49.13.27:4965 , piece_id=24SDWK4JBDZFHMVPNYEKPF3RJ4WERNSZY226RQHR4EC44ZDNHCGA, seg_pos={0 0}, upload=522411]: Done uploading piece. err: ecclient: upload failed (node:12WVhm2QyhAf1y6HXRpyChMLbDLhZJQHEE2kQpstZH2dkwNhd7P, address:64.49.13.27:4965): protocol: storage node overloaded, request limit: 2000 Dec 30 11:26:06.175006566 (595h57m6.046089601s) [upload=522411]: Next returned nil segment Dec 30 11:26:06.175030270 (595h57m6.046113305s) [upload=522411]: Waiting for error group managing segments... Dec 30 11:26:06.583282197 (595h57m6.454365252s) [noise=true , address=145.79.227.15:4902 , piece_id=X4NSOO736TN55MABZDFUW7EJXTKYOSHOBQRGDKVURTJST2PWWVPQ, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:06.583386316 (595h57m6.454469361s) [noise=true , address=145.79.227.17:4978 , piece_id=VKYFNTMMDPMUSIXIFMSPP42J54LW2ZQK32RF3BRJN6FVJUVDDLVQ, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:06.615530644 (595h57m6.486613699s) [noise=true , address=145.79.227.35:4513 , piece_id=5QUQI6CPMVQRUYN3EY6PFDBVHQSXUIJGTBDTYFTBZ3S4HXNO2ARQ, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:06.645605702 (595h57m6.516688737s) [noise=true , address=145.79.227.12:4790 , piece_id=RL4XZCKHUMLXRUCPKVR6ZNPPMOGEXW332ITCZJVHHLBZ765OOBWQ, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:06.648611996 (595h57m6.519695041s) [noise=true , address=145.79.227.12:4781 , piece_id=NIZKPSQXJ46NUI7N6OUXGJGXMBOFF4CVWK4QJDN4MC4DIEN2BMPQ, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:06.847054713 (595h57m6.718137759s) [noise=true , address=77.90.31.15:4710 , piece_id=P45UIV77TIYJJE4NKMKFU7DS75OVLXQ6ARNH52H3HSKXTO6FEXUA, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:06.851594768 (595h57m6.722677803s) [noise=true , address=77.90.31.19:4854 , piece_id=LFNN5M6F5V6PYGO2CRIEADFM2LSXSGZMZ2R75RL5TBZDRCPHQZXA, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:06.886396746 (595h57m6.757479791s) [noise=true , address=77.90.31.18:4817 , piece_id=ZOEZ5AFC3C2VG6ARKT62SNLG4J2NA2MFBSATAKTTVBAH55I7GUSQ, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:06.960886837 (595h57m6.831969883s) [noise=true , address=45.91.116.6:4001 , piece_id=LTUOVU6C624NQYYZCIE5XJAJQINSHZQRB6D7KVAZPJZD4SV46X4A, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:07.456143154 (595h57m7.327226189s) [noise=true , address=145.79.227.30:4784 , piece_id=ZL3HJBEYR36TLMRZI22QMZQRUYDKME3BZZLG7J3L2HXT6UCXB2VA, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:07.478562756 (595h57m7.349645791s) [noise=true , address=145.79.227.44:4222 , piece_id=I7P6DLKF6MXJYGHZS4WARPG5HHFOAD2C2OLVI3RCE2GCWXM5F6VA, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:07.490634482 (595h57m7.361717517s) [noise=true , address=145.79.227.43:4173 , piece_id=JLEI2NQBGOZEPTSNPBOANZM45MYSACNWEGJBGASPXN776W5JSUKQ, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:07.492668925 (595h57m7.36375196s) [noise=true , address=145.79.227.40:4062 , piece_id=TL3XD7TH57B7AUXCAFXSXA424X2YO332QN6FXWGEWS6N3KNBMHOQ, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:07.497087699 (595h57m7.368170734s) [noise=true , address=193.149.164.143:4697 , piece_id=KK6T67EEQX7F6J2E5NQANUBYUHDXZ2LDWSD2K4RMRUL2VTEFX3RA, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:07.497098951 (595h57m7.368181986s) [seg_pos={0 0}, upload=522411]: Segment reached optimal threshold of 14 pieces. Dec 30 11:26:07.497318599 (595h57m7.368401654s) [noise=true , address=64.49.13.169:4708 , piece_id=E7HPPLI24SYOM2POIZJPKFDWNC3CVA3FQ4DMATKB6M7277SOCI6A, seg_pos={0 0}, upload=522411]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:39422->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:39422->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->64.49.13.169:4708: use of closed network connection Dec 30 11:26:07.497330932 (595h57m7.368413977s) [noise=true , address=64.49.13.169:4708 , piece_id=E7HPPLI24SYOM2POIZJPKFDWNC3CVA3FQ4DMATKB6M7277SOCI6A, seg_pos={0 0}, upload=522411]: Long tail cancellation due to optimal threshold Dec 30 11:26:07.497470127 (595h57m7.368553172s) [noise=true , address=145.79.227.29:4961 , piece_id=JCJ4OYJKNI5UEHSHTZEJ4ZSVMXJJDEME4ZES6RTXEASIJB56WRZQ, seg_pos={0 0}, upload=522411]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGPLhM6NsdjKLkLCCLSeaNS8ikfnHPZxc3CCw9mLzSznbEoUa): protocol: expected piece hash; context canceled; EOF Dec 30 11:26:07.497481550 (595h57m7.368564595s) [noise=true , address=145.79.227.29:4961 , piece_id=JCJ4OYJKNI5UEHSHTZEJ4ZSVMXJJDEME4ZES6RTXEASIJB56WRZQ, seg_pos={0 0}, upload=522411]: Long tail cancellation due to optimal threshold Dec 30 11:26:07.497508240 (595h57m7.368591275s) [noise=true , address=145.79.227.33:4894 , piece_id=UNHQSHNBYVTNZTLE5XJCHR4BE4E2I53FY47DPPYM2SLRP272OPHQ, seg_pos={0 0}, upload=522411]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44): protocol: expected piece hash; context canceled; EOF Dec 30 11:26:07.497513561 (595h57m7.368596596s) [noise=true , address=145.79.227.33:4894 , piece_id=UNHQSHNBYVTNZTLE5XJCHR4BE4E2I53FY47DPPYM2SLRP272OPHQ, seg_pos={0 0}, upload=522411]: Long tail cancellation due to optimal threshold Dec 30 11:26:07.497530923 (595h57m7.368613968s) [seg_pos={0 0}, upload=522411]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:26:07.497672453 (595h57m7.368755498s) [upload=522411]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:26:07.497690037 (595h57m7.368773072s) [upload=522411]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:26:07.790670659 (595h57m7.661753704s) [upload=522411]: Done uploading segments... Dec 30 11:26:07.790697249 (595h57m7.661780284s) [upload=522411]: Upload finished. err: <nil> Duration: 46.18363355s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [X.X.X.....XX.X........X.X.XX.X.X.X.X...X.XX..XXX.X..XX......] duration: 3.764416933s trace: https://jaeger.dev.storj.io/trace/33eb99dd4a90f9b8 FAIL (2025-12-30 11:27:43 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.764441881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.......X..XX..X.X.X..........X....X.......X.....X..XXX....] duration: 6.664545483s trace: https://jaeger.dev.storj.io/trace/78c70eef2d7818c2 FAIL (2025-12-30 11:27:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 11:23:34.812028066 (595h54m34.683111101s) [upload=522378]: Starting upload Dec 30 11:23:34.812084975 (595h54m34.68316801s) [upload=522378]: Done starting upload Dec 30 11:23:34.812257994 (595h54m34.683341029s) [upload=522378]: Uploading segments... Dec 30 11:23:34.812298751 (595h54m34.683381787s) [upload=522378]: Got next segment. Inline: false Dec 30 11:23:34.812325192 (595h54m34.683408227s) [upload=522378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:23:34.812329200 (595h54m34.683412235s) [upload=522378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:23:35.056969122 (595h54m34.928052167s) [seg_pos={0 0}, upload=522378]: Begin upload segment... Dec 30 11:23:35.057030338 (595h54m34.928113383s) [seg_pos={0 0}, upload=522378]: Done begin upload segment. Dec 30 11:23:35.057052330 (595h54m34.928135365s) [noise=true , address=145.79.227.42:4123 , piece_id=LUVQKH4P4H5QETKOG26EQBVVM2O36D5YWAJEZSAXMAUPEUX4PKAQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.057065255 (595h54m34.9281483s) [noise=true , address=45.91.116.8:4045 , piece_id=ZIUANYQO3BSXHNEWFYAF3OBUYPOEXHCG7WJNSQ34MAZVX63IGYAA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.057547312 (595h54m34.928630348s) [noise=true , address=145.79.227.29:4941 , piece_id=JT56XVSIPQJKEC5MH5FDTAHONOCIIIDYEQSG7MVBRIBST2VK73KQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.057975839 (595h54m34.929058894s) [noise=true , address=145.79.227.10:4401 , piece_id=2QHEHKDQHC4N7TGQG66LB3LXKOIX7CPFQ3IRTXIJ6KZQZLHAUOTQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.058420205 (595h54m34.92950325s) [noise=true , address=45.91.116.5:4300 , piece_id=XVL2KMCJNIFR6S75Q33Z7JFZAEDUVO7PHOSE6YD5AHX3ONZD3P3A, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.058940144 (595h54m34.930023189s) [noise=true , address=145.79.227.25:4619 , piece_id=4EWAZCWAW62IB7VYTCTRIOAS4NNSB7VAWHL5AL4SIC2HP26EPUPA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.059370244 (595h54m34.930453279s) [noise=true , address=145.79.227.28:4726 , piece_id=QBQTDGKE6AN6M3A2OESAFYJWVBVT55D65YQSNUC5NR3VX7FKMT2Q, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.060849039 (595h54m34.931932074s) [noise=true , address=145.79.227.11:4731 , piece_id=4GQQJRSLOEVIHWOQZMAEOUYV5TU63GHI4ZRKEZ5AZQ4U7MYRE7GQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.062524930 (595h54m34.933607965s) [noise=true , address=145.79.227.14:4839 , piece_id=ZAGQ3R3XF7VVATDETNP3CTEIMWOU7QN44P5ZZQPTR2QKLJ4UFZKA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.064320279 (595h54m34.935403324s) [noise=true , address=145.79.227.35:4483 , piece_id=UGHXPY47MUMBRBWRDYHKQSUU2XX7CKOQA2WF5M54DOUTBW2FO27Q, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.066440937 (595h54m34.937523992s) [noise=true , address=145.79.227.50:4374 , piece_id=WAF22ATE3ZWJV7Q4B6AXAMVH5AQZ3M4AKRZZXAU7MVVAYYXTEL7Q, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.073033140 (595h54m34.944116185s) [noise=true , address=145.79.227.45:4229 , piece_id=WXCGOXRCNT2CDCBQPM235XBINQMCUNU2YJG3CDWYUWKPT55EFNQQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.073372616 (595h54m34.944455651s) [noise=true , address=145.79.227.7:4295 , piece_id=XTRYRQAC22NMIVKWJTOOL5DJRMBD2CSVVHWHF2QTP3KWLBPFUHVQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.074934210 (595h54m34.946017245s) [noise=true , address=45.91.116.7:4009 , piece_id=XPC2UPTHG433TCXRMGPC6VT7HOSF4BWBX5424VD24RFEZ2TVDTUA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.077002518 (595h54m34.948085553s) [noise=true , address=77.90.31.10:4520 , piece_id=TZOHYR6WUHVTN5CIGQU6K3RRUXZB7HGUM6IKO6ZFXCODYADD4Q6Q, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.077339841 (595h54m34.948422876s) [noise=true , address=145.79.227.36:4517 , piece_id=5QTQDPQMZUFYGUC4AW4DKZTMVD6UPJXHRDVCJWI33A7JKOB7G5PA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:36.767459832 (595h54m36.638542867s) [upload=522378]: Next returned nil segment Dec 30 11:23:36.767470743 (595h54m36.638553778s) [upload=522378]: Waiting for error group managing segments... Dec 30 11:23:37.039279379 (595h54m36.910362414s) [noise=true , address=145.79.227.11:4731 , piece_id=4GQQJRSLOEVIHWOQZMAEOUYV5TU63GHI4ZRKEZ5AZQ4U7MYRE7GQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.041352005 (595h54m36.91243504s) [noise=true , address=145.79.227.25:4619 , piece_id=4EWAZCWAW62IB7VYTCTRIOAS4NNSB7VAWHL5AL4SIC2HP26EPUPA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.060470380 (595h54m36.931553425s) [noise=true , address=145.79.227.14:4839 , piece_id=ZAGQ3R3XF7VVATDETNP3CTEIMWOU7QN44P5ZZQPTR2QKLJ4UFZKA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.063542990 (595h54m36.934626025s) [noise=true , address=145.79.227.28:4726 , piece_id=QBQTDGKE6AN6M3A2OESAFYJWVBVT55D65YQSNUC5NR3VX7FKMT2Q, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.075871886 (595h54m36.946954932s) [noise=true , address=145.79.227.10:4401 , piece_id=2QHEHKDQHC4N7TGQG66LB3LXKOIX7CPFQ3IRTXIJ6KZQZLHAUOTQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.077044910 (595h54m36.948127945s) [noise=true , address=145.79.227.42:4123 , piece_id=LUVQKH4P4H5QETKOG26EQBVVM2O36D5YWAJEZSAXMAUPEUX4PKAQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.079648879 (595h54m36.950731914s) [noise=true , address=145.79.227.50:4374 , piece_id=WAF22ATE3ZWJV7Q4B6AXAMVH5AQZ3M4AKRZZXAU7MVVAYYXTEL7Q, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.083355607 (595h54m36.954438652s) [noise=true , address=145.79.227.7:4295 , piece_id=XTRYRQAC22NMIVKWJTOOL5DJRMBD2CSVVHWHF2QTP3KWLBPFUHVQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.083522435 (595h54m36.95460547s) [noise=true , address=145.79.227.45:4229 , piece_id=WXCGOXRCNT2CDCBQPM235XBINQMCUNU2YJG3CDWYUWKPT55EFNQQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.152444123 (595h54m37.023527158s) [noise=true , address=145.79.227.36:4517 , piece_id=5QTQDPQMZUFYGUC4AW4DKZTMVD6UPJXHRDVCJWI33A7JKOB7G5PA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.383004515 (595h54m37.25408755s) [noise=true , address=45.91.116.8:4045 , piece_id=ZIUANYQO3BSXHNEWFYAF3OBUYPOEXHCG7WJNSQ34MAZVX63IGYAA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.475087265 (595h54m37.3461703s) [noise=true , address=77.90.31.10:4520 , piece_id=TZOHYR6WUHVTN5CIGQU6K3RRUXZB7HGUM6IKO6ZFXCODYADD4Q6Q, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.558983667 (595h54m37.430066702s) [noise=true , address=45.91.116.7:4009 , piece_id=XPC2UPTHG433TCXRMGPC6VT7HOSF4BWBX5424VD24RFEZ2TVDTUA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.559026078 (595h54m37.430109113s) [seg_pos={0 0}, upload=522378]: Segment reached optimal threshold of 13 pieces. Dec 30 11:23:37.559336128 (595h54m37.430419164s) [noise=true , address=145.79.227.35:4483 , piece_id=UGHXPY47MUMBRBWRDYHKQSUU2XX7CKOQA2WF5M54DOUTBW2FO27Q, seg_pos={0 0}, upload=522378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 30 11:23:37.559374401 (595h54m37.430457456s) [noise=true , address=145.79.227.35:4483 , piece_id=UGHXPY47MUMBRBWRDYHKQSUU2XX7CKOQA2WF5M54DOUTBW2FO27Q, seg_pos={0 0}, upload=522378]: Long tail cancellation due to optimal threshold Dec 30 11:23:37.559405030 (595h54m37.430488085s) [noise=true , address=45.91.116.5:4300 , piece_id=XVL2KMCJNIFR6S75Q33Z7JFZAEDUVO7PHOSE6YD5AHX3ONZD3P3A, seg_pos={0 0}, upload=522378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:34126->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:34126->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34126->45.91.116.5:4300: use of closed network connection Dec 30 11:23:37.559414177 (595h54m37.430497212s) [noise=true , address=45.91.116.5:4300 , piece_id=XVL2KMCJNIFR6S75Q33Z7JFZAEDUVO7PHOSE6YD5AHX3ONZD3P3A, seg_pos={0 0}, upload=522378]: Long tail cancellation due to optimal threshold Dec 30 11:23:37.559745157 (595h54m37.430828202s) [noise=true , address=145.79.227.29:4941 , piece_id=JT56XVSIPQJKEC5MH5FDTAHONOCIIIDYEQSG7MVBRIBST2VK73KQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 30 11:23:37.559754395 (595h54m37.43083744s) [noise=true , address=145.79.227.29:4941 , piece_id=JT56XVSIPQJKEC5MH5FDTAHONOCIIIDYEQSG7MVBRIBST2VK73KQ, seg_pos={0 0}, upload=522378]: Long tail cancellation due to optimal threshold Dec 30 11:23:37.559777999 (595h54m37.430861034s) [seg_pos={0 0}, upload=522378]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:23:37.559963924 (595h54m37.431046969s) [upload=522378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:23:37.559986977 (595h54m37.431070032s) [upload=522378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:23:37.849730184 (595h54m37.720813219s) [upload=522378]: Done uploading segments... Dec 30 11:23:37.849753237 (595h54m37.720836282s) [upload=522378]: Upload finished. err: <nil> Duration: 6.664573145s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [XX.X.X.....XX.X........X.X.XX.X.X.X.X...X.XX..XXX.X..XX.....] duration: 3.674284697s trace: https://jaeger.dev.storj.io/trace/33eb99dd4a90f9b8 FAIL (2025-12-30 11:27:15 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.674311828s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X.................X..........X...X...........X....X] duration: 1m0.003099669s trace: https://jaeger.dev.storj.io/trace/51943f4f02f0fbef FAIL (2025-12-30 11:27:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=522411]: Uploading piece... Dec 30 11:26:04.713753765 (595h57m4.58483681s) [noise=true , address=193.149.164.143:4697 , piece_id=KK6T67EEQX7F6J2E5NQANUBYUHDXZ2LDWSD2K4RMRUL2VTEFX3RA, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.713866410 (595h57m4.584949445s) [noise=true , address=64.49.13.169:4708 , piece_id=E7HPPLI24SYOM2POIZJPKFDWNC3CVA3FQ4DMATKB6M7277SOCI6A, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.714391520 (595h57m4.585474555s) [noise=true , address=77.90.31.18:4817 , piece_id=ZOEZ5AFC3C2VG6ARKT62SNLG4J2NA2MFBSATAKTTVBAH55I7GUSQ, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.714693134 (595h57m4.585776169s) [noise=true , address=145.79.227.29:4961 , piece_id=JCJ4OYJKNI5UEHSHTZEJ4ZSVMXJJDEME4ZES6RTXEASIJB56WRZQ, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.714791531 (595h57m4.585874566s) [noise=true , address=77.90.31.15:4710 , piece_id=P45UIV77TIYJJE4NKMKFU7DS75OVLXQ6ARNH52H3HSKXTO6FEXUA, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.715141928 (595h57m4.586224963s) [noise=true , address=145.79.227.30:4784 , piece_id=ZL3HJBEYR36TLMRZI22QMZQRUYDKME3BZZLG7J3L2HXT6UCXB2VA, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.715160304 (595h57m4.586243339s) [noise=true , address=145.79.227.40:4062 , piece_id=TL3XD7TH57B7AUXCAFXSXA424X2YO332QN6FXWGEWS6N3KNBMHOQ, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.715475855 (595h57m4.58655889s) [noise=true , address=145.79.227.12:4790 , piece_id=RL4XZCKHUMLXRUCPKVR6ZNPPMOGEXW332ITCZJVHHLBZ765OOBWQ, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.715638173 (595h57m4.586721218s) [noise=true , address=64.49.13.27:4965 , piece_id=24SDWK4JBDZFHMVPNYEKPF3RJ4WERNSZY226RQHR4EC44ZDNHCGA, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.715906915 (595h57m4.58698995s) [noise=true , address=45.91.116.6:4001 , piece_id=LTUOVU6C624NQYYZCIE5XJAJQINSHZQRB6D7KVAZPJZD4SV46X4A, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.716128207 (595h57m4.587211503s) [noise=true , address=145.79.227.43:4173 , piece_id=JLEI2NQBGOZEPTSNPBOANZM45MYSACNWEGJBGASPXN776W5JSUKQ, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.716292970 (595h57m4.587376015s) [noise=true , address=145.79.227.12:4781 , piece_id=NIZKPSQXJ46NUI7N6OUXGJGXMBOFF4CVWK4QJDN4MC4DIEN2BMPQ, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.716552004 (595h57m4.587635039s) [noise=true , address=145.79.227.17:4978 , piece_id=VKYFNTMMDPMUSIXIFMSPP42J54LW2ZQK32RF3BRJN6FVJUVDDLVQ, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.716731205 (595h57m4.58781424s) [noise=true , address=145.79.227.15:4902 , piece_id=X4NSOO736TN55MABZDFUW7EJXTKYOSHOBQRGDKVURTJST2PWWVPQ, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.716897352 (595h57m4.587980387s) [noise=true , address=145.79.227.35:4513 , piece_id=5QUQI6CPMVQRUYN3EY6PFDBVHQSXUIJGTBDTYFTBZ3S4HXNO2ARQ, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.717314065 (595h57m4.5883971s) [noise=true , address=145.79.227.33:4894 , piece_id=UNHQSHNBYVTNZTLE5XJCHR4BE4E2I53FY47DPPYM2SLRP272OPHQ, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:04.717782266 (595h57m4.588865301s) [noise=true , address=145.79.227.44:4222 , piece_id=I7P6DLKF6MXJYGHZS4WARPG5HHFOAD2C2OLVI3RCE2GCWXM5F6VA, seg_pos={0 0}, upload=522411]: Uploading piece... Dec 30 11:26:05.725730011 (595h57m5.596813056s) [noise=true , address=64.49.13.27:4965 , piece_id=24SDWK4JBDZFHMVPNYEKPF3RJ4WERNSZY226RQHR4EC44ZDNHCGA, seg_pos={0 0}, upload=522411]: Done uploading piece. err: ecclient: upload failed (node:12WVhm2QyhAf1y6HXRpyChMLbDLhZJQHEE2kQpstZH2dkwNhd7P, address:64.49.13.27:4965): protocol: storage node overloaded, request limit: 2000 Dec 30 11:26:06.175006566 (595h57m6.046089601s) [upload=522411]: Next returned nil segment Dec 30 11:26:06.175030270 (595h57m6.046113305s) [upload=522411]: Waiting for error group managing segments... Dec 30 11:26:06.583282197 (595h57m6.454365252s) [noise=true , address=145.79.227.15:4902 , piece_id=X4NSOO736TN55MABZDFUW7EJXTKYOSHOBQRGDKVURTJST2PWWVPQ, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:06.583386316 (595h57m6.454469361s) [noise=true , address=145.79.227.17:4978 , piece_id=VKYFNTMMDPMUSIXIFMSPP42J54LW2ZQK32RF3BRJN6FVJUVDDLVQ, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:06.615530644 (595h57m6.486613699s) [noise=true , address=145.79.227.35:4513 , piece_id=5QUQI6CPMVQRUYN3EY6PFDBVHQSXUIJGTBDTYFTBZ3S4HXNO2ARQ, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:06.645605702 (595h57m6.516688737s) [noise=true , address=145.79.227.12:4790 , piece_id=RL4XZCKHUMLXRUCPKVR6ZNPPMOGEXW332ITCZJVHHLBZ765OOBWQ, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:06.648611996 (595h57m6.519695041s) [noise=true , address=145.79.227.12:4781 , piece_id=NIZKPSQXJ46NUI7N6OUXGJGXMBOFF4CVWK4QJDN4MC4DIEN2BMPQ, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:06.847054713 (595h57m6.718137759s) [noise=true , address=77.90.31.15:4710 , piece_id=P45UIV77TIYJJE4NKMKFU7DS75OVLXQ6ARNH52H3HSKXTO6FEXUA, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:06.851594768 (595h57m6.722677803s) [noise=true , address=77.90.31.19:4854 , piece_id=LFNN5M6F5V6PYGO2CRIEADFM2LSXSGZMZ2R75RL5TBZDRCPHQZXA, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:06.886396746 (595h57m6.757479791s) [noise=true , address=77.90.31.18:4817 , piece_id=ZOEZ5AFC3C2VG6ARKT62SNLG4J2NA2MFBSATAKTTVBAH55I7GUSQ, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:06.960886837 (595h57m6.831969883s) [noise=true , address=45.91.116.6:4001 , piece_id=LTUOVU6C624NQYYZCIE5XJAJQINSHZQRB6D7KVAZPJZD4SV46X4A, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:07.456143154 (595h57m7.327226189s) [noise=true , address=145.79.227.30:4784 , piece_id=ZL3HJBEYR36TLMRZI22QMZQRUYDKME3BZZLG7J3L2HXT6UCXB2VA, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:07.478562756 (595h57m7.349645791s) [noise=true , address=145.79.227.44:4222 , piece_id=I7P6DLKF6MXJYGHZS4WARPG5HHFOAD2C2OLVI3RCE2GCWXM5F6VA, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:07.490634482 (595h57m7.361717517s) [noise=true , address=145.79.227.43:4173 , piece_id=JLEI2NQBGOZEPTSNPBOANZM45MYSACNWEGJBGASPXN776W5JSUKQ, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:07.492668925 (595h57m7.36375196s) [noise=true , address=145.79.227.40:4062 , piece_id=TL3XD7TH57B7AUXCAFXSXA424X2YO332QN6FXWGEWS6N3KNBMHOQ, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:07.497087699 (595h57m7.368170734s) [noise=true , address=193.149.164.143:4697 , piece_id=KK6T67EEQX7F6J2E5NQANUBYUHDXZ2LDWSD2K4RMRUL2VTEFX3RA, seg_pos={0 0}, upload=522411]: Done uploading piece. err: <nil> Dec 30 11:26:07.497098951 (595h57m7.368181986s) [seg_pos={0 0}, upload=522411]: Segment reached optimal threshold of 14 pieces. Dec 30 11:26:07.497318599 (595h57m7.368401654s) [noise=true , address=64.49.13.169:4708 , piece_id=E7HPPLI24SYOM2POIZJPKFDWNC3CVA3FQ4DMATKB6M7277SOCI6A, seg_pos={0 0}, upload=522411]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:39422->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:39422->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->64.49.13.169:4708: use of closed network connection Dec 30 11:26:07.497330932 (595h57m7.368413977s) [noise=true , address=64.49.13.169:4708 , piece_id=E7HPPLI24SYOM2POIZJPKFDWNC3CVA3FQ4DMATKB6M7277SOCI6A, seg_pos={0 0}, upload=522411]: Long tail cancellation due to optimal threshold Dec 30 11:26:07.497470127 (595h57m7.368553172s) [noise=true , address=145.79.227.29:4961 , piece_id=JCJ4OYJKNI5UEHSHTZEJ4ZSVMXJJDEME4ZES6RTXEASIJB56WRZQ, seg_pos={0 0}, upload=522411]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGPLhM6NsdjKLkLCCLSeaNS8ikfnHPZxc3CCw9mLzSznbEoUa): protocol: expected piece hash; context canceled; EOF Dec 30 11:26:07.497481550 (595h57m7.368564595s) [noise=true , address=145.79.227.29:4961 , piece_id=JCJ4OYJKNI5UEHSHTZEJ4ZSVMXJJDEME4ZES6RTXEASIJB56WRZQ, seg_pos={0 0}, upload=522411]: Long tail cancellation due to optimal threshold Dec 30 11:26:07.497508240 (595h57m7.368591275s) [noise=true , address=145.79.227.33:4894 , piece_id=UNHQSHNBYVTNZTLE5XJCHR4BE4E2I53FY47DPPYM2SLRP272OPHQ, seg_pos={0 0}, upload=522411]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44): protocol: expected piece hash; context canceled; EOF Dec 30 11:26:07.497513561 (595h57m7.368596596s) [noise=true , address=145.79.227.33:4894 , piece_id=UNHQSHNBYVTNZTLE5XJCHR4BE4E2I53FY47DPPYM2SLRP272OPHQ, seg_pos={0 0}, upload=522411]: Long tail cancellation due to optimal threshold Dec 30 11:26:07.497530923 (595h57m7.368613968s) [seg_pos={0 0}, upload=522411]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:26:07.497672453 (595h57m7.368755498s) [upload=522411]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:26:07.497690037 (595h57m7.368773072s) [upload=522411]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:26:07.790670659 (595h57m7.661753704s) [upload=522411]: Done uploading segments... Dec 30 11:26:07.790697249 (595h57m7.661780284s) [upload=522411]: Upload finished. err: <nil> Duration: 1m0.003112874s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.XX.X.X.....XX.X........X.X.XX.X.X.X.X...X.XX..XXX.X..XX....] duration: 3.522802534s trace: https://jaeger.dev.storj.io/trace/33eb99dd4a90f9b8 FAIL (2025-12-30 11:26:34 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.522836348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X.......X..XX..X.X.X..........X....X.......X.....X..XXX...] duration: 6.37455228s trace: https://jaeger.dev.storj.io/trace/78c70eef2d7818c2 FAIL (2025-12-30 11:26:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 11:23:34.812028066 (595h54m34.683111101s) [upload=522378]: Starting upload Dec 30 11:23:34.812084975 (595h54m34.68316801s) [upload=522378]: Done starting upload Dec 30 11:23:34.812257994 (595h54m34.683341029s) [upload=522378]: Uploading segments... Dec 30 11:23:34.812298751 (595h54m34.683381787s) [upload=522378]: Got next segment. Inline: false Dec 30 11:23:34.812325192 (595h54m34.683408227s) [upload=522378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:23:34.812329200 (595h54m34.683412235s) [upload=522378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:23:35.056969122 (595h54m34.928052167s) [seg_pos={0 0}, upload=522378]: Begin upload segment... Dec 30 11:23:35.057030338 (595h54m34.928113383s) [seg_pos={0 0}, upload=522378]: Done begin upload segment. Dec 30 11:23:35.057052330 (595h54m34.928135365s) [noise=true , address=145.79.227.42:4123 , piece_id=LUVQKH4P4H5QETKOG26EQBVVM2O36D5YWAJEZSAXMAUPEUX4PKAQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.057065255 (595h54m34.9281483s) [noise=true , address=45.91.116.8:4045 , piece_id=ZIUANYQO3BSXHNEWFYAF3OBUYPOEXHCG7WJNSQ34MAZVX63IGYAA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.057547312 (595h54m34.928630348s) [noise=true , address=145.79.227.29:4941 , piece_id=JT56XVSIPQJKEC5MH5FDTAHONOCIIIDYEQSG7MVBRIBST2VK73KQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.057975839 (595h54m34.929058894s) [noise=true , address=145.79.227.10:4401 , piece_id=2QHEHKDQHC4N7TGQG66LB3LXKOIX7CPFQ3IRTXIJ6KZQZLHAUOTQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.058420205 (595h54m34.92950325s) [noise=true , address=45.91.116.5:4300 , piece_id=XVL2KMCJNIFR6S75Q33Z7JFZAEDUVO7PHOSE6YD5AHX3ONZD3P3A, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.058940144 (595h54m34.930023189s) [noise=true , address=145.79.227.25:4619 , piece_id=4EWAZCWAW62IB7VYTCTRIOAS4NNSB7VAWHL5AL4SIC2HP26EPUPA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.059370244 (595h54m34.930453279s) [noise=true , address=145.79.227.28:4726 , piece_id=QBQTDGKE6AN6M3A2OESAFYJWVBVT55D65YQSNUC5NR3VX7FKMT2Q, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.060849039 (595h54m34.931932074s) [noise=true , address=145.79.227.11:4731 , piece_id=4GQQJRSLOEVIHWOQZMAEOUYV5TU63GHI4ZRKEZ5AZQ4U7MYRE7GQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.062524930 (595h54m34.933607965s) [noise=true , address=145.79.227.14:4839 , piece_id=ZAGQ3R3XF7VVATDETNP3CTEIMWOU7QN44P5ZZQPTR2QKLJ4UFZKA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.064320279 (595h54m34.935403324s) [noise=true , address=145.79.227.35:4483 , piece_id=UGHXPY47MUMBRBWRDYHKQSUU2XX7CKOQA2WF5M54DOUTBW2FO27Q, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.066440937 (595h54m34.937523992s) [noise=true , address=145.79.227.50:4374 , piece_id=WAF22ATE3ZWJV7Q4B6AXAMVH5AQZ3M4AKRZZXAU7MVVAYYXTEL7Q, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.073033140 (595h54m34.944116185s) [noise=true , address=145.79.227.45:4229 , piece_id=WXCGOXRCNT2CDCBQPM235XBINQMCUNU2YJG3CDWYUWKPT55EFNQQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.073372616 (595h54m34.944455651s) [noise=true , address=145.79.227.7:4295 , piece_id=XTRYRQAC22NMIVKWJTOOL5DJRMBD2CSVVHWHF2QTP3KWLBPFUHVQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.074934210 (595h54m34.946017245s) [noise=true , address=45.91.116.7:4009 , piece_id=XPC2UPTHG433TCXRMGPC6VT7HOSF4BWBX5424VD24RFEZ2TVDTUA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.077002518 (595h54m34.948085553s) [noise=true , address=77.90.31.10:4520 , piece_id=TZOHYR6WUHVTN5CIGQU6K3RRUXZB7HGUM6IKO6ZFXCODYADD4Q6Q, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.077339841 (595h54m34.948422876s) [noise=true , address=145.79.227.36:4517 , piece_id=5QTQDPQMZUFYGUC4AW4DKZTMVD6UPJXHRDVCJWI33A7JKOB7G5PA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:36.767459832 (595h54m36.638542867s) [upload=522378]: Next returned nil segment Dec 30 11:23:36.767470743 (595h54m36.638553778s) [upload=522378]: Waiting for error group managing segments... Dec 30 11:23:37.039279379 (595h54m36.910362414s) [noise=true , address=145.79.227.11:4731 , piece_id=4GQQJRSLOEVIHWOQZMAEOUYV5TU63GHI4ZRKEZ5AZQ4U7MYRE7GQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.041352005 (595h54m36.91243504s) [noise=true , address=145.79.227.25:4619 , piece_id=4EWAZCWAW62IB7VYTCTRIOAS4NNSB7VAWHL5AL4SIC2HP26EPUPA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.060470380 (595h54m36.931553425s) [noise=true , address=145.79.227.14:4839 , piece_id=ZAGQ3R3XF7VVATDETNP3CTEIMWOU7QN44P5ZZQPTR2QKLJ4UFZKA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.063542990 (595h54m36.934626025s) [noise=true , address=145.79.227.28:4726 , piece_id=QBQTDGKE6AN6M3A2OESAFYJWVBVT55D65YQSNUC5NR3VX7FKMT2Q, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.075871886 (595h54m36.946954932s) [noise=true , address=145.79.227.10:4401 , piece_id=2QHEHKDQHC4N7TGQG66LB3LXKOIX7CPFQ3IRTXIJ6KZQZLHAUOTQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.077044910 (595h54m36.948127945s) [noise=true , address=145.79.227.42:4123 , piece_id=LUVQKH4P4H5QETKOG26EQBVVM2O36D5YWAJEZSAXMAUPEUX4PKAQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.079648879 (595h54m36.950731914s) [noise=true , address=145.79.227.50:4374 , piece_id=WAF22ATE3ZWJV7Q4B6AXAMVH5AQZ3M4AKRZZXAU7MVVAYYXTEL7Q, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.083355607 (595h54m36.954438652s) [noise=true , address=145.79.227.7:4295 , piece_id=XTRYRQAC22NMIVKWJTOOL5DJRMBD2CSVVHWHF2QTP3KWLBPFUHVQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.083522435 (595h54m36.95460547s) [noise=true , address=145.79.227.45:4229 , piece_id=WXCGOXRCNT2CDCBQPM235XBINQMCUNU2YJG3CDWYUWKPT55EFNQQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.152444123 (595h54m37.023527158s) [noise=true , address=145.79.227.36:4517 , piece_id=5QTQDPQMZUFYGUC4AW4DKZTMVD6UPJXHRDVCJWI33A7JKOB7G5PA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.383004515 (595h54m37.25408755s) [noise=true , address=45.91.116.8:4045 , piece_id=ZIUANYQO3BSXHNEWFYAF3OBUYPOEXHCG7WJNSQ34MAZVX63IGYAA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.475087265 (595h54m37.3461703s) [noise=true , address=77.90.31.10:4520 , piece_id=TZOHYR6WUHVTN5CIGQU6K3RRUXZB7HGUM6IKO6ZFXCODYADD4Q6Q, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.558983667 (595h54m37.430066702s) [noise=true , address=45.91.116.7:4009 , piece_id=XPC2UPTHG433TCXRMGPC6VT7HOSF4BWBX5424VD24RFEZ2TVDTUA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.559026078 (595h54m37.430109113s) [seg_pos={0 0}, upload=522378]: Segment reached optimal threshold of 13 pieces. Dec 30 11:23:37.559336128 (595h54m37.430419164s) [noise=true , address=145.79.227.35:4483 , piece_id=UGHXPY47MUMBRBWRDYHKQSUU2XX7CKOQA2WF5M54DOUTBW2FO27Q, seg_pos={0 0}, upload=522378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 30 11:23:37.559374401 (595h54m37.430457456s) [noise=true , address=145.79.227.35:4483 , piece_id=UGHXPY47MUMBRBWRDYHKQSUU2XX7CKOQA2WF5M54DOUTBW2FO27Q, seg_pos={0 0}, upload=522378]: Long tail cancellation due to optimal threshold Dec 30 11:23:37.559405030 (595h54m37.430488085s) [noise=true , address=45.91.116.5:4300 , piece_id=XVL2KMCJNIFR6S75Q33Z7JFZAEDUVO7PHOSE6YD5AHX3ONZD3P3A, seg_pos={0 0}, upload=522378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:34126->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:34126->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34126->45.91.116.5:4300: use of closed network connection Dec 30 11:23:37.559414177 (595h54m37.430497212s) [noise=true , address=45.91.116.5:4300 , piece_id=XVL2KMCJNIFR6S75Q33Z7JFZAEDUVO7PHOSE6YD5AHX3ONZD3P3A, seg_pos={0 0}, upload=522378]: Long tail cancellation due to optimal threshold Dec 30 11:23:37.559745157 (595h54m37.430828202s) [noise=true , address=145.79.227.29:4941 , piece_id=JT56XVSIPQJKEC5MH5FDTAHONOCIIIDYEQSG7MVBRIBST2VK73KQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 30 11:23:37.559754395 (595h54m37.43083744s) [noise=true , address=145.79.227.29:4941 , piece_id=JT56XVSIPQJKEC5MH5FDTAHONOCIIIDYEQSG7MVBRIBST2VK73KQ, seg_pos={0 0}, upload=522378]: Long tail cancellation due to optimal threshold Dec 30 11:23:37.559777999 (595h54m37.430861034s) [seg_pos={0 0}, upload=522378]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:23:37.559963924 (595h54m37.431046969s) [upload=522378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:23:37.559986977 (595h54m37.431070032s) [upload=522378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:23:37.849730184 (595h54m37.720813219s) [upload=522378]: Done uploading segments... Dec 30 11:23:37.849753237 (595h54m37.720836282s) [upload=522378]: Upload finished. err: <nil> Duration: 6.374577157s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...X.......X..XX..X.X.X..........X....X.......X.....X..XXX..] duration: 5.711206944s trace: https://jaeger.dev.storj.io/trace/78c70eef2d7818c2 FAIL (2025-12-30 11:26:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 11:23:34.812028066 (595h54m34.683111101s) [upload=522378]: Starting upload Dec 30 11:23:34.812084975 (595h54m34.68316801s) [upload=522378]: Done starting upload Dec 30 11:23:34.812257994 (595h54m34.683341029s) [upload=522378]: Uploading segments... Dec 30 11:23:34.812298751 (595h54m34.683381787s) [upload=522378]: Got next segment. Inline: false Dec 30 11:23:34.812325192 (595h54m34.683408227s) [upload=522378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:23:34.812329200 (595h54m34.683412235s) [upload=522378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:23:35.056969122 (595h54m34.928052167s) [seg_pos={0 0}, upload=522378]: Begin upload segment... Dec 30 11:23:35.057030338 (595h54m34.928113383s) [seg_pos={0 0}, upload=522378]: Done begin upload segment. Dec 30 11:23:35.057052330 (595h54m34.928135365s) [noise=true , address=145.79.227.42:4123 , piece_id=LUVQKH4P4H5QETKOG26EQBVVM2O36D5YWAJEZSAXMAUPEUX4PKAQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.057065255 (595h54m34.9281483s) [noise=true , address=45.91.116.8:4045 , piece_id=ZIUANYQO3BSXHNEWFYAF3OBUYPOEXHCG7WJNSQ34MAZVX63IGYAA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.057547312 (595h54m34.928630348s) [noise=true , address=145.79.227.29:4941 , piece_id=JT56XVSIPQJKEC5MH5FDTAHONOCIIIDYEQSG7MVBRIBST2VK73KQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.057975839 (595h54m34.929058894s) [noise=true , address=145.79.227.10:4401 , piece_id=2QHEHKDQHC4N7TGQG66LB3LXKOIX7CPFQ3IRTXIJ6KZQZLHAUOTQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.058420205 (595h54m34.92950325s) [noise=true , address=45.91.116.5:4300 , piece_id=XVL2KMCJNIFR6S75Q33Z7JFZAEDUVO7PHOSE6YD5AHX3ONZD3P3A, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.058940144 (595h54m34.930023189s) [noise=true , address=145.79.227.25:4619 , piece_id=4EWAZCWAW62IB7VYTCTRIOAS4NNSB7VAWHL5AL4SIC2HP26EPUPA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.059370244 (595h54m34.930453279s) [noise=true , address=145.79.227.28:4726 , piece_id=QBQTDGKE6AN6M3A2OESAFYJWVBVT55D65YQSNUC5NR3VX7FKMT2Q, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.060849039 (595h54m34.931932074s) [noise=true , address=145.79.227.11:4731 , piece_id=4GQQJRSLOEVIHWOQZMAEOUYV5TU63GHI4ZRKEZ5AZQ4U7MYRE7GQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.062524930 (595h54m34.933607965s) [noise=true , address=145.79.227.14:4839 , piece_id=ZAGQ3R3XF7VVATDETNP3CTEIMWOU7QN44P5ZZQPTR2QKLJ4UFZKA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.064320279 (595h54m34.935403324s) [noise=true , address=145.79.227.35:4483 , piece_id=UGHXPY47MUMBRBWRDYHKQSUU2XX7CKOQA2WF5M54DOUTBW2FO27Q, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.066440937 (595h54m34.937523992s) [noise=true , address=145.79.227.50:4374 , piece_id=WAF22ATE3ZWJV7Q4B6AXAMVH5AQZ3M4AKRZZXAU7MVVAYYXTEL7Q, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.073033140 (595h54m34.944116185s) [noise=true , address=145.79.227.45:4229 , piece_id=WXCGOXRCNT2CDCBQPM235XBINQMCUNU2YJG3CDWYUWKPT55EFNQQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.073372616 (595h54m34.944455651s) [noise=true , address=145.79.227.7:4295 , piece_id=XTRYRQAC22NMIVKWJTOOL5DJRMBD2CSVVHWHF2QTP3KWLBPFUHVQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.074934210 (595h54m34.946017245s) [noise=true , address=45.91.116.7:4009 , piece_id=XPC2UPTHG433TCXRMGPC6VT7HOSF4BWBX5424VD24RFEZ2TVDTUA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.077002518 (595h54m34.948085553s) [noise=true , address=77.90.31.10:4520 , piece_id=TZOHYR6WUHVTN5CIGQU6K3RRUXZB7HGUM6IKO6ZFXCODYADD4Q6Q, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.077339841 (595h54m34.948422876s) [noise=true , address=145.79.227.36:4517 , piece_id=5QTQDPQMZUFYGUC4AW4DKZTMVD6UPJXHRDVCJWI33A7JKOB7G5PA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:36.767459832 (595h54m36.638542867s) [upload=522378]: Next returned nil segment Dec 30 11:23:36.767470743 (595h54m36.638553778s) [upload=522378]: Waiting for error group managing segments... Dec 30 11:23:37.039279379 (595h54m36.910362414s) [noise=true , address=145.79.227.11:4731 , piece_id=4GQQJRSLOEVIHWOQZMAEOUYV5TU63GHI4ZRKEZ5AZQ4U7MYRE7GQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.041352005 (595h54m36.91243504s) [noise=true , address=145.79.227.25:4619 , piece_id=4EWAZCWAW62IB7VYTCTRIOAS4NNSB7VAWHL5AL4SIC2HP26EPUPA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.060470380 (595h54m36.931553425s) [noise=true , address=145.79.227.14:4839 , piece_id=ZAGQ3R3XF7VVATDETNP3CTEIMWOU7QN44P5ZZQPTR2QKLJ4UFZKA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.063542990 (595h54m36.934626025s) [noise=true , address=145.79.227.28:4726 , piece_id=QBQTDGKE6AN6M3A2OESAFYJWVBVT55D65YQSNUC5NR3VX7FKMT2Q, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.075871886 (595h54m36.946954932s) [noise=true , address=145.79.227.10:4401 , piece_id=2QHEHKDQHC4N7TGQG66LB3LXKOIX7CPFQ3IRTXIJ6KZQZLHAUOTQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.077044910 (595h54m36.948127945s) [noise=true , address=145.79.227.42:4123 , piece_id=LUVQKH4P4H5QETKOG26EQBVVM2O36D5YWAJEZSAXMAUPEUX4PKAQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.079648879 (595h54m36.950731914s) [noise=true , address=145.79.227.50:4374 , piece_id=WAF22ATE3ZWJV7Q4B6AXAMVH5AQZ3M4AKRZZXAU7MVVAYYXTEL7Q, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.083355607 (595h54m36.954438652s) [noise=true , address=145.79.227.7:4295 , piece_id=XTRYRQAC22NMIVKWJTOOL5DJRMBD2CSVVHWHF2QTP3KWLBPFUHVQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.083522435 (595h54m36.95460547s) [noise=true , address=145.79.227.45:4229 , piece_id=WXCGOXRCNT2CDCBQPM235XBINQMCUNU2YJG3CDWYUWKPT55EFNQQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.152444123 (595h54m37.023527158s) [noise=true , address=145.79.227.36:4517 , piece_id=5QTQDPQMZUFYGUC4AW4DKZTMVD6UPJXHRDVCJWI33A7JKOB7G5PA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.383004515 (595h54m37.25408755s) [noise=true , address=45.91.116.8:4045 , piece_id=ZIUANYQO3BSXHNEWFYAF3OBUYPOEXHCG7WJNSQ34MAZVX63IGYAA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.475087265 (595h54m37.3461703s) [noise=true , address=77.90.31.10:4520 , piece_id=TZOHYR6WUHVTN5CIGQU6K3RRUXZB7HGUM6IKO6ZFXCODYADD4Q6Q, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.558983667 (595h54m37.430066702s) [noise=true , address=45.91.116.7:4009 , piece_id=XPC2UPTHG433TCXRMGPC6VT7HOSF4BWBX5424VD24RFEZ2TVDTUA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.559026078 (595h54m37.430109113s) [seg_pos={0 0}, upload=522378]: Segment reached optimal threshold of 13 pieces. Dec 30 11:23:37.559336128 (595h54m37.430419164s) [noise=true , address=145.79.227.35:4483 , piece_id=UGHXPY47MUMBRBWRDYHKQSUU2XX7CKOQA2WF5M54DOUTBW2FO27Q, seg_pos={0 0}, upload=522378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 30 11:23:37.559374401 (595h54m37.430457456s) [noise=true , address=145.79.227.35:4483 , piece_id=UGHXPY47MUMBRBWRDYHKQSUU2XX7CKOQA2WF5M54DOUTBW2FO27Q, seg_pos={0 0}, upload=522378]: Long tail cancellation due to optimal threshold Dec 30 11:23:37.559405030 (595h54m37.430488085s) [noise=true , address=45.91.116.5:4300 , piece_id=XVL2KMCJNIFR6S75Q33Z7JFZAEDUVO7PHOSE6YD5AHX3ONZD3P3A, seg_pos={0 0}, upload=522378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:34126->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:34126->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34126->45.91.116.5:4300: use of closed network connection Dec 30 11:23:37.559414177 (595h54m37.430497212s) [noise=true , address=45.91.116.5:4300 , piece_id=XVL2KMCJNIFR6S75Q33Z7JFZAEDUVO7PHOSE6YD5AHX3ONZD3P3A, seg_pos={0 0}, upload=522378]: Long tail cancellation due to optimal threshold Dec 30 11:23:37.559745157 (595h54m37.430828202s) [noise=true , address=145.79.227.29:4941 , piece_id=JT56XVSIPQJKEC5MH5FDTAHONOCIIIDYEQSG7MVBRIBST2VK73KQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 30 11:23:37.559754395 (595h54m37.43083744s) [noise=true , address=145.79.227.29:4941 , piece_id=JT56XVSIPQJKEC5MH5FDTAHONOCIIIDYEQSG7MVBRIBST2VK73KQ, seg_pos={0 0}, upload=522378]: Long tail cancellation due to optimal threshold Dec 30 11:23:37.559777999 (595h54m37.430861034s) [seg_pos={0 0}, upload=522378]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:23:37.559963924 (595h54m37.431046969s) [upload=522378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:23:37.559986977 (595h54m37.431070032s) [upload=522378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:23:37.849730184 (595h54m37.720813219s) [upload=522378]: Done uploading segments... Dec 30 11:23:37.849753237 (595h54m37.720836282s) [upload=522378]: Upload finished. err: <nil> Duration: 5.711231901s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X.................X..........X...X...........X....] duration: 59.710935778s trace: https://jaeger.dev.storj.io/trace/320f2449de9aa447 FAIL (2025-12-30 11:26:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8.212630018 (595h52m28.083713053s) [noise=true , address=145.79.227.44:4206 , piece_id=7QM25KZUSPAD4GTIDR67OJNMAEKLCBZRLUK3GZDB7TO2ZI5TEREA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.213211245 (595h52m28.08429428s) [noise=true , address=145.79.227.30:4788 , piece_id=TWTEPHC7KMREU47W7NCYNL6DEZ5XHPYD5QIKQZN6HZKR3OBPUT4A, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.213600286 (595h52m28.084683321s) [noise=true , address=77.90.31.11:4575 , piece_id=O3DCIOB6TOACOAKWBFE5IZTTYKUSWJBOKHC7LC2CRACYYBHXAHMA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.213779527 (595h52m28.084862562s) [noise=true , address=45.91.116.8:4049 , piece_id=XBRESQJASTTIPV3G2SJUGXKPJP4UD22X7B4QDP5YJUVTSLMJVTNA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.213888514 (595h52m28.084971559s) [noise=true , address=145.79.227.34:4937 , piece_id=4NXNJDUCZM5KLWS3K4ABDHODPCJHJYU42D2W5ON4LRS5ZM4LREHQ, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214146827 (595h52m28.085229862s) [noise=true , address=193.149.164.143:4703 , piece_id=EBPBXTU4XLXLN5FEV3WC3JMXBGWRSKRZOIOLIUP5ANY5AYTPLEIQ, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214153439 (595h52m28.085236474s) [noise=true , address=145.79.227.40:4057 , piece_id=JKX3DD2VZGTTXJDYLJVSOJ4HEAAGUZIPG7AMLT5HX4GLLJ7EJDLA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214505840 (595h52m28.085588875s) [noise=true , address=145.79.227.35:4489 , piece_id=3I25VU7FHAW24CCOAWF7YF6AIUA5LS37UL2BX7TJ4LBXINZT3ZCQ, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214601683 (595h52m28.085684718s) [noise=true , address=77.90.31.10:4529 , piece_id=433RPSCXRLK7I7746NZN33AKHX7QVW2QK6FU4YJMUXQXM6UW2KFA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214836058 (595h52m28.085919093s) [noise=true , address=145.79.227.33:4876 , piece_id=ULKTOEKRBWK3WSNO6YTJYXD7WRZTCKD2TVRXIGCT66SJA2HKMXEA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214886896 (595h52m28.085969931s) [noise=true , address=77.90.31.14:4683 , piece_id=DD3WJKOZA34IDFGTBRZFCMYBOU3EWARB2772KQHIVBIDS6XBG73A, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215143213 (595h52m28.086226248s) [noise=true , address=77.90.31.17:4790 , piece_id=7AQKSCDTY4TIVXYUIE2XXXUB2R2M7LVDA6W3RHZBVPAUDUHYX4PQ, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215224017 (595h52m28.086307052s) [noise=true , address=145.79.227.9:4385 , piece_id=O35GQDZODEC6HZNMUZMT47OLZFIMFGIYHYNPIIC66KW6MUJHVNPA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215431352 (595h52m28.086514387s) [noise=true , address=64.49.13.27:4965 , piece_id=V5NCPS3EFV7NABYT5KZ7E4TEQHICP4RFK5GKCHMMT45PJ2CE4S4Q, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215543906 (595h52m28.086626951s) [noise=true , address=64.49.13.26:4954 , piece_id=ZN2G36BDM3WARBVKBOCMGGKFQG7XKOFRIRVY3VKW4UMNSL4QHSTA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215913380 (595h52m28.086996415s) [noise=true , address=64.49.13.169:4709 , piece_id=L6DBEGT6YAR5J4I3FRIMLCDKAHJ5HTUGH7VATKOM5NXF2L3FYM7A, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.216329021 (595h52m28.087412056s) [noise=true , address=145.79.227.38:4995 , piece_id=XOROFSBZOSPYSC6LSNZJMSRS4KYNH7JMH3E542G77L3ZQ3SADW7A, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:29.220053219 (595h52m29.091136254s) [noise=true , address=64.49.13.27:4965 , piece_id=V5NCPS3EFV7NABYT5KZ7E4TEQHICP4RFK5GKCHMMT45PJ2CE4S4Q, seg_pos={0 0}, upload=522349]: Done uploading piece. err: ecclient: upload failed (node:12WVhm2QyhAf1y6HXRpyChMLbDLhZJQHEE2kQpstZH2dkwNhd7P, address:64.49.13.27:4965): protocol: storage node overloaded, request limit: 2000 Dec 30 11:21:29.637185454 (595h52m29.508268499s) [upload=522349]: Next returned nil segment Dec 30 11:21:29.637201474 (595h52m29.508284539s) [upload=522349]: Waiting for error group managing segments... Dec 30 11:21:30.066275167 (595h52m29.937358302s) [noise=true , address=145.79.227.9:4385 , piece_id=O35GQDZODEC6HZNMUZMT47OLZFIMFGIYHYNPIIC66KW6MUJHVNPA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.073548216 (595h52m29.944631251s) [noise=true , address=64.49.13.26:4954 , piece_id=ZN2G36BDM3WARBVKBOCMGGKFQG7XKOFRIRVY3VKW4UMNSL4QHSTA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.144482906 (595h52m30.015565942s) [noise=true , address=145.79.227.38:4995 , piece_id=XOROFSBZOSPYSC6LSNZJMSRS4KYNH7JMH3E542G77L3ZQ3SADW7A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.238892497 (595h52m30.109975532s) [noise=true , address=77.90.31.11:4575 , piece_id=O3DCIOB6TOACOAKWBFE5IZTTYKUSWJBOKHC7LC2CRACYYBHXAHMA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.366857342 (595h52m30.237940387s) [noise=true , address=45.91.116.11:4129 , piece_id=SQUUXN3L7HVY6PLVPCH5KAZK43MTWTZ3MI5NG6H2RANCMDOGIY5A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.375068928 (595h52m30.246151973s) [noise=true , address=77.90.31.14:4683 , piece_id=DD3WJKOZA34IDFGTBRZFCMYBOU3EWARB2772KQHIVBIDS6XBG73A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.641601427 (595h52m30.512684462s) [noise=true , address=145.79.227.40:4057 , piece_id=JKX3DD2VZGTTXJDYLJVSOJ4HEAAGUZIPG7AMLT5HX4GLLJ7EJDLA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.979636920 (595h52m30.850719955s) [noise=true , address=145.79.227.30:4788 , piece_id=TWTEPHC7KMREU47W7NCYNL6DEZ5XHPYD5QIKQZN6HZKR3OBPUT4A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.983123610 (595h52m30.854206645s) [noise=true , address=145.79.227.34:4937 , piece_id=4NXNJDUCZM5KLWS3K4ABDHODPCJHJYU42D2W5ON4LRS5ZM4LREHQ, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.031177872 (595h52m30.902260907s) [noise=true , address=145.79.227.44:4206 , piece_id=7QM25KZUSPAD4GTIDR67OJNMAEKLCBZRLUK3GZDB7TO2ZI5TEREA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.064351170 (595h52m30.935434205s) [noise=true , address=145.79.227.35:4489 , piece_id=3I25VU7FHAW24CCOAWF7YF6AIUA5LS37UL2BX7TJ4LBXINZT3ZCQ, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.125491260 (595h52m30.996574295s) [noise=true , address=145.79.227.33:4876 , piece_id=ULKTOEKRBWK3WSNO6YTJYXD7WRZTCKD2TVRXIGCT66SJA2HKMXEA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.261432112 (595h52m31.132515157s) [noise=true , address=45.91.116.8:4049 , piece_id=XBRESQJASTTIPV3G2SJUGXKPJP4UD22X7B4QDP5YJUVTSLMJVTNA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.703247748 (595h52m31.574330783s) [noise=true , address=77.90.31.10:4529 , piece_id=433RPSCXRLK7I7746NZN33AKHX7QVW2QK6FU4YJMUXQXM6UW2KFA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.703272755 (595h52m31.574355801s) [seg_pos={0 0}, upload=522349]: Segment reached optimal threshold of 14 pieces. Dec 30 11:21:31.703520197 (595h52m31.574603252s) [noise=true , address=64.49.13.169:4709 , piece_id=L6DBEGT6YAR5J4I3FRIMLCDKAHJ5HTUGH7VATKOM5NXF2L3FYM7A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:56546->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:56546->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56546->64.49.13.169:4709: use of closed network connection Dec 30 11:21:31.703529584 (595h52m31.574612629s) [noise=true , address=64.49.13.169:4709 , piece_id=L6DBEGT6YAR5J4I3FRIMLCDKAHJ5HTUGH7VATKOM5NXF2L3FYM7A, seg_pos={0 0}, upload=522349]: Long tail cancellation due to optimal threshold Dec 30 11:21:31.703752048 (595h52m31.574835093s) [noise=true , address=193.149.164.143:4703 , piece_id=EBPBXTU4XLXLN5FEV3WC3JMXBGWRSKRZOIOLIUP5ANY5AYTPLEIQ, seg_pos={0 0}, upload=522349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eomwcuHK58QuxBp3sAARJjm6Doc7zEzRYCodsTzFT4v6Nti9g): protocol: expected piece hash; context canceled; EOF Dec 30 11:21:31.703758611 (595h52m31.574841646s) [noise=true , address=77.90.31.17:4790 , piece_id=7AQKSCDTY4TIVXYUIE2XXXUB2R2M7LVDA6W3RHZBVPAUDUHYX4PQ, seg_pos={0 0}, upload=522349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b): protocol: expected piece hash; context canceled; EOF Dec 30 11:21:31.703778147 (595h52m31.574861192s) [noise=true , address=77.90.31.17:4790 , piece_id=7AQKSCDTY4TIVXYUIE2XXXUB2R2M7LVDA6W3RHZBVPAUDUHYX4PQ, seg_pos={0 0}, upload=522349]: Long tail cancellation due to optimal threshold Dec 30 11:21:31.703806712 (595h52m31.574889747s) [noise=true , address=193.149.164.143:4703 , piece_id=EBPBXTU4XLXLN5FEV3WC3JMXBGWRSKRZOIOLIUP5ANY5AYTPLEIQ, seg_pos={0 0}, upload=522349]: Long tail cancellation due to optimal threshold Dec 30 11:21:31.703837070 (595h52m31.574920115s) [seg_pos={0 0}, upload=522349]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:21:31.703965946 (595h52m31.575048981s) [upload=522349]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:21:31.703994861 (595h52m31.575077896s) [upload=522349]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:21:31.987657769 (595h52m31.858740814s) [upload=522349]: Done uploading segments... Dec 30 11:21:31.987682225 (595h52m31.85876526s) [upload=522349]: Upload finished. err: <nil> Duration: 59.711044796s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [X.XX.X.X.....XX.X........X.X.XX.X.X.X.X...X.XX..XXX.X..XX...] duration: 3.654758318s trace: https://jaeger.dev.storj.io/trace/33eb99dd4a90f9b8 FAIL (2025-12-30 11:25:41 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.654841445s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.X.XX.X.X.....XX.X........X.X.XX.X.X.X.X...X.XX..XXX.X..XX..] duration: 3.626529344s trace: https://jaeger.dev.storj.io/trace/33eb99dd4a90f9b8 FAIL (2025-12-30 11:25:18 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.626572468s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....X.......X..XX..X.X.X..........X....X.......X.....X..XXX.] duration: 5.835952737s trace: https://jaeger.dev.storj.io/trace/78c70eef2d7818c2 FAIL (2025-12-30 11:24:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 11:23:34.812028066 (595h54m34.683111101s) [upload=522378]: Starting upload Dec 30 11:23:34.812084975 (595h54m34.68316801s) [upload=522378]: Done starting upload Dec 30 11:23:34.812257994 (595h54m34.683341029s) [upload=522378]: Uploading segments... Dec 30 11:23:34.812298751 (595h54m34.683381787s) [upload=522378]: Got next segment. Inline: false Dec 30 11:23:34.812325192 (595h54m34.683408227s) [upload=522378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:23:34.812329200 (595h54m34.683412235s) [upload=522378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:23:35.056969122 (595h54m34.928052167s) [seg_pos={0 0}, upload=522378]: Begin upload segment... Dec 30 11:23:35.057030338 (595h54m34.928113383s) [seg_pos={0 0}, upload=522378]: Done begin upload segment. Dec 30 11:23:35.057052330 (595h54m34.928135365s) [noise=true , address=145.79.227.42:4123 , piece_id=LUVQKH4P4H5QETKOG26EQBVVM2O36D5YWAJEZSAXMAUPEUX4PKAQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.057065255 (595h54m34.9281483s) [noise=true , address=45.91.116.8:4045 , piece_id=ZIUANYQO3BSXHNEWFYAF3OBUYPOEXHCG7WJNSQ34MAZVX63IGYAA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.057547312 (595h54m34.928630348s) [noise=true , address=145.79.227.29:4941 , piece_id=JT56XVSIPQJKEC5MH5FDTAHONOCIIIDYEQSG7MVBRIBST2VK73KQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.057975839 (595h54m34.929058894s) [noise=true , address=145.79.227.10:4401 , piece_id=2QHEHKDQHC4N7TGQG66LB3LXKOIX7CPFQ3IRTXIJ6KZQZLHAUOTQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.058420205 (595h54m34.92950325s) [noise=true , address=45.91.116.5:4300 , piece_id=XVL2KMCJNIFR6S75Q33Z7JFZAEDUVO7PHOSE6YD5AHX3ONZD3P3A, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.058940144 (595h54m34.930023189s) [noise=true , address=145.79.227.25:4619 , piece_id=4EWAZCWAW62IB7VYTCTRIOAS4NNSB7VAWHL5AL4SIC2HP26EPUPA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.059370244 (595h54m34.930453279s) [noise=true , address=145.79.227.28:4726 , piece_id=QBQTDGKE6AN6M3A2OESAFYJWVBVT55D65YQSNUC5NR3VX7FKMT2Q, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.060849039 (595h54m34.931932074s) [noise=true , address=145.79.227.11:4731 , piece_id=4GQQJRSLOEVIHWOQZMAEOUYV5TU63GHI4ZRKEZ5AZQ4U7MYRE7GQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.062524930 (595h54m34.933607965s) [noise=true , address=145.79.227.14:4839 , piece_id=ZAGQ3R3XF7VVATDETNP3CTEIMWOU7QN44P5ZZQPTR2QKLJ4UFZKA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.064320279 (595h54m34.935403324s) [noise=true , address=145.79.227.35:4483 , piece_id=UGHXPY47MUMBRBWRDYHKQSUU2XX7CKOQA2WF5M54DOUTBW2FO27Q, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.066440937 (595h54m34.937523992s) [noise=true , address=145.79.227.50:4374 , piece_id=WAF22ATE3ZWJV7Q4B6AXAMVH5AQZ3M4AKRZZXAU7MVVAYYXTEL7Q, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.073033140 (595h54m34.944116185s) [noise=true , address=145.79.227.45:4229 , piece_id=WXCGOXRCNT2CDCBQPM235XBINQMCUNU2YJG3CDWYUWKPT55EFNQQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.073372616 (595h54m34.944455651s) [noise=true , address=145.79.227.7:4295 , piece_id=XTRYRQAC22NMIVKWJTOOL5DJRMBD2CSVVHWHF2QTP3KWLBPFUHVQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.074934210 (595h54m34.946017245s) [noise=true , address=45.91.116.7:4009 , piece_id=XPC2UPTHG433TCXRMGPC6VT7HOSF4BWBX5424VD24RFEZ2TVDTUA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.077002518 (595h54m34.948085553s) [noise=true , address=77.90.31.10:4520 , piece_id=TZOHYR6WUHVTN5CIGQU6K3RRUXZB7HGUM6IKO6ZFXCODYADD4Q6Q, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.077339841 (595h54m34.948422876s) [noise=true , address=145.79.227.36:4517 , piece_id=5QTQDPQMZUFYGUC4AW4DKZTMVD6UPJXHRDVCJWI33A7JKOB7G5PA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:36.767459832 (595h54m36.638542867s) [upload=522378]: Next returned nil segment Dec 30 11:23:36.767470743 (595h54m36.638553778s) [upload=522378]: Waiting for error group managing segments... Dec 30 11:23:37.039279379 (595h54m36.910362414s) [noise=true , address=145.79.227.11:4731 , piece_id=4GQQJRSLOEVIHWOQZMAEOUYV5TU63GHI4ZRKEZ5AZQ4U7MYRE7GQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.041352005 (595h54m36.91243504s) [noise=true , address=145.79.227.25:4619 , piece_id=4EWAZCWAW62IB7VYTCTRIOAS4NNSB7VAWHL5AL4SIC2HP26EPUPA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.060470380 (595h54m36.931553425s) [noise=true , address=145.79.227.14:4839 , piece_id=ZAGQ3R3XF7VVATDETNP3CTEIMWOU7QN44P5ZZQPTR2QKLJ4UFZKA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.063542990 (595h54m36.934626025s) [noise=true , address=145.79.227.28:4726 , piece_id=QBQTDGKE6AN6M3A2OESAFYJWVBVT55D65YQSNUC5NR3VX7FKMT2Q, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.075871886 (595h54m36.946954932s) [noise=true , address=145.79.227.10:4401 , piece_id=2QHEHKDQHC4N7TGQG66LB3LXKOIX7CPFQ3IRTXIJ6KZQZLHAUOTQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.077044910 (595h54m36.948127945s) [noise=true , address=145.79.227.42:4123 , piece_id=LUVQKH4P4H5QETKOG26EQBVVM2O36D5YWAJEZSAXMAUPEUX4PKAQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.079648879 (595h54m36.950731914s) [noise=true , address=145.79.227.50:4374 , piece_id=WAF22ATE3ZWJV7Q4B6AXAMVH5AQZ3M4AKRZZXAU7MVVAYYXTEL7Q, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.083355607 (595h54m36.954438652s) [noise=true , address=145.79.227.7:4295 , piece_id=XTRYRQAC22NMIVKWJTOOL5DJRMBD2CSVVHWHF2QTP3KWLBPFUHVQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.083522435 (595h54m36.95460547s) [noise=true , address=145.79.227.45:4229 , piece_id=WXCGOXRCNT2CDCBQPM235XBINQMCUNU2YJG3CDWYUWKPT55EFNQQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.152444123 (595h54m37.023527158s) [noise=true , address=145.79.227.36:4517 , piece_id=5QTQDPQMZUFYGUC4AW4DKZTMVD6UPJXHRDVCJWI33A7JKOB7G5PA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.383004515 (595h54m37.25408755s) [noise=true , address=45.91.116.8:4045 , piece_id=ZIUANYQO3BSXHNEWFYAF3OBUYPOEXHCG7WJNSQ34MAZVX63IGYAA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.475087265 (595h54m37.3461703s) [noise=true , address=77.90.31.10:4520 , piece_id=TZOHYR6WUHVTN5CIGQU6K3RRUXZB7HGUM6IKO6ZFXCODYADD4Q6Q, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.558983667 (595h54m37.430066702s) [noise=true , address=45.91.116.7:4009 , piece_id=XPC2UPTHG433TCXRMGPC6VT7HOSF4BWBX5424VD24RFEZ2TVDTUA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.559026078 (595h54m37.430109113s) [seg_pos={0 0}, upload=522378]: Segment reached optimal threshold of 13 pieces. Dec 30 11:23:37.559336128 (595h54m37.430419164s) [noise=true , address=145.79.227.35:4483 , piece_id=UGHXPY47MUMBRBWRDYHKQSUU2XX7CKOQA2WF5M54DOUTBW2FO27Q, seg_pos={0 0}, upload=522378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 30 11:23:37.559374401 (595h54m37.430457456s) [noise=true , address=145.79.227.35:4483 , piece_id=UGHXPY47MUMBRBWRDYHKQSUU2XX7CKOQA2WF5M54DOUTBW2FO27Q, seg_pos={0 0}, upload=522378]: Long tail cancellation due to optimal threshold Dec 30 11:23:37.559405030 (595h54m37.430488085s) [noise=true , address=45.91.116.5:4300 , piece_id=XVL2KMCJNIFR6S75Q33Z7JFZAEDUVO7PHOSE6YD5AHX3ONZD3P3A, seg_pos={0 0}, upload=522378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:34126->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:34126->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34126->45.91.116.5:4300: use of closed network connection Dec 30 11:23:37.559414177 (595h54m37.430497212s) [noise=true , address=45.91.116.5:4300 , piece_id=XVL2KMCJNIFR6S75Q33Z7JFZAEDUVO7PHOSE6YD5AHX3ONZD3P3A, seg_pos={0 0}, upload=522378]: Long tail cancellation due to optimal threshold Dec 30 11:23:37.559745157 (595h54m37.430828202s) [noise=true , address=145.79.227.29:4941 , piece_id=JT56XVSIPQJKEC5MH5FDTAHONOCIIIDYEQSG7MVBRIBST2VK73KQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 30 11:23:37.559754395 (595h54m37.43083744s) [noise=true , address=145.79.227.29:4941 , piece_id=JT56XVSIPQJKEC5MH5FDTAHONOCIIIDYEQSG7MVBRIBST2VK73KQ, seg_pos={0 0}, upload=522378]: Long tail cancellation due to optimal threshold Dec 30 11:23:37.559777999 (595h54m37.430861034s) [seg_pos={0 0}, upload=522378]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:23:37.559963924 (595h54m37.431046969s) [upload=522378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:23:37.559986977 (595h54m37.431070032s) [upload=522378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:23:37.849730184 (595h54m37.720813219s) [upload=522378]: Done uploading segments... Dec 30 11:23:37.849753237 (595h54m37.720836282s) [upload=522378]: Upload finished. err: <nil> Duration: 5.83598017s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [X.X.XX.X.X.....XX.X........X.X.XX.X.X.X.X...X.XX..XXX.X..XX.] duration: 3.554662536s trace: https://jaeger.dev.storj.io/trace/33eb99dd4a90f9b8 FAIL (2025-12-30 11:24:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.554737077s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X.................X..........X...X...........X...] duration: 44.349607689s trace: https://jaeger.dev.storj.io/trace/320f2449de9aa447 FAIL (2025-12-30 11:24:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8.212630018 (595h52m28.083713053s) [noise=true , address=145.79.227.44:4206 , piece_id=7QM25KZUSPAD4GTIDR67OJNMAEKLCBZRLUK3GZDB7TO2ZI5TEREA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.213211245 (595h52m28.08429428s) [noise=true , address=145.79.227.30:4788 , piece_id=TWTEPHC7KMREU47W7NCYNL6DEZ5XHPYD5QIKQZN6HZKR3OBPUT4A, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.213600286 (595h52m28.084683321s) [noise=true , address=77.90.31.11:4575 , piece_id=O3DCIOB6TOACOAKWBFE5IZTTYKUSWJBOKHC7LC2CRACYYBHXAHMA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.213779527 (595h52m28.084862562s) [noise=true , address=45.91.116.8:4049 , piece_id=XBRESQJASTTIPV3G2SJUGXKPJP4UD22X7B4QDP5YJUVTSLMJVTNA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.213888514 (595h52m28.084971559s) [noise=true , address=145.79.227.34:4937 , piece_id=4NXNJDUCZM5KLWS3K4ABDHODPCJHJYU42D2W5ON4LRS5ZM4LREHQ, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214146827 (595h52m28.085229862s) [noise=true , address=193.149.164.143:4703 , piece_id=EBPBXTU4XLXLN5FEV3WC3JMXBGWRSKRZOIOLIUP5ANY5AYTPLEIQ, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214153439 (595h52m28.085236474s) [noise=true , address=145.79.227.40:4057 , piece_id=JKX3DD2VZGTTXJDYLJVSOJ4HEAAGUZIPG7AMLT5HX4GLLJ7EJDLA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214505840 (595h52m28.085588875s) [noise=true , address=145.79.227.35:4489 , piece_id=3I25VU7FHAW24CCOAWF7YF6AIUA5LS37UL2BX7TJ4LBXINZT3ZCQ, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214601683 (595h52m28.085684718s) [noise=true , address=77.90.31.10:4529 , piece_id=433RPSCXRLK7I7746NZN33AKHX7QVW2QK6FU4YJMUXQXM6UW2KFA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214836058 (595h52m28.085919093s) [noise=true , address=145.79.227.33:4876 , piece_id=ULKTOEKRBWK3WSNO6YTJYXD7WRZTCKD2TVRXIGCT66SJA2HKMXEA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214886896 (595h52m28.085969931s) [noise=true , address=77.90.31.14:4683 , piece_id=DD3WJKOZA34IDFGTBRZFCMYBOU3EWARB2772KQHIVBIDS6XBG73A, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215143213 (595h52m28.086226248s) [noise=true , address=77.90.31.17:4790 , piece_id=7AQKSCDTY4TIVXYUIE2XXXUB2R2M7LVDA6W3RHZBVPAUDUHYX4PQ, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215224017 (595h52m28.086307052s) [noise=true , address=145.79.227.9:4385 , piece_id=O35GQDZODEC6HZNMUZMT47OLZFIMFGIYHYNPIIC66KW6MUJHVNPA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215431352 (595h52m28.086514387s) [noise=true , address=64.49.13.27:4965 , piece_id=V5NCPS3EFV7NABYT5KZ7E4TEQHICP4RFK5GKCHMMT45PJ2CE4S4Q, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215543906 (595h52m28.086626951s) [noise=true , address=64.49.13.26:4954 , piece_id=ZN2G36BDM3WARBVKBOCMGGKFQG7XKOFRIRVY3VKW4UMNSL4QHSTA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215913380 (595h52m28.086996415s) [noise=true , address=64.49.13.169:4709 , piece_id=L6DBEGT6YAR5J4I3FRIMLCDKAHJ5HTUGH7VATKOM5NXF2L3FYM7A, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.216329021 (595h52m28.087412056s) [noise=true , address=145.79.227.38:4995 , piece_id=XOROFSBZOSPYSC6LSNZJMSRS4KYNH7JMH3E542G77L3ZQ3SADW7A, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:29.220053219 (595h52m29.091136254s) [noise=true , address=64.49.13.27:4965 , piece_id=V5NCPS3EFV7NABYT5KZ7E4TEQHICP4RFK5GKCHMMT45PJ2CE4S4Q, seg_pos={0 0}, upload=522349]: Done uploading piece. err: ecclient: upload failed (node:12WVhm2QyhAf1y6HXRpyChMLbDLhZJQHEE2kQpstZH2dkwNhd7P, address:64.49.13.27:4965): protocol: storage node overloaded, request limit: 2000 Dec 30 11:21:29.637185454 (595h52m29.508268499s) [upload=522349]: Next returned nil segment Dec 30 11:21:29.637201474 (595h52m29.508284539s) [upload=522349]: Waiting for error group managing segments... Dec 30 11:21:30.066275167 (595h52m29.937358302s) [noise=true , address=145.79.227.9:4385 , piece_id=O35GQDZODEC6HZNMUZMT47OLZFIMFGIYHYNPIIC66KW6MUJHVNPA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.073548216 (595h52m29.944631251s) [noise=true , address=64.49.13.26:4954 , piece_id=ZN2G36BDM3WARBVKBOCMGGKFQG7XKOFRIRVY3VKW4UMNSL4QHSTA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.144482906 (595h52m30.015565942s) [noise=true , address=145.79.227.38:4995 , piece_id=XOROFSBZOSPYSC6LSNZJMSRS4KYNH7JMH3E542G77L3ZQ3SADW7A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.238892497 (595h52m30.109975532s) [noise=true , address=77.90.31.11:4575 , piece_id=O3DCIOB6TOACOAKWBFE5IZTTYKUSWJBOKHC7LC2CRACYYBHXAHMA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.366857342 (595h52m30.237940387s) [noise=true , address=45.91.116.11:4129 , piece_id=SQUUXN3L7HVY6PLVPCH5KAZK43MTWTZ3MI5NG6H2RANCMDOGIY5A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.375068928 (595h52m30.246151973s) [noise=true , address=77.90.31.14:4683 , piece_id=DD3WJKOZA34IDFGTBRZFCMYBOU3EWARB2772KQHIVBIDS6XBG73A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.641601427 (595h52m30.512684462s) [noise=true , address=145.79.227.40:4057 , piece_id=JKX3DD2VZGTTXJDYLJVSOJ4HEAAGUZIPG7AMLT5HX4GLLJ7EJDLA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.979636920 (595h52m30.850719955s) [noise=true , address=145.79.227.30:4788 , piece_id=TWTEPHC7KMREU47W7NCYNL6DEZ5XHPYD5QIKQZN6HZKR3OBPUT4A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.983123610 (595h52m30.854206645s) [noise=true , address=145.79.227.34:4937 , piece_id=4NXNJDUCZM5KLWS3K4ABDHODPCJHJYU42D2W5ON4LRS5ZM4LREHQ, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.031177872 (595h52m30.902260907s) [noise=true , address=145.79.227.44:4206 , piece_id=7QM25KZUSPAD4GTIDR67OJNMAEKLCBZRLUK3GZDB7TO2ZI5TEREA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.064351170 (595h52m30.935434205s) [noise=true , address=145.79.227.35:4489 , piece_id=3I25VU7FHAW24CCOAWF7YF6AIUA5LS37UL2BX7TJ4LBXINZT3ZCQ, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.125491260 (595h52m30.996574295s) [noise=true , address=145.79.227.33:4876 , piece_id=ULKTOEKRBWK3WSNO6YTJYXD7WRZTCKD2TVRXIGCT66SJA2HKMXEA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.261432112 (595h52m31.132515157s) [noise=true , address=45.91.116.8:4049 , piece_id=XBRESQJASTTIPV3G2SJUGXKPJP4UD22X7B4QDP5YJUVTSLMJVTNA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.703247748 (595h52m31.574330783s) [noise=true , address=77.90.31.10:4529 , piece_id=433RPSCXRLK7I7746NZN33AKHX7QVW2QK6FU4YJMUXQXM6UW2KFA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.703272755 (595h52m31.574355801s) [seg_pos={0 0}, upload=522349]: Segment reached optimal threshold of 14 pieces. Dec 30 11:21:31.703520197 (595h52m31.574603252s) [noise=true , address=64.49.13.169:4709 , piece_id=L6DBEGT6YAR5J4I3FRIMLCDKAHJ5HTUGH7VATKOM5NXF2L3FYM7A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:56546->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:56546->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56546->64.49.13.169:4709: use of closed network connection Dec 30 11:21:31.703529584 (595h52m31.574612629s) [noise=true , address=64.49.13.169:4709 , piece_id=L6DBEGT6YAR5J4I3FRIMLCDKAHJ5HTUGH7VATKOM5NXF2L3FYM7A, seg_pos={0 0}, upload=522349]: Long tail cancellation due to optimal threshold Dec 30 11:21:31.703752048 (595h52m31.574835093s) [noise=true , address=193.149.164.143:4703 , piece_id=EBPBXTU4XLXLN5FEV3WC3JMXBGWRSKRZOIOLIUP5ANY5AYTPLEIQ, seg_pos={0 0}, upload=522349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eomwcuHK58QuxBp3sAARJjm6Doc7zEzRYCodsTzFT4v6Nti9g): protocol: expected piece hash; context canceled; EOF Dec 30 11:21:31.703758611 (595h52m31.574841646s) [noise=true , address=77.90.31.17:4790 , piece_id=7AQKSCDTY4TIVXYUIE2XXXUB2R2M7LVDA6W3RHZBVPAUDUHYX4PQ, seg_pos={0 0}, upload=522349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b): protocol: expected piece hash; context canceled; EOF Dec 30 11:21:31.703778147 (595h52m31.574861192s) [noise=true , address=77.90.31.17:4790 , piece_id=7AQKSCDTY4TIVXYUIE2XXXUB2R2M7LVDA6W3RHZBVPAUDUHYX4PQ, seg_pos={0 0}, upload=522349]: Long tail cancellation due to optimal threshold Dec 30 11:21:31.703806712 (595h52m31.574889747s) [noise=true , address=193.149.164.143:4703 , piece_id=EBPBXTU4XLXLN5FEV3WC3JMXBGWRSKRZOIOLIUP5ANY5AYTPLEIQ, seg_pos={0 0}, upload=522349]: Long tail cancellation due to optimal threshold Dec 30 11:21:31.703837070 (595h52m31.574920115s) [seg_pos={0 0}, upload=522349]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:21:31.703965946 (595h52m31.575048981s) [upload=522349]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:21:31.703994861 (595h52m31.575077896s) [upload=522349]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:21:31.987657769 (595h52m31.858740814s) [upload=522349]: Done uploading segments... Dec 30 11:21:31.987682225 (595h52m31.85876526s) [upload=522349]: Upload finished. err: <nil> Duration: 44.349638618s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.....X.......X..XX..X.X.X..........X....X.......X.....X..XXX] duration: 30.00335722s trace: https://jaeger.dev.storj.io/trace/78c70eef2d7818c2 FAIL (2025-12-30 11:24:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 11:23:34.812028066 (595h54m34.683111101s) [upload=522378]: Starting upload Dec 30 11:23:34.812084975 (595h54m34.68316801s) [upload=522378]: Done starting upload Dec 30 11:23:34.812257994 (595h54m34.683341029s) [upload=522378]: Uploading segments... Dec 30 11:23:34.812298751 (595h54m34.683381787s) [upload=522378]: Got next segment. Inline: false Dec 30 11:23:34.812325192 (595h54m34.683408227s) [upload=522378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:23:34.812329200 (595h54m34.683412235s) [upload=522378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:23:35.056969122 (595h54m34.928052167s) [seg_pos={0 0}, upload=522378]: Begin upload segment... Dec 30 11:23:35.057030338 (595h54m34.928113383s) [seg_pos={0 0}, upload=522378]: Done begin upload segment. Dec 30 11:23:35.057052330 (595h54m34.928135365s) [noise=true , address=145.79.227.42:4123 , piece_id=LUVQKH4P4H5QETKOG26EQBVVM2O36D5YWAJEZSAXMAUPEUX4PKAQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.057065255 (595h54m34.9281483s) [noise=true , address=45.91.116.8:4045 , piece_id=ZIUANYQO3BSXHNEWFYAF3OBUYPOEXHCG7WJNSQ34MAZVX63IGYAA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.057547312 (595h54m34.928630348s) [noise=true , address=145.79.227.29:4941 , piece_id=JT56XVSIPQJKEC5MH5FDTAHONOCIIIDYEQSG7MVBRIBST2VK73KQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.057975839 (595h54m34.929058894s) [noise=true , address=145.79.227.10:4401 , piece_id=2QHEHKDQHC4N7TGQG66LB3LXKOIX7CPFQ3IRTXIJ6KZQZLHAUOTQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.058420205 (595h54m34.92950325s) [noise=true , address=45.91.116.5:4300 , piece_id=XVL2KMCJNIFR6S75Q33Z7JFZAEDUVO7PHOSE6YD5AHX3ONZD3P3A, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.058940144 (595h54m34.930023189s) [noise=true , address=145.79.227.25:4619 , piece_id=4EWAZCWAW62IB7VYTCTRIOAS4NNSB7VAWHL5AL4SIC2HP26EPUPA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.059370244 (595h54m34.930453279s) [noise=true , address=145.79.227.28:4726 , piece_id=QBQTDGKE6AN6M3A2OESAFYJWVBVT55D65YQSNUC5NR3VX7FKMT2Q, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.060849039 (595h54m34.931932074s) [noise=true , address=145.79.227.11:4731 , piece_id=4GQQJRSLOEVIHWOQZMAEOUYV5TU63GHI4ZRKEZ5AZQ4U7MYRE7GQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.062524930 (595h54m34.933607965s) [noise=true , address=145.79.227.14:4839 , piece_id=ZAGQ3R3XF7VVATDETNP3CTEIMWOU7QN44P5ZZQPTR2QKLJ4UFZKA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.064320279 (595h54m34.935403324s) [noise=true , address=145.79.227.35:4483 , piece_id=UGHXPY47MUMBRBWRDYHKQSUU2XX7CKOQA2WF5M54DOUTBW2FO27Q, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.066440937 (595h54m34.937523992s) [noise=true , address=145.79.227.50:4374 , piece_id=WAF22ATE3ZWJV7Q4B6AXAMVH5AQZ3M4AKRZZXAU7MVVAYYXTEL7Q, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.073033140 (595h54m34.944116185s) [noise=true , address=145.79.227.45:4229 , piece_id=WXCGOXRCNT2CDCBQPM235XBINQMCUNU2YJG3CDWYUWKPT55EFNQQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.073372616 (595h54m34.944455651s) [noise=true , address=145.79.227.7:4295 , piece_id=XTRYRQAC22NMIVKWJTOOL5DJRMBD2CSVVHWHF2QTP3KWLBPFUHVQ, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.074934210 (595h54m34.946017245s) [noise=true , address=45.91.116.7:4009 , piece_id=XPC2UPTHG433TCXRMGPC6VT7HOSF4BWBX5424VD24RFEZ2TVDTUA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.077002518 (595h54m34.948085553s) [noise=true , address=77.90.31.10:4520 , piece_id=TZOHYR6WUHVTN5CIGQU6K3RRUXZB7HGUM6IKO6ZFXCODYADD4Q6Q, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:35.077339841 (595h54m34.948422876s) [noise=true , address=145.79.227.36:4517 , piece_id=5QTQDPQMZUFYGUC4AW4DKZTMVD6UPJXHRDVCJWI33A7JKOB7G5PA, seg_pos={0 0}, upload=522378]: Uploading piece... Dec 30 11:23:36.767459832 (595h54m36.638542867s) [upload=522378]: Next returned nil segment Dec 30 11:23:36.767470743 (595h54m36.638553778s) [upload=522378]: Waiting for error group managing segments... Dec 30 11:23:37.039279379 (595h54m36.910362414s) [noise=true , address=145.79.227.11:4731 , piece_id=4GQQJRSLOEVIHWOQZMAEOUYV5TU63GHI4ZRKEZ5AZQ4U7MYRE7GQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.041352005 (595h54m36.91243504s) [noise=true , address=145.79.227.25:4619 , piece_id=4EWAZCWAW62IB7VYTCTRIOAS4NNSB7VAWHL5AL4SIC2HP26EPUPA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.060470380 (595h54m36.931553425s) [noise=true , address=145.79.227.14:4839 , piece_id=ZAGQ3R3XF7VVATDETNP3CTEIMWOU7QN44P5ZZQPTR2QKLJ4UFZKA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.063542990 (595h54m36.934626025s) [noise=true , address=145.79.227.28:4726 , piece_id=QBQTDGKE6AN6M3A2OESAFYJWVBVT55D65YQSNUC5NR3VX7FKMT2Q, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.075871886 (595h54m36.946954932s) [noise=true , address=145.79.227.10:4401 , piece_id=2QHEHKDQHC4N7TGQG66LB3LXKOIX7CPFQ3IRTXIJ6KZQZLHAUOTQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.077044910 (595h54m36.948127945s) [noise=true , address=145.79.227.42:4123 , piece_id=LUVQKH4P4H5QETKOG26EQBVVM2O36D5YWAJEZSAXMAUPEUX4PKAQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.079648879 (595h54m36.950731914s) [noise=true , address=145.79.227.50:4374 , piece_id=WAF22ATE3ZWJV7Q4B6AXAMVH5AQZ3M4AKRZZXAU7MVVAYYXTEL7Q, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.083355607 (595h54m36.954438652s) [noise=true , address=145.79.227.7:4295 , piece_id=XTRYRQAC22NMIVKWJTOOL5DJRMBD2CSVVHWHF2QTP3KWLBPFUHVQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.083522435 (595h54m36.95460547s) [noise=true , address=145.79.227.45:4229 , piece_id=WXCGOXRCNT2CDCBQPM235XBINQMCUNU2YJG3CDWYUWKPT55EFNQQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.152444123 (595h54m37.023527158s) [noise=true , address=145.79.227.36:4517 , piece_id=5QTQDPQMZUFYGUC4AW4DKZTMVD6UPJXHRDVCJWI33A7JKOB7G5PA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.383004515 (595h54m37.25408755s) [noise=true , address=45.91.116.8:4045 , piece_id=ZIUANYQO3BSXHNEWFYAF3OBUYPOEXHCG7WJNSQ34MAZVX63IGYAA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.475087265 (595h54m37.3461703s) [noise=true , address=77.90.31.10:4520 , piece_id=TZOHYR6WUHVTN5CIGQU6K3RRUXZB7HGUM6IKO6ZFXCODYADD4Q6Q, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.558983667 (595h54m37.430066702s) [noise=true , address=45.91.116.7:4009 , piece_id=XPC2UPTHG433TCXRMGPC6VT7HOSF4BWBX5424VD24RFEZ2TVDTUA, seg_pos={0 0}, upload=522378]: Done uploading piece. err: <nil> Dec 30 11:23:37.559026078 (595h54m37.430109113s) [seg_pos={0 0}, upload=522378]: Segment reached optimal threshold of 13 pieces. Dec 30 11:23:37.559336128 (595h54m37.430419164s) [noise=true , address=145.79.227.35:4483 , piece_id=UGHXPY47MUMBRBWRDYHKQSUU2XX7CKOQA2WF5M54DOUTBW2FO27Q, seg_pos={0 0}, upload=522378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 30 11:23:37.559374401 (595h54m37.430457456s) [noise=true , address=145.79.227.35:4483 , piece_id=UGHXPY47MUMBRBWRDYHKQSUU2XX7CKOQA2WF5M54DOUTBW2FO27Q, seg_pos={0 0}, upload=522378]: Long tail cancellation due to optimal threshold Dec 30 11:23:37.559405030 (595h54m37.430488085s) [noise=true , address=45.91.116.5:4300 , piece_id=XVL2KMCJNIFR6S75Q33Z7JFZAEDUVO7PHOSE6YD5AHX3ONZD3P3A, seg_pos={0 0}, upload=522378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:34126->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:34126->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34126->45.91.116.5:4300: use of closed network connection Dec 30 11:23:37.559414177 (595h54m37.430497212s) [noise=true , address=45.91.116.5:4300 , piece_id=XVL2KMCJNIFR6S75Q33Z7JFZAEDUVO7PHOSE6YD5AHX3ONZD3P3A, seg_pos={0 0}, upload=522378]: Long tail cancellation due to optimal threshold Dec 30 11:23:37.559745157 (595h54m37.430828202s) [noise=true , address=145.79.227.29:4941 , piece_id=JT56XVSIPQJKEC5MH5FDTAHONOCIIIDYEQSG7MVBRIBST2VK73KQ, seg_pos={0 0}, upload=522378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 30 11:23:37.559754395 (595h54m37.43083744s) [noise=true , address=145.79.227.29:4941 , piece_id=JT56XVSIPQJKEC5MH5FDTAHONOCIIIDYEQSG7MVBRIBST2VK73KQ, seg_pos={0 0}, upload=522378]: Long tail cancellation due to optimal threshold Dec 30 11:23:37.559777999 (595h54m37.430861034s) [seg_pos={0 0}, upload=522378]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:23:37.559963924 (595h54m37.431046969s) [upload=522378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:23:37.559986977 (595h54m37.431070032s) [upload=522378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:23:37.849730184 (595h54m37.720813219s) [upload=522378]: Done uploading segments... Dec 30 11:23:37.849753237 (595h54m37.720836282s) [upload=522378]: Upload finished. err: <nil> Duration: 30.003379933s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [XX.X.XX.X.X.....XX.X........X.X.XX.X.X.X.X...X.XX..XXX.X..XX] duration: 5.001441424s trace: https://jaeger.dev.storj.io/trace/33eb99dd4a90f9b8 FAIL (2025-12-30 11:23:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001462515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X.................X..........X...X...........X..] duration: 56.29618991s trace: https://jaeger.dev.storj.io/trace/320f2449de9aa447 FAIL (2025-12-30 11:23:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8.212630018 (595h52m28.083713053s) [noise=true , address=145.79.227.44:4206 , piece_id=7QM25KZUSPAD4GTIDR67OJNMAEKLCBZRLUK3GZDB7TO2ZI5TEREA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.213211245 (595h52m28.08429428s) [noise=true , address=145.79.227.30:4788 , piece_id=TWTEPHC7KMREU47W7NCYNL6DEZ5XHPYD5QIKQZN6HZKR3OBPUT4A, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.213600286 (595h52m28.084683321s) [noise=true , address=77.90.31.11:4575 , piece_id=O3DCIOB6TOACOAKWBFE5IZTTYKUSWJBOKHC7LC2CRACYYBHXAHMA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.213779527 (595h52m28.084862562s) [noise=true , address=45.91.116.8:4049 , piece_id=XBRESQJASTTIPV3G2SJUGXKPJP4UD22X7B4QDP5YJUVTSLMJVTNA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.213888514 (595h52m28.084971559s) [noise=true , address=145.79.227.34:4937 , piece_id=4NXNJDUCZM5KLWS3K4ABDHODPCJHJYU42D2W5ON4LRS5ZM4LREHQ, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214146827 (595h52m28.085229862s) [noise=true , address=193.149.164.143:4703 , piece_id=EBPBXTU4XLXLN5FEV3WC3JMXBGWRSKRZOIOLIUP5ANY5AYTPLEIQ, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214153439 (595h52m28.085236474s) [noise=true , address=145.79.227.40:4057 , piece_id=JKX3DD2VZGTTXJDYLJVSOJ4HEAAGUZIPG7AMLT5HX4GLLJ7EJDLA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214505840 (595h52m28.085588875s) [noise=true , address=145.79.227.35:4489 , piece_id=3I25VU7FHAW24CCOAWF7YF6AIUA5LS37UL2BX7TJ4LBXINZT3ZCQ, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214601683 (595h52m28.085684718s) [noise=true , address=77.90.31.10:4529 , piece_id=433RPSCXRLK7I7746NZN33AKHX7QVW2QK6FU4YJMUXQXM6UW2KFA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214836058 (595h52m28.085919093s) [noise=true , address=145.79.227.33:4876 , piece_id=ULKTOEKRBWK3WSNO6YTJYXD7WRZTCKD2TVRXIGCT66SJA2HKMXEA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214886896 (595h52m28.085969931s) [noise=true , address=77.90.31.14:4683 , piece_id=DD3WJKOZA34IDFGTBRZFCMYBOU3EWARB2772KQHIVBIDS6XBG73A, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215143213 (595h52m28.086226248s) [noise=true , address=77.90.31.17:4790 , piece_id=7AQKSCDTY4TIVXYUIE2XXXUB2R2M7LVDA6W3RHZBVPAUDUHYX4PQ, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215224017 (595h52m28.086307052s) [noise=true , address=145.79.227.9:4385 , piece_id=O35GQDZODEC6HZNMUZMT47OLZFIMFGIYHYNPIIC66KW6MUJHVNPA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215431352 (595h52m28.086514387s) [noise=true , address=64.49.13.27:4965 , piece_id=V5NCPS3EFV7NABYT5KZ7E4TEQHICP4RFK5GKCHMMT45PJ2CE4S4Q, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215543906 (595h52m28.086626951s) [noise=true , address=64.49.13.26:4954 , piece_id=ZN2G36BDM3WARBVKBOCMGGKFQG7XKOFRIRVY3VKW4UMNSL4QHSTA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215913380 (595h52m28.086996415s) [noise=true , address=64.49.13.169:4709 , piece_id=L6DBEGT6YAR5J4I3FRIMLCDKAHJ5HTUGH7VATKOM5NXF2L3FYM7A, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.216329021 (595h52m28.087412056s) [noise=true , address=145.79.227.38:4995 , piece_id=XOROFSBZOSPYSC6LSNZJMSRS4KYNH7JMH3E542G77L3ZQ3SADW7A, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:29.220053219 (595h52m29.091136254s) [noise=true , address=64.49.13.27:4965 , piece_id=V5NCPS3EFV7NABYT5KZ7E4TEQHICP4RFK5GKCHMMT45PJ2CE4S4Q, seg_pos={0 0}, upload=522349]: Done uploading piece. err: ecclient: upload failed (node:12WVhm2QyhAf1y6HXRpyChMLbDLhZJQHEE2kQpstZH2dkwNhd7P, address:64.49.13.27:4965): protocol: storage node overloaded, request limit: 2000 Dec 30 11:21:29.637185454 (595h52m29.508268499s) [upload=522349]: Next returned nil segment Dec 30 11:21:29.637201474 (595h52m29.508284539s) [upload=522349]: Waiting for error group managing segments... Dec 30 11:21:30.066275167 (595h52m29.937358302s) [noise=true , address=145.79.227.9:4385 , piece_id=O35GQDZODEC6HZNMUZMT47OLZFIMFGIYHYNPIIC66KW6MUJHVNPA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.073548216 (595h52m29.944631251s) [noise=true , address=64.49.13.26:4954 , piece_id=ZN2G36BDM3WARBVKBOCMGGKFQG7XKOFRIRVY3VKW4UMNSL4QHSTA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.144482906 (595h52m30.015565942s) [noise=true , address=145.79.227.38:4995 , piece_id=XOROFSBZOSPYSC6LSNZJMSRS4KYNH7JMH3E542G77L3ZQ3SADW7A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.238892497 (595h52m30.109975532s) [noise=true , address=77.90.31.11:4575 , piece_id=O3DCIOB6TOACOAKWBFE5IZTTYKUSWJBOKHC7LC2CRACYYBHXAHMA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.366857342 (595h52m30.237940387s) [noise=true , address=45.91.116.11:4129 , piece_id=SQUUXN3L7HVY6PLVPCH5KAZK43MTWTZ3MI5NG6H2RANCMDOGIY5A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.375068928 (595h52m30.246151973s) [noise=true , address=77.90.31.14:4683 , piece_id=DD3WJKOZA34IDFGTBRZFCMYBOU3EWARB2772KQHIVBIDS6XBG73A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.641601427 (595h52m30.512684462s) [noise=true , address=145.79.227.40:4057 , piece_id=JKX3DD2VZGTTXJDYLJVSOJ4HEAAGUZIPG7AMLT5HX4GLLJ7EJDLA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.979636920 (595h52m30.850719955s) [noise=true , address=145.79.227.30:4788 , piece_id=TWTEPHC7KMREU47W7NCYNL6DEZ5XHPYD5QIKQZN6HZKR3OBPUT4A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.983123610 (595h52m30.854206645s) [noise=true , address=145.79.227.34:4937 , piece_id=4NXNJDUCZM5KLWS3K4ABDHODPCJHJYU42D2W5ON4LRS5ZM4LREHQ, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.031177872 (595h52m30.902260907s) [noise=true , address=145.79.227.44:4206 , piece_id=7QM25KZUSPAD4GTIDR67OJNMAEKLCBZRLUK3GZDB7TO2ZI5TEREA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.064351170 (595h52m30.935434205s) [noise=true , address=145.79.227.35:4489 , piece_id=3I25VU7FHAW24CCOAWF7YF6AIUA5LS37UL2BX7TJ4LBXINZT3ZCQ, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.125491260 (595h52m30.996574295s) [noise=true , address=145.79.227.33:4876 , piece_id=ULKTOEKRBWK3WSNO6YTJYXD7WRZTCKD2TVRXIGCT66SJA2HKMXEA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.261432112 (595h52m31.132515157s) [noise=true , address=45.91.116.8:4049 , piece_id=XBRESQJASTTIPV3G2SJUGXKPJP4UD22X7B4QDP5YJUVTSLMJVTNA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.703247748 (595h52m31.574330783s) [noise=true , address=77.90.31.10:4529 , piece_id=433RPSCXRLK7I7746NZN33AKHX7QVW2QK6FU4YJMUXQXM6UW2KFA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.703272755 (595h52m31.574355801s) [seg_pos={0 0}, upload=522349]: Segment reached optimal threshold of 14 pieces. Dec 30 11:21:31.703520197 (595h52m31.574603252s) [noise=true , address=64.49.13.169:4709 , piece_id=L6DBEGT6YAR5J4I3FRIMLCDKAHJ5HTUGH7VATKOM5NXF2L3FYM7A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:56546->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:56546->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56546->64.49.13.169:4709: use of closed network connection Dec 30 11:21:31.703529584 (595h52m31.574612629s) [noise=true , address=64.49.13.169:4709 , piece_id=L6DBEGT6YAR5J4I3FRIMLCDKAHJ5HTUGH7VATKOM5NXF2L3FYM7A, seg_pos={0 0}, upload=522349]: Long tail cancellation due to optimal threshold Dec 30 11:21:31.703752048 (595h52m31.574835093s) [noise=true , address=193.149.164.143:4703 , piece_id=EBPBXTU4XLXLN5FEV3WC3JMXBGWRSKRZOIOLIUP5ANY5AYTPLEIQ, seg_pos={0 0}, upload=522349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eomwcuHK58QuxBp3sAARJjm6Doc7zEzRYCodsTzFT4v6Nti9g): protocol: expected piece hash; context canceled; EOF Dec 30 11:21:31.703758611 (595h52m31.574841646s) [noise=true , address=77.90.31.17:4790 , piece_id=7AQKSCDTY4TIVXYUIE2XXXUB2R2M7LVDA6W3RHZBVPAUDUHYX4PQ, seg_pos={0 0}, upload=522349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b): protocol: expected piece hash; context canceled; EOF Dec 30 11:21:31.703778147 (595h52m31.574861192s) [noise=true , address=77.90.31.17:4790 , piece_id=7AQKSCDTY4TIVXYUIE2XXXUB2R2M7LVDA6W3RHZBVPAUDUHYX4PQ, seg_pos={0 0}, upload=522349]: Long tail cancellation due to optimal threshold Dec 30 11:21:31.703806712 (595h52m31.574889747s) [noise=true , address=193.149.164.143:4703 , piece_id=EBPBXTU4XLXLN5FEV3WC3JMXBGWRSKRZOIOLIUP5ANY5AYTPLEIQ, seg_pos={0 0}, upload=522349]: Long tail cancellation due to optimal threshold Dec 30 11:21:31.703837070 (595h52m31.574920115s) [seg_pos={0 0}, upload=522349]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:21:31.703965946 (595h52m31.575048981s) [upload=522349]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:21:31.703994861 (595h52m31.575077896s) [upload=522349]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:21:31.987657769 (595h52m31.858740814s) [upload=522349]: Done uploading segments... Dec 30 11:21:31.987682225 (595h52m31.85876526s) [upload=522349]: Upload finished. err: <nil> Duration: 56.29621614s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [XXX.X.XX.X.X.....XX.X........X.X.XX.X.X.X.X...X.XX..XXX.X..X] duration: 5.001499244s trace: https://jaeger.dev.storj.io/trace/21b50a98bf9f70b FAIL (2025-12-30 11:23:17 UTC) orbiter warning: context deadline exceeded Duration: 5.001510987s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [......X.......X..XX..X.X.X..........X....X.......X.....X..XX] duration: 30.002294902s trace: https://jaeger.dev.storj.io/trace/79c2126606331189 FAIL (2025-12-30 11:22:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 30 11:22:23.374803431 (595h53m23.245886466s) [upload=522360]: Starting upload Dec 30 11:22:23.374865468 (595h53m23.245948513s) [upload=522360]: Done starting upload Dec 30 11:22:23.374982592 (595h53m23.246065637s) [upload=522360]: Uploading segments... Dec 30 11:22:23.375120835 (595h53m23.24620387s) [upload=522360]: Got next segment. Inline: false Dec 30 11:22:23.375932420 (595h53m23.247015455s) [upload=522360]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:22:23.375940766 (595h53m23.247023801s) [upload=522360]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:22:23.622276318 (595h53m23.493359353s) [seg_pos={0 0}, upload=522360]: Begin upload segment... Dec 30 11:22:23.622333446 (595h53m23.493416491s) [seg_pos={0 0}, upload=522360]: Done begin upload segment. Dec 30 11:22:23.622367000 (595h53m23.493450045s) [noise=true , address=45.91.116.7:4011 , piece_id=ZOXPIR24N44ZCBTK3E7EMPTIYTV7KKUCNVDW7BVMQQCJIZUVLFFA, seg_pos={0 0}, upload=522360]: Uploading piece... Dec 30 11:22:23.622407898 (595h53m23.493490933s) [noise=true , address=45.91.116.12:4225 , piece_id=FHYNQKXJM5RAEWPWVAY4IEZNDEWY7VDECHK4M2DQK6D2FW4AB5TA, seg_pos={0 0}, upload=522360]: Uploading piece... Dec 30 11:22:23.622802760 (595h53m23.493885805s) [noise=true , address=145.79.227.9:4365 , piece_id=ASCFXF4ES4TOY7BDTLPFBAA5JN2JN2INCCOI22XXVWH76HXCPMRQ, seg_pos={0 0}, upload=522360]: Uploading piece... Dec 30 11:22:23.623156453 (595h53m23.494239489s) [noise=true , address=145.79.227.38:4978 , piece_id=472LEJDC5YKZMDS56VRMYPTPB5RAHG7CV3INNZ2IEGXF4II2BBGQ, seg_pos={0 0}, upload=522360]: Uploading piece... Dec 30 11:22:23.623512181 (595h53m23.494595226s) [noise=true , address=145.79.227.20:4437 , piece_id=U5BJUI72SIJ2VI7D36TRRFCB2T54TSUKTSAMQQI3KPWWH62M7OHA, seg_pos={0 0}, upload=522360]: Uploading piece... Dec 30 11:22:23.623888097 (595h53m23.494971142s) [noise=true , address=145.79.227.44:4193 , piece_id=VKSHKZ34JDN6RJOMZVCBN25SBG2OXDCLMEHSKECVOR4EEAY5JN6Q, seg_pos={0 0}, upload=522360]: Uploading piece... Dec 30 11:22:23.624267559 (595h53m23.495350604s) [noise=true , address=145.79.227.45:4231 , piece_id=4ML2WRRLGBJSWXA27MGDHDC7OZHE2Q7MMRIQY2W76QKUEKA6IZXA, seg_pos={0 0}, upload=522360]: Uploading piece... Dec 30 11:22:23.624656520 (595h53m23.495739565s) [noise=true , address=145.79.227.14:4840 , piece_id=YICEO5ELBYJLP4QW3AO436MVMR5Z6QJUX632HF2KRWXYTFXDEXMA, seg_pos={0 0}, upload=522360]: Uploading piece... Dec 30 11:22:23.625254368 (595h53m23.496337403s) [noise=true , address=145.79.227.23:4547 , piece_id=DMOSNKVVMBA6X3ME727TEXNGCGC7AW52AURFU55LZRP63PCYAI3Q, seg_pos={0 0}, upload=522360]: Uploading piece... Dec 30 11:22:23.626887488 (595h53m23.497970523s) [noise=true , address=145.79.227.34:4905 , piece_id=NZEGGXF5EU64MG2JTYITNSBQJQ2YOE3RUEBXMLMWNOZGW6G2JS3A, seg_pos={0 0}, upload=522360]: Uploading piece... Dec 30 11:22:23.627209901 (595h53m23.498292937s) [noise=true , address=77.90.31.14:4663 , piece_id=U67TFMG72TDUBDDP2WAZWX3T3YQ377EKW7RKS753TJNPLCVRSAGQ, seg_pos={0 0}, upload=522360]: Uploading piece... Dec 30 11:22:23.628464510 (595h53m23.499547545s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2Z2XTJJSC5JJUAHNDKCPIYUIGQNKZZ3K2JA7FAXRS4JBRY2ACTQ, seg_pos={0 0}, upload=522360]: Uploading piece... Dec 30 11:22:23.629879736 (595h53m23.500962771s) [noise=true , address=45.91.116.6:4976 , piece_id=SN77G4J4R7N46PDKY6T2PGVTVAAL47M4SKTY2M4YSILQ6URQCRRA, seg_pos={0 0}, upload=522360]: Uploading piece... Dec 30 11:22:23.631528164 (595h53m23.502611199s) [noise=true , address=145.79.227.7:4293 , piece_id=NZ3JOHB55T4ZC3ZMVWM7HJ5NP6RKKCQH43PSRQIRD66346ESNCYQ, seg_pos={0 0}, upload=522360]: Uploading piece... Dec 30 11:22:23.633144162 (595h53m23.504227197s) [noise=true , address=145.79.227.36:4517 , piece_id=6XSQDVGPRTXQ5QKR5SVVJUMYZ56PN53JEX7AVDIVRPA6MVJKPEDQ, seg_pos={0 0}, upload=522360]: Uploading piece... Dec 30 11:22:23.634915875 (595h53m23.50599891s) [noise=true , address=145.79.227.13:4802 , piece_id=VVZSLFFXO5WAFQT7SGQQ5NMOJNWVMYUMQQMSDADDH3VSBDXJDGTA, seg_pos={0 0}, upload=522360]: Uploading piece... Dec 30 11:22:25.320776626 (595h53m25.191859661s) [upload=522360]: Next returned nil segment Dec 30 11:22:25.320794350 (595h53m25.191877386s) [upload=522360]: Waiting for error group managing segments... Dec 30 11:22:25.619899453 (595h53m25.490982488s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2Z2XTJJSC5JJUAHNDKCPIYUIGQNKZZ3K2JA7FAXRS4JBRY2ACTQ, seg_pos={0 0}, upload=522360]: Done uploading piece. err: <nil> Dec 30 11:22:25.620867967 (595h53m25.491951002s) [noise=true , address=145.79.227.38:4978 , piece_id=472LEJDC5YKZMDS56VRMYPTPB5RAHG7CV3INNZ2IEGXF4II2BBGQ, seg_pos={0 0}, upload=522360]: Done uploading piece. err: <nil> Dec 30 11:22:25.626349915 (595h53m25.497432961s) [noise=true , address=145.79.227.23:4547 , piece_id=DMOSNKVVMBA6X3ME727TEXNGCGC7AW52AURFU55LZRP63PCYAI3Q, seg_pos={0 0}, upload=522360]: Done uploading piece. err: <nil> Dec 30 11:22:25.630270762 (595h53m25.501353797s) [noise=true , address=145.79.227.20:4437 , piece_id=U5BJUI72SIJ2VI7D36TRRFCB2T54TSUKTSAMQQI3KPWWH62M7OHA, seg_pos={0 0}, upload=522360]: Done uploading piece. err: <nil> Dec 30 11:22:25.630681173 (595h53m25.501764219s) [noise=true , address=145.79.227.7:4293 , piece_id=NZ3JOHB55T4ZC3ZMVWM7HJ5NP6RKKCQH43PSRQIRD66346ESNCYQ, seg_pos={0 0}, upload=522360]: Done uploading piece. err: <nil> Dec 30 11:22:25.631914281 (595h53m25.502997326s) [noise=true , address=145.79.227.45:4231 , piece_id=4ML2WRRLGBJSWXA27MGDHDC7OZHE2Q7MMRIQY2W76QKUEKA6IZXA, seg_pos={0 0}, upload=522360]: Done uploading piece. err: <nil> Dec 30 11:22:25.632165129 (595h53m25.503248164s) [noise=true , address=145.79.227.9:4365 , piece_id=ASCFXF4ES4TOY7BDTLPFBAA5JN2JN2INCCOI22XXVWH76HXCPMRQ, seg_pos={0 0}, upload=522360]: Done uploading piece. err: <nil> Dec 30 11:22:25.638239556 (595h53m25.509322601s) [noise=true , address=145.79.227.36:4517 , piece_id=6XSQDVGPRTXQ5QKR5SVVJUMYZ56PN53JEX7AVDIVRPA6MVJKPEDQ, seg_pos={0 0}, upload=522360]: Done uploading piece. err: <nil> Dec 30 11:22:25.640163479 (595h53m25.511246524s) [noise=true , address=145.79.227.14:4840 , piece_id=YICEO5ELBYJLP4QW3AO436MVMR5Z6QJUX632HF2KRWXYTFXDEXMA, seg_pos={0 0}, upload=522360]: Done uploading piece. err: <nil> Dec 30 11:22:25.834504694 (595h53m25.70558773s) [noise=true , address=145.79.227.13:4802 , piece_id=VVZSLFFXO5WAFQT7SGQQ5NMOJNWVMYUMQQMSDADDH3VSBDXJDGTA, seg_pos={0 0}, upload=522360]: Done uploading piece. err: <nil> Dec 30 11:22:26.488801047 (595h53m26.359884082s) [noise=true , address=45.91.116.6:4976 , piece_id=SN77G4J4R7N46PDKY6T2PGVTVAAL47M4SKTY2M4YSILQ6URQCRRA, seg_pos={0 0}, upload=522360]: Done uploading piece. err: <nil> Dec 30 11:22:26.591379242 (595h53m26.462462287s) [noise=true , address=45.91.116.12:4225 , piece_id=FHYNQKXJM5RAEWPWVAY4IEZNDEWY7VDECHK4M2DQK6D2FW4AB5TA, seg_pos={0 0}, upload=522360]: Done uploading piece. err: <nil> Dec 30 11:22:26.719835773 (595h53m26.590918818s) [noise=true , address=77.90.31.14:4663 , piece_id=U67TFMG72TDUBDDP2WAZWX3T3YQ377EKW7RKS753TJNPLCVRSAGQ, seg_pos={0 0}, upload=522360]: Done uploading piece. err: <nil> Dec 30 11:22:26.719862825 (595h53m26.59094587s) [seg_pos={0 0}, upload=522360]: Segment reached optimal threshold of 13 pieces. Dec 30 11:22:26.720199295 (595h53m26.59128234s) [noise=true , address=145.79.227.34:4905 , piece_id=NZEGGXF5EU64MG2JTYITNSBQJQ2YOE3RUEBXMLMWNOZGW6G2JS3A, seg_pos={0 0}, upload=522360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 11:22:26.720208283 (595h53m26.591291328s) [noise=true , address=145.79.227.34:4905 , piece_id=NZEGGXF5EU64MG2JTYITNSBQJQ2YOE3RUEBXMLMWNOZGW6G2JS3A, seg_pos={0 0}, upload=522360]: Long tail cancellation due to optimal threshold Dec 30 11:22:26.720577896 (595h53m26.591660941s) [noise=true , address=45.91.116.7:4011 , piece_id=ZOXPIR24N44ZCBTK3E7EMPTIYTV7KKUCNVDW7BVMQQCJIZUVLFFA, seg_pos={0 0}, upload=522360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:46298->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:46298->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46298->45.91.116.7:4011: use of closed network connection Dec 30 11:22:26.720587535 (595h53m26.59167057s) [noise=true , address=45.91.116.7:4011 , piece_id=ZOXPIR24N44ZCBTK3E7EMPTIYTV7KKUCNVDW7BVMQQCJIZUVLFFA, seg_pos={0 0}, upload=522360]: Long tail cancellation due to optimal threshold Dec 30 11:22:26.720620237 (595h53m26.591703282s) [noise=true , address=145.79.227.44:4193 , piece_id=VKSHKZ34JDN6RJOMZVCBN25SBG2OXDCLMEHSKECVOR4EEAY5JN6Q, seg_pos={0 0}, upload=522360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 11:22:26.720642148 (595h53m26.591725193s) [noise=true , address=145.79.227.44:4193 , piece_id=VKSHKZ34JDN6RJOMZVCBN25SBG2OXDCLMEHSKECVOR4EEAY5JN6Q, seg_pos={0 0}, upload=522360]: Long tail cancellation due to optimal threshold Dec 30 11:22:26.720688276 (595h53m26.591771321s) [seg_pos={0 0}, upload=522360]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:22:26.720836218 (595h53m26.591919263s) [upload=522360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:22:26.720844133 (595h53m26.591927178s) [upload=522360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:22:27.012933697 (595h53m26.884016742s) [upload=522360]: Done uploading segments... Dec 30 11:22:27.012953596 (595h53m26.884036631s) [upload=522360]: Upload finished. err: <nil> Duration: 30.002316803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X.................X..........X...X...........X.] duration: 15.831038123s trace: https://jaeger.dev.storj.io/trace/320f2449de9aa447 FAIL (2025-12-30 11:22:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8.212630018 (595h52m28.083713053s) [noise=true , address=145.79.227.44:4206 , piece_id=7QM25KZUSPAD4GTIDR67OJNMAEKLCBZRLUK3GZDB7TO2ZI5TEREA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.213211245 (595h52m28.08429428s) [noise=true , address=145.79.227.30:4788 , piece_id=TWTEPHC7KMREU47W7NCYNL6DEZ5XHPYD5QIKQZN6HZKR3OBPUT4A, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.213600286 (595h52m28.084683321s) [noise=true , address=77.90.31.11:4575 , piece_id=O3DCIOB6TOACOAKWBFE5IZTTYKUSWJBOKHC7LC2CRACYYBHXAHMA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.213779527 (595h52m28.084862562s) [noise=true , address=45.91.116.8:4049 , piece_id=XBRESQJASTTIPV3G2SJUGXKPJP4UD22X7B4QDP5YJUVTSLMJVTNA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.213888514 (595h52m28.084971559s) [noise=true , address=145.79.227.34:4937 , piece_id=4NXNJDUCZM5KLWS3K4ABDHODPCJHJYU42D2W5ON4LRS5ZM4LREHQ, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214146827 (595h52m28.085229862s) [noise=true , address=193.149.164.143:4703 , piece_id=EBPBXTU4XLXLN5FEV3WC3JMXBGWRSKRZOIOLIUP5ANY5AYTPLEIQ, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214153439 (595h52m28.085236474s) [noise=true , address=145.79.227.40:4057 , piece_id=JKX3DD2VZGTTXJDYLJVSOJ4HEAAGUZIPG7AMLT5HX4GLLJ7EJDLA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214505840 (595h52m28.085588875s) [noise=true , address=145.79.227.35:4489 , piece_id=3I25VU7FHAW24CCOAWF7YF6AIUA5LS37UL2BX7TJ4LBXINZT3ZCQ, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214601683 (595h52m28.085684718s) [noise=true , address=77.90.31.10:4529 , piece_id=433RPSCXRLK7I7746NZN33AKHX7QVW2QK6FU4YJMUXQXM6UW2KFA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214836058 (595h52m28.085919093s) [noise=true , address=145.79.227.33:4876 , piece_id=ULKTOEKRBWK3WSNO6YTJYXD7WRZTCKD2TVRXIGCT66SJA2HKMXEA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214886896 (595h52m28.085969931s) [noise=true , address=77.90.31.14:4683 , piece_id=DD3WJKOZA34IDFGTBRZFCMYBOU3EWARB2772KQHIVBIDS6XBG73A, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215143213 (595h52m28.086226248s) [noise=true , address=77.90.31.17:4790 , piece_id=7AQKSCDTY4TIVXYUIE2XXXUB2R2M7LVDA6W3RHZBVPAUDUHYX4PQ, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215224017 (595h52m28.086307052s) [noise=true , address=145.79.227.9:4385 , piece_id=O35GQDZODEC6HZNMUZMT47OLZFIMFGIYHYNPIIC66KW6MUJHVNPA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215431352 (595h52m28.086514387s) [noise=true , address=64.49.13.27:4965 , piece_id=V5NCPS3EFV7NABYT5KZ7E4TEQHICP4RFK5GKCHMMT45PJ2CE4S4Q, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215543906 (595h52m28.086626951s) [noise=true , address=64.49.13.26:4954 , piece_id=ZN2G36BDM3WARBVKBOCMGGKFQG7XKOFRIRVY3VKW4UMNSL4QHSTA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215913380 (595h52m28.086996415s) [noise=true , address=64.49.13.169:4709 , piece_id=L6DBEGT6YAR5J4I3FRIMLCDKAHJ5HTUGH7VATKOM5NXF2L3FYM7A, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.216329021 (595h52m28.087412056s) [noise=true , address=145.79.227.38:4995 , piece_id=XOROFSBZOSPYSC6LSNZJMSRS4KYNH7JMH3E542G77L3ZQ3SADW7A, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:29.220053219 (595h52m29.091136254s) [noise=true , address=64.49.13.27:4965 , piece_id=V5NCPS3EFV7NABYT5KZ7E4TEQHICP4RFK5GKCHMMT45PJ2CE4S4Q, seg_pos={0 0}, upload=522349]: Done uploading piece. err: ecclient: upload failed (node:12WVhm2QyhAf1y6HXRpyChMLbDLhZJQHEE2kQpstZH2dkwNhd7P, address:64.49.13.27:4965): protocol: storage node overloaded, request limit: 2000 Dec 30 11:21:29.637185454 (595h52m29.508268499s) [upload=522349]: Next returned nil segment Dec 30 11:21:29.637201474 (595h52m29.508284539s) [upload=522349]: Waiting for error group managing segments... Dec 30 11:21:30.066275167 (595h52m29.937358302s) [noise=true , address=145.79.227.9:4385 , piece_id=O35GQDZODEC6HZNMUZMT47OLZFIMFGIYHYNPIIC66KW6MUJHVNPA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.073548216 (595h52m29.944631251s) [noise=true , address=64.49.13.26:4954 , piece_id=ZN2G36BDM3WARBVKBOCMGGKFQG7XKOFRIRVY3VKW4UMNSL4QHSTA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.144482906 (595h52m30.015565942s) [noise=true , address=145.79.227.38:4995 , piece_id=XOROFSBZOSPYSC6LSNZJMSRS4KYNH7JMH3E542G77L3ZQ3SADW7A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.238892497 (595h52m30.109975532s) [noise=true , address=77.90.31.11:4575 , piece_id=O3DCIOB6TOACOAKWBFE5IZTTYKUSWJBOKHC7LC2CRACYYBHXAHMA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.366857342 (595h52m30.237940387s) [noise=true , address=45.91.116.11:4129 , piece_id=SQUUXN3L7HVY6PLVPCH5KAZK43MTWTZ3MI5NG6H2RANCMDOGIY5A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.375068928 (595h52m30.246151973s) [noise=true , address=77.90.31.14:4683 , piece_id=DD3WJKOZA34IDFGTBRZFCMYBOU3EWARB2772KQHIVBIDS6XBG73A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.641601427 (595h52m30.512684462s) [noise=true , address=145.79.227.40:4057 , piece_id=JKX3DD2VZGTTXJDYLJVSOJ4HEAAGUZIPG7AMLT5HX4GLLJ7EJDLA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.979636920 (595h52m30.850719955s) [noise=true , address=145.79.227.30:4788 , piece_id=TWTEPHC7KMREU47W7NCYNL6DEZ5XHPYD5QIKQZN6HZKR3OBPUT4A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.983123610 (595h52m30.854206645s) [noise=true , address=145.79.227.34:4937 , piece_id=4NXNJDUCZM5KLWS3K4ABDHODPCJHJYU42D2W5ON4LRS5ZM4LREHQ, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.031177872 (595h52m30.902260907s) [noise=true , address=145.79.227.44:4206 , piece_id=7QM25KZUSPAD4GTIDR67OJNMAEKLCBZRLUK3GZDB7TO2ZI5TEREA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.064351170 (595h52m30.935434205s) [noise=true , address=145.79.227.35:4489 , piece_id=3I25VU7FHAW24CCOAWF7YF6AIUA5LS37UL2BX7TJ4LBXINZT3ZCQ, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.125491260 (595h52m30.996574295s) [noise=true , address=145.79.227.33:4876 , piece_id=ULKTOEKRBWK3WSNO6YTJYXD7WRZTCKD2TVRXIGCT66SJA2HKMXEA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.261432112 (595h52m31.132515157s) [noise=true , address=45.91.116.8:4049 , piece_id=XBRESQJASTTIPV3G2SJUGXKPJP4UD22X7B4QDP5YJUVTSLMJVTNA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.703247748 (595h52m31.574330783s) [noise=true , address=77.90.31.10:4529 , piece_id=433RPSCXRLK7I7746NZN33AKHX7QVW2QK6FU4YJMUXQXM6UW2KFA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.703272755 (595h52m31.574355801s) [seg_pos={0 0}, upload=522349]: Segment reached optimal threshold of 14 pieces. Dec 30 11:21:31.703520197 (595h52m31.574603252s) [noise=true , address=64.49.13.169:4709 , piece_id=L6DBEGT6YAR5J4I3FRIMLCDKAHJ5HTUGH7VATKOM5NXF2L3FYM7A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:56546->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:56546->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56546->64.49.13.169:4709: use of closed network connection Dec 30 11:21:31.703529584 (595h52m31.574612629s) [noise=true , address=64.49.13.169:4709 , piece_id=L6DBEGT6YAR5J4I3FRIMLCDKAHJ5HTUGH7VATKOM5NXF2L3FYM7A, seg_pos={0 0}, upload=522349]: Long tail cancellation due to optimal threshold Dec 30 11:21:31.703752048 (595h52m31.574835093s) [noise=true , address=193.149.164.143:4703 , piece_id=EBPBXTU4XLXLN5FEV3WC3JMXBGWRSKRZOIOLIUP5ANY5AYTPLEIQ, seg_pos={0 0}, upload=522349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eomwcuHK58QuxBp3sAARJjm6Doc7zEzRYCodsTzFT4v6Nti9g): protocol: expected piece hash; context canceled; EOF Dec 30 11:21:31.703758611 (595h52m31.574841646s) [noise=true , address=77.90.31.17:4790 , piece_id=7AQKSCDTY4TIVXYUIE2XXXUB2R2M7LVDA6W3RHZBVPAUDUHYX4PQ, seg_pos={0 0}, upload=522349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b): protocol: expected piece hash; context canceled; EOF Dec 30 11:21:31.703778147 (595h52m31.574861192s) [noise=true , address=77.90.31.17:4790 , piece_id=7AQKSCDTY4TIVXYUIE2XXXUB2R2M7LVDA6W3RHZBVPAUDUHYX4PQ, seg_pos={0 0}, upload=522349]: Long tail cancellation due to optimal threshold Dec 30 11:21:31.703806712 (595h52m31.574889747s) [noise=true , address=193.149.164.143:4703 , piece_id=EBPBXTU4XLXLN5FEV3WC3JMXBGWRSKRZOIOLIUP5ANY5AYTPLEIQ, seg_pos={0 0}, upload=522349]: Long tail cancellation due to optimal threshold Dec 30 11:21:31.703837070 (595h52m31.574920115s) [seg_pos={0 0}, upload=522349]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:21:31.703965946 (595h52m31.575048981s) [upload=522349]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:21:31.703994861 (595h52m31.575077896s) [upload=522349]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:21:31.987657769 (595h52m31.858740814s) [upload=522349]: Done uploading segments... Dec 30 11:21:31.987682225 (595h52m31.85876526s) [upload=522349]: Upload finished. err: <nil> Duration: 15.83105265s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [XXXX.X.XX.X.X.....XX.X........X.X.XX.X.X.X.X...X.XX..XXX.X..] duration: 4.105180753s trace: https://jaeger.dev.storj.io/trace/5cf4b0dbd1997ef4 FAIL (2025-12-30 11:22:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.105202124s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X.................X..........X...X...........X] duration: 1m0.002583598s trace: https://jaeger.dev.storj.io/trace/320f2449de9aa447 FAIL (2025-12-30 11:22:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 8.212630018 (595h52m28.083713053s) [noise=true , address=145.79.227.44:4206 , piece_id=7QM25KZUSPAD4GTIDR67OJNMAEKLCBZRLUK3GZDB7TO2ZI5TEREA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.213211245 (595h52m28.08429428s) [noise=true , address=145.79.227.30:4788 , piece_id=TWTEPHC7KMREU47W7NCYNL6DEZ5XHPYD5QIKQZN6HZKR3OBPUT4A, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.213600286 (595h52m28.084683321s) [noise=true , address=77.90.31.11:4575 , piece_id=O3DCIOB6TOACOAKWBFE5IZTTYKUSWJBOKHC7LC2CRACYYBHXAHMA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.213779527 (595h52m28.084862562s) [noise=true , address=45.91.116.8:4049 , piece_id=XBRESQJASTTIPV3G2SJUGXKPJP4UD22X7B4QDP5YJUVTSLMJVTNA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.213888514 (595h52m28.084971559s) [noise=true , address=145.79.227.34:4937 , piece_id=4NXNJDUCZM5KLWS3K4ABDHODPCJHJYU42D2W5ON4LRS5ZM4LREHQ, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214146827 (595h52m28.085229862s) [noise=true , address=193.149.164.143:4703 , piece_id=EBPBXTU4XLXLN5FEV3WC3JMXBGWRSKRZOIOLIUP5ANY5AYTPLEIQ, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214153439 (595h52m28.085236474s) [noise=true , address=145.79.227.40:4057 , piece_id=JKX3DD2VZGTTXJDYLJVSOJ4HEAAGUZIPG7AMLT5HX4GLLJ7EJDLA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214505840 (595h52m28.085588875s) [noise=true , address=145.79.227.35:4489 , piece_id=3I25VU7FHAW24CCOAWF7YF6AIUA5LS37UL2BX7TJ4LBXINZT3ZCQ, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214601683 (595h52m28.085684718s) [noise=true , address=77.90.31.10:4529 , piece_id=433RPSCXRLK7I7746NZN33AKHX7QVW2QK6FU4YJMUXQXM6UW2KFA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214836058 (595h52m28.085919093s) [noise=true , address=145.79.227.33:4876 , piece_id=ULKTOEKRBWK3WSNO6YTJYXD7WRZTCKD2TVRXIGCT66SJA2HKMXEA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.214886896 (595h52m28.085969931s) [noise=true , address=77.90.31.14:4683 , piece_id=DD3WJKOZA34IDFGTBRZFCMYBOU3EWARB2772KQHIVBIDS6XBG73A, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215143213 (595h52m28.086226248s) [noise=true , address=77.90.31.17:4790 , piece_id=7AQKSCDTY4TIVXYUIE2XXXUB2R2M7LVDA6W3RHZBVPAUDUHYX4PQ, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215224017 (595h52m28.086307052s) [noise=true , address=145.79.227.9:4385 , piece_id=O35GQDZODEC6HZNMUZMT47OLZFIMFGIYHYNPIIC66KW6MUJHVNPA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215431352 (595h52m28.086514387s) [noise=true , address=64.49.13.27:4965 , piece_id=V5NCPS3EFV7NABYT5KZ7E4TEQHICP4RFK5GKCHMMT45PJ2CE4S4Q, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215543906 (595h52m28.086626951s) [noise=true , address=64.49.13.26:4954 , piece_id=ZN2G36BDM3WARBVKBOCMGGKFQG7XKOFRIRVY3VKW4UMNSL4QHSTA, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.215913380 (595h52m28.086996415s) [noise=true , address=64.49.13.169:4709 , piece_id=L6DBEGT6YAR5J4I3FRIMLCDKAHJ5HTUGH7VATKOM5NXF2L3FYM7A, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:28.216329021 (595h52m28.087412056s) [noise=true , address=145.79.227.38:4995 , piece_id=XOROFSBZOSPYSC6LSNZJMSRS4KYNH7JMH3E542G77L3ZQ3SADW7A, seg_pos={0 0}, upload=522349]: Uploading piece... Dec 30 11:21:29.220053219 (595h52m29.091136254s) [noise=true , address=64.49.13.27:4965 , piece_id=V5NCPS3EFV7NABYT5KZ7E4TEQHICP4RFK5GKCHMMT45PJ2CE4S4Q, seg_pos={0 0}, upload=522349]: Done uploading piece. err: ecclient: upload failed (node:12WVhm2QyhAf1y6HXRpyChMLbDLhZJQHEE2kQpstZH2dkwNhd7P, address:64.49.13.27:4965): protocol: storage node overloaded, request limit: 2000 Dec 30 11:21:29.637185454 (595h52m29.508268499s) [upload=522349]: Next returned nil segment Dec 30 11:21:29.637201474 (595h52m29.508284539s) [upload=522349]: Waiting for error group managing segments... Dec 30 11:21:30.066275167 (595h52m29.937358302s) [noise=true , address=145.79.227.9:4385 , piece_id=O35GQDZODEC6HZNMUZMT47OLZFIMFGIYHYNPIIC66KW6MUJHVNPA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.073548216 (595h52m29.944631251s) [noise=true , address=64.49.13.26:4954 , piece_id=ZN2G36BDM3WARBVKBOCMGGKFQG7XKOFRIRVY3VKW4UMNSL4QHSTA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.144482906 (595h52m30.015565942s) [noise=true , address=145.79.227.38:4995 , piece_id=XOROFSBZOSPYSC6LSNZJMSRS4KYNH7JMH3E542G77L3ZQ3SADW7A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.238892497 (595h52m30.109975532s) [noise=true , address=77.90.31.11:4575 , piece_id=O3DCIOB6TOACOAKWBFE5IZTTYKUSWJBOKHC7LC2CRACYYBHXAHMA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.366857342 (595h52m30.237940387s) [noise=true , address=45.91.116.11:4129 , piece_id=SQUUXN3L7HVY6PLVPCH5KAZK43MTWTZ3MI5NG6H2RANCMDOGIY5A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.375068928 (595h52m30.246151973s) [noise=true , address=77.90.31.14:4683 , piece_id=DD3WJKOZA34IDFGTBRZFCMYBOU3EWARB2772KQHIVBIDS6XBG73A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.641601427 (595h52m30.512684462s) [noise=true , address=145.79.227.40:4057 , piece_id=JKX3DD2VZGTTXJDYLJVSOJ4HEAAGUZIPG7AMLT5HX4GLLJ7EJDLA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.979636920 (595h52m30.850719955s) [noise=true , address=145.79.227.30:4788 , piece_id=TWTEPHC7KMREU47W7NCYNL6DEZ5XHPYD5QIKQZN6HZKR3OBPUT4A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:30.983123610 (595h52m30.854206645s) [noise=true , address=145.79.227.34:4937 , piece_id=4NXNJDUCZM5KLWS3K4ABDHODPCJHJYU42D2W5ON4LRS5ZM4LREHQ, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.031177872 (595h52m30.902260907s) [noise=true , address=145.79.227.44:4206 , piece_id=7QM25KZUSPAD4GTIDR67OJNMAEKLCBZRLUK3GZDB7TO2ZI5TEREA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.064351170 (595h52m30.935434205s) [noise=true , address=145.79.227.35:4489 , piece_id=3I25VU7FHAW24CCOAWF7YF6AIUA5LS37UL2BX7TJ4LBXINZT3ZCQ, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.125491260 (595h52m30.996574295s) [noise=true , address=145.79.227.33:4876 , piece_id=ULKTOEKRBWK3WSNO6YTJYXD7WRZTCKD2TVRXIGCT66SJA2HKMXEA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.261432112 (595h52m31.132515157s) [noise=true , address=45.91.116.8:4049 , piece_id=XBRESQJASTTIPV3G2SJUGXKPJP4UD22X7B4QDP5YJUVTSLMJVTNA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.703247748 (595h52m31.574330783s) [noise=true , address=77.90.31.10:4529 , piece_id=433RPSCXRLK7I7746NZN33AKHX7QVW2QK6FU4YJMUXQXM6UW2KFA, seg_pos={0 0}, upload=522349]: Done uploading piece. err: <nil> Dec 30 11:21:31.703272755 (595h52m31.574355801s) [seg_pos={0 0}, upload=522349]: Segment reached optimal threshold of 14 pieces. Dec 30 11:21:31.703520197 (595h52m31.574603252s) [noise=true , address=64.49.13.169:4709 , piece_id=L6DBEGT6YAR5J4I3FRIMLCDKAHJ5HTUGH7VATKOM5NXF2L3FYM7A, seg_pos={0 0}, upload=522349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:56546->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:56546->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56546->64.49.13.169:4709: use of closed network connection Dec 30 11:21:31.703529584 (595h52m31.574612629s) [noise=true , address=64.49.13.169:4709 , piece_id=L6DBEGT6YAR5J4I3FRIMLCDKAHJ5HTUGH7VATKOM5NXF2L3FYM7A, seg_pos={0 0}, upload=522349]: Long tail cancellation due to optimal threshold Dec 30 11:21:31.703752048 (595h52m31.574835093s) [noise=true , address=193.149.164.143:4703 , piece_id=EBPBXTU4XLXLN5FEV3WC3JMXBGWRSKRZOIOLIUP5ANY5AYTPLEIQ, seg_pos={0 0}, upload=522349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eomwcuHK58QuxBp3sAARJjm6Doc7zEzRYCodsTzFT4v6Nti9g): protocol: expected piece hash; context canceled; EOF Dec 30 11:21:31.703758611 (595h52m31.574841646s) [noise=true , address=77.90.31.17:4790 , piece_id=7AQKSCDTY4TIVXYUIE2XXXUB2R2M7LVDA6W3RHZBVPAUDUHYX4PQ, seg_pos={0 0}, upload=522349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b): protocol: expected piece hash; context canceled; EOF Dec 30 11:21:31.703778147 (595h52m31.574861192s) [noise=true , address=77.90.31.17:4790 , piece_id=7AQKSCDTY4TIVXYUIE2XXXUB2R2M7LVDA6W3RHZBVPAUDUHYX4PQ, seg_pos={0 0}, upload=522349]: Long tail cancellation due to optimal threshold Dec 30 11:21:31.703806712 (595h52m31.574889747s) [noise=true , address=193.149.164.143:4703 , piece_id=EBPBXTU4XLXLN5FEV3WC3JMXBGWRSKRZOIOLIUP5ANY5AYTPLEIQ, seg_pos={0 0}, upload=522349]: Long tail cancellation due to optimal threshold Dec 30 11:21:31.703837070 (595h52m31.574920115s) [seg_pos={0 0}, upload=522349]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:21:31.703965946 (595h52m31.575048981s) [upload=522349]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:21:31.703994861 (595h52m31.575077896s) [upload=522349]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:21:31.987657769 (595h52m31.858740814s) [upload=522349]: Done uploading segments... Dec 30 11:21:31.987682225 (595h52m31.85876526s) [upload=522349]: Upload finished. err: <nil> Duration: 1m0.002592615s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [.XXXX.X.XX.X.X.....XX.X........X.X.XX.X.X.X.X...X.XX..XXX.X.] duration: 3.814549324s trace: https://jaeger.dev.storj.io/trace/5cf4b0dbd1997ef4 FAIL (2025-12-30 11:22:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.814583661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.......X.......X..XX..X.X.X..........X....X.......X.....X..X] duration: 30.003162253s trace: https://jaeger.dev.storj.io/trace/7f517abb442427f7 FAIL (2025-12-30 11:21:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: XSDJRDRRHL2GKVL2PFVYLDYWAUOUM3U2SXRFDRY5ABJQK7FQ4XUQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: SH7SBFD3KEGQ2LKDSXGNXDRCZ3BDE2Q6M7CYBEVUOSXLAU6HWX2Q) context deadline exceeded ======LOGS====== Dec 30 11:21:26.788066666 (595h52m26.659149711s) [upload=522348]: Starting upload Dec 30 11:21:26.788132812 (595h52m26.659215858s) [upload=522348]: Done starting upload Dec 30 11:21:26.788320300 (595h52m26.659403355s) [upload=522348]: Uploading segments... Dec 30 11:21:26.788417454 (595h52m26.659500489s) [upload=522348]: Got next segment. Inline: false Dec 30 11:21:26.788458823 (595h52m26.659541868s) [upload=522348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:21:26.788465506 (595h52m26.659548571s) [upload=522348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:21:27.031930121 (595h52m26.903013186s) [seg_pos={0 0}, upload=522348]: Begin upload segment... Dec 30 11:21:27.031987570 (595h52m26.903070635s) [seg_pos={0 0}, upload=522348]: Done begin upload segment. Dec 30 11:21:27.032023869 (595h52m26.903106934s) [noise=true , address=45.91.116.5:4300 , piece_id=FJXFEVNWD2ZESV7R5MM5D23PK3WWHQHX3SLBWKGD7P4VWHFOH4CA, seg_pos={0 0}, upload=522348]: Uploading piece... Dec 30 11:21:27.032102178 (595h52m26.903185243s) [noise=true , address=77.90.31.14:4663 , piece_id=M3Q5RNCKAYJHV6K5AYS42RGVA4E4MJWEZKZQ2B52BXXEBLACHPAA, seg_pos={0 0}, upload=522348]: Uploading piece... Dec 30 11:21:27.032759179 (595h52m26.903842214s) [noise=true , address=145.79.227.47:4267 , piece_id=ZLRCKHWZAF6FJAYEWMPDE6BWTSAGW5YIWJ5MKIM25UVD74FCMTTQ, seg_pos={0 0}, upload=522348]: Uploading piece... Dec 30 11:21:27.033063298 (595h52m26.904146333s) [noise=true , address=145.79.227.52:4446 , piece_id=EJU2D7ERSJMN44T6PHX6UZLJU66SHYQC2G5DHFJE6DQB5FF6K2QQ, seg_pos={0 0}, upload=522348]: Uploading piece... Dec 30 11:21:27.033350184 (595h52m26.904433219s) [noise=true , address=45.91.116.12:4228 , piece_id=TQYA3TYWGT7OAW6IEKJJMX54RP66B6HDYZ2VBV2BEHVY3WO7PY2A, seg_pos={0 0}, upload=522348]: Uploading piece... Dec 30 11:21:27.033769763 (595h52m26.904852808s) [noise=true , address=145.79.227.42:4123 , piece_id=JQD7ZD7TTYYRAWHN7CB56JWBDASPE74LSC6IJLQTMFCWGDECST2A, seg_pos={0 0}, upload=522348]: Uploading piece... Dec 30 11:21:27.034084603 (595h52m26.905167648s) [noise=true , address=77.90.31.19:4844 , piece_id=SH7SBFD3KEGQ2LKDSXGNXDRCZ3BDE2Q6M7CYBEVUOSXLAU6HWX2Q, seg_pos={0 0}, upload=522348]: Uploading piece... Dec 30 11:21:27.034429469 (595h52m26.905512515s) [noise=true , address=145.79.227.37:4566 , piece_id=2ZF76YW7EKPGF64PASDXPWZDSJ676YW4OWCIRYGH2X4XZYNCARXQ, seg_pos={0 0}, upload=522348]: Uploading piece... Dec 30 11:21:27.034612268 (595h52m26.905695303s) [noise=true , address=77.90.31.13:4626 , piece_id=XSDJRDRRHL2GKVL2PFVYLDYWAUOUM3U2SXRFDRY5ABJQK7FQ4XUQ, seg_pos={0 0}, upload=522348]: Uploading piece... Dec 30 11:21:27.034980659 (595h52m26.906063694s) [noise=true , address=145.79.227.10:4403 , piece_id=BVQZEMLY32EQV3SF3YEUSBS7LXEVPSQSS7PVSFHQA5QSGUVB3DZQ, seg_pos={0 0}, upload=522348]: Uploading piece... Dec 30 11:21:27.035013231 (595h52m26.906096266s) [noise=true , address=145.79.227.29:4943 , piece_id=WMEHL3GATJXWHS5RFIQ33UGJN7TSMFVLF6AMXYYTAKJJ5DCIF7JQ, seg_pos={0 0}, upload=522348]: Uploading piece... Dec 30 11:21:27.035440565 (595h52m26.9065236s) [noise=true , address=145.79.227.35:4481 , piece_id=6JB6QBWHUUUPNR5NLB5FJCYLFXZ2GUP772Y7UQ6RERT3ONIPWC3Q, seg_pos={0 0}, upload=522348]: Uploading piece... Dec 30 11:21:27.035531548 (595h52m26.906614593s) [noise=true , address=145.79.227.3:4151 , piece_id=SOLNQYPHACMSCAHJAZBMA4XUO2VQXEVPHHJQ7HTW3RUUN6OOY2LQ, seg_pos={0 0}, upload=522348]: Uploading piece... Dec 30 11:21:27.035826289 (595h52m26.906909324s) [noise=true , address=145.79.227.41:4087 , piece_id=YM2ZIKEZILUTXMTRQIXJ4NRI7MPYBDMDRUB6FCM4OLIGSQM6HYRA, seg_pos={0 0}, upload=522348]: Uploading piece... Dec 30 11:21:27.036128365 (595h52m26.90721141s) [noise=true , address=145.79.227.36:4517 , piece_id=ELK7DAFDBGKGTFPSLNZIDFIFPIEULPINIHGOF7H5C7LM4PRKXVAA, seg_pos={0 0}, upload=522348]: Uploading piece... Dec 30 11:21:27.036483631 (595h52m26.907566666s) [noise=true , address=45.91.116.6:4976 , piece_id=DXV5P6VMPOUHNIQRUGANTR7HOKTODLO73SXUYTZ3J27TJBV7OILA, seg_pos={0 0}, upload=522348]: Uploading piece... Dec 30 11:21:28.736578507 (595h52m28.607661552s) [upload=522348]: Next returned nil segment Dec 30 11:21:28.736599658 (595h52m28.607682703s) [upload=522348]: Waiting for error group managing segments... Dec 30 11:21:29.021773803 (595h52m28.892856838s) [noise=true , address=145.79.227.41:4087 , piece_id=YM2ZIKEZILUTXMTRQIXJ4NRI7MPYBDMDRUB6FCM4OLIGSQM6HYRA, seg_pos={0 0}, upload=522348]: Done uploading piece. err: <nil> Dec 30 11:21:29.026828840 (595h52m28.897911875s) [noise=true , address=145.79.227.10:4403 , piece_id=BVQZEMLY32EQV3SF3YEUSBS7LXEVPSQSS7PVSFHQA5QSGUVB3DZQ, seg_pos={0 0}, upload=522348]: Done uploading piece. err: <nil> Dec 30 11:21:29.031030680 (595h52m28.902113715s) [noise=true , address=145.79.227.29:4943 , piece_id=WMEHL3GATJXWHS5RFIQ33UGJN7TSMFVLF6AMXYYTAKJJ5DCIF7JQ, seg_pos={0 0}, upload=522348]: Done uploading piece. err: <nil> Dec 30 11:21:29.039493575 (595h52m28.91057661s) [noise=true , address=145.79.227.47:4267 , piece_id=ZLRCKHWZAF6FJAYEWMPDE6BWTSAGW5YIWJ5MKIM25UVD74FCMTTQ, seg_pos={0 0}, upload=522348]: Done uploading piece. err: <nil> Dec 30 11:21:29.043177289 (595h52m28.914260334s) [noise=true , address=145.79.227.52:4446 , piece_id=EJU2D7ERSJMN44T6PHX6UZLJU66SHYQC2G5DHFJE6DQB5FF6K2QQ, seg_pos={0 0}, upload=522348]: Done uploading piece. err: <nil> Dec 30 11:21:29.046399014 (595h52m28.917482049s) [noise=true , address=145.79.227.37:4566 , piece_id=2ZF76YW7EKPGF64PASDXPWZDSJ676YW4OWCIRYGH2X4XZYNCARXQ, seg_pos={0 0}, upload=522348]: Done uploading piece. err: <nil> Dec 30 11:21:29.056259210 (595h52m28.927342265s) [noise=true , address=145.79.227.42:4123 , piece_id=JQD7ZD7TTYYRAWHN7CB56JWBDASPE74LSC6IJLQTMFCWGDECST2A, seg_pos={0 0}, upload=522348]: Done uploading piece. err: <nil> Dec 30 11:21:29.107048771 (595h52m28.978131806s) [noise=true , address=145.79.227.36:4517 , piece_id=ELK7DAFDBGKGTFPSLNZIDFIFPIEULPINIHGOF7H5C7LM4PRKXVAA, seg_pos={0 0}, upload=522348]: Done uploading piece. err: <nil> Dec 30 11:21:29.112271876 (595h52m28.983354911s) [noise=true , address=145.79.227.3:4151 , piece_id=SOLNQYPHACMSCAHJAZBMA4XUO2VQXEVPHHJQ7HTW3RUUN6OOY2LQ, seg_pos={0 0}, upload=522348]: Done uploading piece. err: <nil> Dec 30 11:21:29.199229647 (595h52m29.070312682s) [noise=true , address=77.90.31.14:4663 , piece_id=M3Q5RNCKAYJHV6K5AYS42RGVA4E4MJWEZKZQ2B52BXXEBLACHPAA, seg_pos={0 0}, upload=522348]: Done uploading piece. err: <nil> Dec 30 11:21:29.230520711 (595h52m29.101603746s) [noise=true , address=77.90.31.19:4844 , piece_id=SH7SBFD3KEGQ2LKDSXGNXDRCZ3BDE2Q6M7CYBEVUOSXLAU6HWX2Q, seg_pos={0 0}, upload=522348]: Done uploading piece. err: <nil> Dec 30 11:21:29.310487940 (595h52m29.181570985s) [noise=true , address=77.90.31.13:4626 , piece_id=XSDJRDRRHL2GKVL2PFVYLDYWAUOUM3U2SXRFDRY5ABJQK7FQ4XUQ, seg_pos={0 0}, upload=522348]: Done uploading piece. err: <nil> Dec 30 11:21:29.352650885 (595h52m29.22373393s) [noise=true , address=45.91.116.5:4300 , piece_id=FJXFEVNWD2ZESV7R5MM5D23PK3WWHQHX3SLBWKGD7P4VWHFOH4CA, seg_pos={0 0}, upload=522348]: Done uploading piece. err: <nil> Dec 30 11:21:29.352670352 (595h52m29.223753387s) [seg_pos={0 0}, upload=522348]: Segment reached optimal threshold of 13 pieces. Dec 30 11:21:29.352841368 (595h52m29.223924403s) [noise=true , address=145.79.227.35:4481 , piece_id=6JB6QBWHUUUPNR5NLB5FJCYLFXZ2GUP772Y7UQ6RERT3ONIPWC3Q, seg_pos={0 0}, upload=522348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 30 11:21:29.352847569 (595h52m29.223930604s) [noise=true , address=145.79.227.35:4481 , piece_id=6JB6QBWHUUUPNR5NLB5FJCYLFXZ2GUP772Y7UQ6RERT3ONIPWC3Q, seg_pos={0 0}, upload=522348]: Long tail cancellation due to optimal threshold Dec 30 11:21:29.352975142 (595h52m29.224058178s) [noise=true , address=45.91.116.12:4228 , piece_id=TQYA3TYWGT7OAW6IEKJJMX54RP66B6HDYZ2VBV2BEHVY3WO7PY2A, seg_pos={0 0}, upload=522348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 30 11:21:29.352982126 (595h52m29.224065161s) [noise=true , address=45.91.116.12:4228 , piece_id=TQYA3TYWGT7OAW6IEKJJMX54RP66B6HDYZ2VBV2BEHVY3WO7PY2A, seg_pos={0 0}, upload=522348]: Long tail cancellation due to optimal threshold Dec 30 11:21:29.353126752 (595h52m29.224209787s) [noise=true , address=45.91.116.6:4976 , piece_id=DXV5P6VMPOUHNIQRUGANTR7HOKTODLO73SXUYTZ3J27TJBV7OILA, seg_pos={0 0}, upload=522348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 30 11:21:29.353142321 (595h52m29.224225366s) [noise=true , address=45.91.116.6:4976 , piece_id=DXV5P6VMPOUHNIQRUGANTR7HOKTODLO73SXUYTZ3J27TJBV7OILA, seg_pos={0 0}, upload=522348]: Long tail cancellation due to optimal threshold Dec 30 11:21:29.353168030 (595h52m29.224251065s) [seg_pos={0 0}, upload=522348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:21:29.353285273 (595h52m29.224368308s) [upload=522348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:21:29.353290733 (595h52m29.224373768s) [upload=522348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:21:29.646848313 (595h52m29.517931358s) [upload=522348]: Done uploading segments... Dec 30 11:21:29.646867679 (595h52m29.517950714s) [upload=522348]: Upload finished. err: <nil> Duration: 30.003179936s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [X.XXXX.X.XX.X.X.....XX.X........X.X.XX.X.X.X.X...X.XX..XXX.X] duration: 5.001686832s trace: https://jaeger.dev.storj.io/trace/5cf4b0dbd1997ef4 FAIL (2025-12-30 11:21:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001706268s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X.................X..........X...X...........] duration: 58.695602673s trace: https://jaeger.dev.storj.io/trace/7f3400b3387747 FAIL (2025-12-30 11:21:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, Piece ID: OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA) context deadline exceeded; piecestore: (Node ID: 12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, Piece ID: SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ) context deadline exceeded; piecestore: (Node ID: 1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC, Piece ID: 6ZSKVY2RVZAJBTRBKT4RTVQKMJ2BVQDU53I453YBIBBHCZOQCW4A) context deadline exceeded; piecestore: (Node ID: 12YZZnCo54XknuvoBoVtRudXSnoWdNdrn23Lp6Gou3g451BzUwG, Piece ID: 7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ) context deadline exceeded ======LOGS====== s) [noise=true , address=145.79.227.18:4038 , piece_id=K6PIAQ572FP7PVR5G3BRZWSGVYK7XTXG6Z4KBZZ3G5A732BOLS4A, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.892668738 (595h43m54.763751783s) [noise=true , address=145.79.227.40:4079 , piece_id=43LPLVACHLV3VIUR7YZ4RGRCG65FGVSYM5EOPOSG6W6YUSL6OLPQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893082295 (595h43m54.76416534s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893484291 (595h43m54.764567336s) [noise=true , address=77.90.31.12:4613 , piece_id=OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893955759 (595h43m54.765038794s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.894536003 (595h43m54.765619048s) [noise=true , address=145.79.227.44:4226 , piece_id=JZ5CAXU3XK56URASDAYQHQOCON7SCFEECMSM4NJ7O33M647UWAZQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.894858427 (595h43m54.765941462s) [noise=true , address=64.49.13.155:4920 , piece_id=UE53XZKQ5VW5CDBMS7SF3R2FMA3FVNUQA2RLMFMYIIHPDDSNXTZQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.895172897 (595h43m54.766255942s) [noise=true , address=145.79.227.33:4886 , piece_id=GYC5TKM6I5QRBZJCJVRO35ITEGFZWRXEUTA3V76Y65CZ7ZZEDZSQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.896682591 (595h43m54.767765626s) [noise=true , address=145.79.227.29:4953 , piece_id=VTIGSELICMISVDGASJP4W6Z4IOIALQKRPZEULAODHJL4JQHUU7RA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.897870683 (595h43m54.768953728s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.899025873 (595h43m54.770108918s) [noise=true , address=145.79.227.30:4792 , piece_id=T7WHB25ZUMXYNNPCPLQOKLGUPLTQB3XQJ6ZCE3XRXDOYJWJHXFLA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.900208054 (595h43m54.771291099s) [noise=true , address=77.90.31.18:4827 , piece_id=RXOZKEZUKJA4GXKQUZ6ITP2ZW7TPDDXMF7VINT36HGCQNCDCCB2Q, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.901469476 (595h43m54.772552521s) [noise=true , address=77.90.31.15:4706 , piece_id=7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.902861116 (595h43m54.773944161s) [noise=true , address=145.79.227.17:4987 , piece_id=JNZPAZIFJMCQWJELT26MLRIF46UUIYRYD3EKVFX42VIAXYQU4B5Q, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.904040151 (595h43m54.775123196s) [noise=true , address=77.90.31.16:4743 , piece_id=SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.905225688 (595h43m54.776308723s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:56.312272123 (595h43m56.183355158s) [upload=522223]: Next returned nil segment Dec 30 11:12:56.312288104 (595h43m56.183371139s) [upload=522223]: Waiting for error group managing segments... Dec 30 11:12:56.674443063 (595h43m56.545526118s) [noise=true , address=64.49.13.155:4920 , piece_id=UE53XZKQ5VW5CDBMS7SF3R2FMA3FVNUQA2RLMFMYIIHPDDSNXTZQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.748148810 (595h43m56.619231855s) [noise=true , address=145.79.227.17:4987 , piece_id=JNZPAZIFJMCQWJELT26MLRIF46UUIYRYD3EKVFX42VIAXYQU4B5Q, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.748294467 (595h43m56.619377502s) [noise=true , address=145.79.227.18:4038 , piece_id=K6PIAQ572FP7PVR5G3BRZWSGVYK7XTXG6Z4KBZZ3G5A732BOLS4A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.898059835 (595h43m56.76914288s) [noise=true , address=145.79.227.3:4166 , piece_id=GJBVBRV7YZE3GTXEKBW7LJYW35EXK4SDEVIFNZMETDKCZOXCIWKA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.936973813 (595h43m56.808056848s) [noise=true , address=77.90.31.21:4893 , piece_id=6ZSKVY2RVZAJBTRBKT4RTVQKMJ2BVQDU53I453YBIBBHCZOQCW4A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.961395065 (595h43m56.83247811s) [noise=true , address=77.90.31.18:4827 , piece_id=RXOZKEZUKJA4GXKQUZ6ITP2ZW7TPDDXMF7VINT36HGCQNCDCCB2Q, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.967043209 (595h43m56.838126254s) [noise=true , address=77.90.31.15:4706 , piece_id=7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.059782038 (595h43m56.930865073s) [noise=true , address=77.90.31.16:4743 , piece_id=SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.692145558 (595h43m57.563228633s) [noise=true , address=145.79.227.30:4792 , piece_id=T7WHB25ZUMXYNNPCPLQOKLGUPLTQB3XQJ6ZCE3XRXDOYJWJHXFLA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.705427259 (595h43m57.576510304s) [noise=true , address=145.79.227.33:4886 , piece_id=GYC5TKM6I5QRBZJCJVRO35ITEGFZWRXEUTA3V76Y65CZ7ZZEDZSQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.745163051 (595h43m57.616246096s) [noise=true , address=145.79.227.40:4079 , piece_id=43LPLVACHLV3VIUR7YZ4RGRCG65FGVSYM5EOPOSG6W6YUSL6OLPQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.776805663 (595h43m57.647888708s) [noise=true , address=145.79.227.44:4226 , piece_id=JZ5CAXU3XK56URASDAYQHQOCON7SCFEECMSM4NJ7O33M647UWAZQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.859064686 (595h43m57.730147731s) [noise=true , address=145.79.227.29:4953 , piece_id=VTIGSELICMISVDGASJP4W6Z4IOIALQKRPZEULAODHJL4JQHUU7RA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.882701444 (595h43m57.753784479s) [noise=true , address=77.90.31.12:4613 , piece_id=OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.882722825 (595h43m57.75380587s) [seg_pos={0 0}, upload=522223]: Segment reached optimal threshold of 14 pieces. Dec 30 11:12:57.882954816 (595h43m57.754037851s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.882964956 (595h43m57.754048011s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883110403 (595h43m57.754193448s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.883123828 (595h43m57.754206883s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883237274 (595h43m57.754320339s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection Dec 30 11:12:57.883248406 (595h43m57.754331451s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883390406 (595h43m57.754473451s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.883415915 (595h43m57.75449896s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883463475 (595h43m57.75454652s) [seg_pos={0 0}, upload=522223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:12:57.883549108 (595h43m57.754632153s) [upload=522223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:12:57.883554960 (595h43m57.754638005s) [upload=522223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:12:58.175520738 (595h43m58.046603773s) [upload=522223]: Done uploading segments... Dec 30 11:12:58.175538943 (595h43m58.046621978s) [upload=522223]: Upload finished. err: <nil> Duration: 58.695621528s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [.X.XXXX.X.XX.X.X.....XX.X........X.X.XX.X.X.X.X...X.XX..XXX.] duration: 3.40359796s trace: https://jaeger.dev.storj.io/trace/74d686768daecf5f FAIL (2025-12-30 11:21:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.4036239s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X.................X..........X...X..........] duration: 22.429462598s trace: https://jaeger.dev.storj.io/trace/7f3400b3387747 FAIL (2025-12-30 11:20:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, Piece ID: OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA) context deadline exceeded; piecestore: (Node ID: 12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, Piece ID: SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ) context deadline exceeded; piecestore: (Node ID: 1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC, Piece ID: 6ZSKVY2RVZAJBTRBKT4RTVQKMJ2BVQDU53I453YBIBBHCZOQCW4A) context deadline exceeded; piecestore: (Node ID: 12YZZnCo54XknuvoBoVtRudXSnoWdNdrn23Lp6Gou3g451BzUwG, Piece ID: 7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ) context deadline exceeded ======LOGS====== s) [noise=true , address=145.79.227.18:4038 , piece_id=K6PIAQ572FP7PVR5G3BRZWSGVYK7XTXG6Z4KBZZ3G5A732BOLS4A, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.892668738 (595h43m54.763751783s) [noise=true , address=145.79.227.40:4079 , piece_id=43LPLVACHLV3VIUR7YZ4RGRCG65FGVSYM5EOPOSG6W6YUSL6OLPQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893082295 (595h43m54.76416534s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893484291 (595h43m54.764567336s) [noise=true , address=77.90.31.12:4613 , piece_id=OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893955759 (595h43m54.765038794s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.894536003 (595h43m54.765619048s) [noise=true , address=145.79.227.44:4226 , piece_id=JZ5CAXU3XK56URASDAYQHQOCON7SCFEECMSM4NJ7O33M647UWAZQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.894858427 (595h43m54.765941462s) [noise=true , address=64.49.13.155:4920 , piece_id=UE53XZKQ5VW5CDBMS7SF3R2FMA3FVNUQA2RLMFMYIIHPDDSNXTZQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.895172897 (595h43m54.766255942s) [noise=true , address=145.79.227.33:4886 , piece_id=GYC5TKM6I5QRBZJCJVRO35ITEGFZWRXEUTA3V76Y65CZ7ZZEDZSQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.896682591 (595h43m54.767765626s) [noise=true , address=145.79.227.29:4953 , piece_id=VTIGSELICMISVDGASJP4W6Z4IOIALQKRPZEULAODHJL4JQHUU7RA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.897870683 (595h43m54.768953728s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.899025873 (595h43m54.770108918s) [noise=true , address=145.79.227.30:4792 , piece_id=T7WHB25ZUMXYNNPCPLQOKLGUPLTQB3XQJ6ZCE3XRXDOYJWJHXFLA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.900208054 (595h43m54.771291099s) [noise=true , address=77.90.31.18:4827 , piece_id=RXOZKEZUKJA4GXKQUZ6ITP2ZW7TPDDXMF7VINT36HGCQNCDCCB2Q, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.901469476 (595h43m54.772552521s) [noise=true , address=77.90.31.15:4706 , piece_id=7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.902861116 (595h43m54.773944161s) [noise=true , address=145.79.227.17:4987 , piece_id=JNZPAZIFJMCQWJELT26MLRIF46UUIYRYD3EKVFX42VIAXYQU4B5Q, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.904040151 (595h43m54.775123196s) [noise=true , address=77.90.31.16:4743 , piece_id=SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.905225688 (595h43m54.776308723s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:56.312272123 (595h43m56.183355158s) [upload=522223]: Next returned nil segment Dec 30 11:12:56.312288104 (595h43m56.183371139s) [upload=522223]: Waiting for error group managing segments... Dec 30 11:12:56.674443063 (595h43m56.545526118s) [noise=true , address=64.49.13.155:4920 , piece_id=UE53XZKQ5VW5CDBMS7SF3R2FMA3FVNUQA2RLMFMYIIHPDDSNXTZQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.748148810 (595h43m56.619231855s) [noise=true , address=145.79.227.17:4987 , piece_id=JNZPAZIFJMCQWJELT26MLRIF46UUIYRYD3EKVFX42VIAXYQU4B5Q, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.748294467 (595h43m56.619377502s) [noise=true , address=145.79.227.18:4038 , piece_id=K6PIAQ572FP7PVR5G3BRZWSGVYK7XTXG6Z4KBZZ3G5A732BOLS4A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.898059835 (595h43m56.76914288s) [noise=true , address=145.79.227.3:4166 , piece_id=GJBVBRV7YZE3GTXEKBW7LJYW35EXK4SDEVIFNZMETDKCZOXCIWKA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.936973813 (595h43m56.808056848s) [noise=true , address=77.90.31.21:4893 , piece_id=6ZSKVY2RVZAJBTRBKT4RTVQKMJ2BVQDU53I453YBIBBHCZOQCW4A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.961395065 (595h43m56.83247811s) [noise=true , address=77.90.31.18:4827 , piece_id=RXOZKEZUKJA4GXKQUZ6ITP2ZW7TPDDXMF7VINT36HGCQNCDCCB2Q, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.967043209 (595h43m56.838126254s) [noise=true , address=77.90.31.15:4706 , piece_id=7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.059782038 (595h43m56.930865073s) [noise=true , address=77.90.31.16:4743 , piece_id=SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.692145558 (595h43m57.563228633s) [noise=true , address=145.79.227.30:4792 , piece_id=T7WHB25ZUMXYNNPCPLQOKLGUPLTQB3XQJ6ZCE3XRXDOYJWJHXFLA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.705427259 (595h43m57.576510304s) [noise=true , address=145.79.227.33:4886 , piece_id=GYC5TKM6I5QRBZJCJVRO35ITEGFZWRXEUTA3V76Y65CZ7ZZEDZSQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.745163051 (595h43m57.616246096s) [noise=true , address=145.79.227.40:4079 , piece_id=43LPLVACHLV3VIUR7YZ4RGRCG65FGVSYM5EOPOSG6W6YUSL6OLPQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.776805663 (595h43m57.647888708s) [noise=true , address=145.79.227.44:4226 , piece_id=JZ5CAXU3XK56URASDAYQHQOCON7SCFEECMSM4NJ7O33M647UWAZQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.859064686 (595h43m57.730147731s) [noise=true , address=145.79.227.29:4953 , piece_id=VTIGSELICMISVDGASJP4W6Z4IOIALQKRPZEULAODHJL4JQHUU7RA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.882701444 (595h43m57.753784479s) [noise=true , address=77.90.31.12:4613 , piece_id=OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.882722825 (595h43m57.75380587s) [seg_pos={0 0}, upload=522223]: Segment reached optimal threshold of 14 pieces. Dec 30 11:12:57.882954816 (595h43m57.754037851s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.882964956 (595h43m57.754048011s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883110403 (595h43m57.754193448s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.883123828 (595h43m57.754206883s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883237274 (595h43m57.754320339s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection Dec 30 11:12:57.883248406 (595h43m57.754331451s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883390406 (595h43m57.754473451s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.883415915 (595h43m57.75449896s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883463475 (595h43m57.75454652s) [seg_pos={0 0}, upload=522223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:12:57.883549108 (595h43m57.754632153s) [upload=522223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:12:57.883554960 (595h43m57.754638005s) [upload=522223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:12:58.175520738 (595h43m58.046603773s) [upload=522223]: Done uploading segments... Dec 30 11:12:58.175538943 (595h43m58.046621978s) [upload=522223]: Upload finished. err: <nil> Duration: 22.429478438s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [........X.......X..XX..X.X.X..........X....X.......X.....X..] duration: 6.464320052s trace: https://jaeger.dev.storj.io/trace/1a9a678f09614d00 FAIL (2025-12-30 11:20:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: RTICAA2XJEZPUX23SBGYDKWUO4BDKG35SMUGQ7FYOJX6B7COVTPA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 5BGCJES4XINC5HTFQ7ZVHKFBS74C3AZNIW77F7AUU273OW25KA3Q) context deadline exceeded ======LOGS====== Dec 30 11:18:36.555685702 (595h49m36.426768737s) [upload=522305]: Starting upload Dec 30 11:18:36.555777035 (595h49m36.42686009s) [upload=522305]: Done starting upload Dec 30 11:18:36.555904328 (595h49m36.426987373s) [upload=522305]: Uploading segments... Dec 30 11:18:36.555997475 (595h49m36.42708052s) [upload=522305]: Got next segment. Inline: false Dec 30 11:18:36.556039665 (595h49m36.4271227s) [upload=522305]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:18:36.556049073 (595h49m36.427132118s) [upload=522305]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:18:36.804348713 (595h49m36.675431748s) [seg_pos={0 0}, upload=522305]: Begin upload segment... Dec 30 11:18:36.804403778 (595h49m36.675486823s) [seg_pos={0 0}, upload=522305]: Done begin upload segment. Dec 30 11:18:36.804466347 (595h49m36.675549382s) [noise=true , address=45.91.116.8:4048 , piece_id=5BGCJES4XINC5HTFQ7ZVHKFBS74C3AZNIW77F7AUU273OW25KA3Q, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.804489642 (595h49m36.675572677s) [noise=true , address=145.79.227.11:4732 , piece_id=WOQ56KBDNBMDGG527CKLE3TYCWXSQ42B7NMQHGAEUTTKKOGWLMWQ, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.804950720 (595h49m36.676033755s) [noise=true , address=145.79.227.6:4257 , piece_id=VZC2OETBOJ5XDCEZVD7Z3UVN2MEIWD62SUJJMVV742DROS7UKVBA, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.805624893 (595h49m36.676707928s) [noise=true , address=145.79.227.40:4049 , piece_id=JO7HTRW6NI46BEYDBLXDJPQ7RFUCJL5SSTDC6U3ZJB2YO3XLIXOA, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.805977484 (595h49m36.677060519s) [noise=true , address=145.79.227.15:4875 , piece_id=AXY25K6VQHIABA3PVCPA6IFYXUPXBTCPPJRGYBRRKN575T4KLSZA, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.806382325 (595h49m36.677465361s) [noise=true , address=145.79.227.5:4222 , piece_id=6HQW2SPGYSY33W5O7L7ED7ZCQ6OX4WW4BFDJ5QZUESJTEV5UMPOQ, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.806714858 (595h49m36.677797893s) [noise=true , address=77.90.31.16:4735 , piece_id=RTICAA2XJEZPUX23SBGYDKWUO4BDKG35SMUGQ7FYOJX6B7COVTPA, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.807051950 (595h49m36.678134985s) [noise=true , address=45.91.116.6:4975 , piece_id=3H2KWCIQITGN4G7BIJMKXTUKUZXDUKR5M4AFGITSHWP4ITCT6H6A, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.807960430 (595h49m36.679043465s) [noise=true , address=145.79.227.19:4077 , piece_id=SRCJAWKMZ2QMKFG5ARUG5KTWZGJAGG2S5C5NAKYDRJ4UMGPBKKRA, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.808094646 (595h49m36.679177681s) [noise=true , address=77.90.31.19:4844 , piece_id=WILZFHIL4QKSO7NLT2N6I4VLOWJNWXNSM3AMXHXXNHBQC3E6LFHA, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.809220710 (595h49m36.680303755s) [noise=true , address=145.79.227.3:4151 , piece_id=O2N6MCM2EJSSFYR57HE4HYGECADOUENNO25ATUUM2AJIMHQZJE2A, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.810618843 (595h49m36.681701878s) [noise=true , address=45.91.116.12:4226 , piece_id=UYD2I5Z52P5KMZLYMUYOQS6OHFMG34JEJA2BHR6DETUWH6467FDQ, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.812039998 (595h49m36.683123043s) [noise=true , address=145.79.227.47:4266 , piece_id=SE2TB4XQXKZEPU2PSRXTVX6BHHBCSKYJ6A4PAHY2OIU4KFQW2WJA, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.818186933 (595h49m36.689269978s) [noise=true , address=45.91.116.9:4263 , piece_id=CFILRAG744ZFMK7KIRMC2DTIV5JDXA67QC3G6WLOB2AYDQSBYJ3Q, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.820605169 (595h49m36.691688204s) [noise=true , address=145.79.227.9:4365 , piece_id=G6PII4JBEHQUW4JVJKJNR3ZXF6FY5CTAS5HIGIDKJ3KTZJ2WMF6A, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.821581848 (595h49m36.692664883s) [noise=true , address=145.79.227.32:4835 , piece_id=J4OA7BZS2JHHD4VYC65UDVQ64THPLW7C7DGM427KPXANYMXEMHEQ, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:38.526165881 (595h49m38.397248917s) [upload=522305]: Next returned nil segment Dec 30 11:18:38.526181822 (595h49m38.397264867s) [upload=522305]: Waiting for error group managing segments... Dec 30 11:18:38.806812517 (595h49m38.677895562s) [noise=true , address=145.79.227.5:4222 , piece_id=6HQW2SPGYSY33W5O7L7ED7ZCQ6OX4WW4BFDJ5QZUESJTEV5UMPOQ, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.808610099 (595h49m38.679693134s) [noise=true , address=145.79.227.19:4077 , piece_id=SRCJAWKMZ2QMKFG5ARUG5KTWZGJAGG2S5C5NAKYDRJ4UMGPBKKRA, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.810040463 (595h49m38.681123508s) [noise=true , address=145.79.227.15:4875 , piece_id=AXY25K6VQHIABA3PVCPA6IFYXUPXBTCPPJRGYBRRKN575T4KLSZA, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.812714575 (595h49m38.68379762s) [noise=true , address=145.79.227.47:4266 , piece_id=SE2TB4XQXKZEPU2PSRXTVX6BHHBCSKYJ6A4PAHY2OIU4KFQW2WJA, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.814668165 (595h49m38.68575121s) [noise=true , address=145.79.227.6:4257 , piece_id=VZC2OETBOJ5XDCEZVD7Z3UVN2MEIWD62SUJJMVV742DROS7UKVBA, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.815888208 (595h49m38.686971253s) [noise=true , address=145.79.227.11:4732 , piece_id=WOQ56KBDNBMDGG527CKLE3TYCWXSQ42B7NMQHGAEUTTKKOGWLMWQ, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.829989509 (595h49m38.701072554s) [noise=true , address=145.79.227.9:4365 , piece_id=G6PII4JBEHQUW4JVJKJNR3ZXF6FY5CTAS5HIGIDKJ3KTZJ2WMF6A, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.831214852 (595h49m38.702297897s) [noise=true , address=145.79.227.3:4151 , piece_id=O2N6MCM2EJSSFYR57HE4HYGECADOUENNO25ATUUM2AJIMHQZJE2A, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.843511968 (595h49m38.714595013s) [noise=true , address=145.79.227.32:4835 , piece_id=J4OA7BZS2JHHD4VYC65UDVQ64THPLW7C7DGM427KPXANYMXEMHEQ, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:39.240376560 (595h49m39.111459595s) [noise=true , address=45.91.116.8:4048 , piece_id=5BGCJES4XINC5HTFQ7ZVHKFBS74C3AZNIW77F7AUU273OW25KA3Q, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:39.272617302 (595h49m39.143700337s) [noise=true , address=45.91.116.12:4226 , piece_id=UYD2I5Z52P5KMZLYMUYOQS6OHFMG34JEJA2BHR6DETUWH6467FDQ, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:39.407566657 (595h49m39.278649692s) [noise=true , address=77.90.31.19:4844 , piece_id=WILZFHIL4QKSO7NLT2N6I4VLOWJNWXNSM3AMXHXXNHBQC3E6LFHA, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:40.033396193 (595h49m39.904479228s) [noise=true , address=77.90.31.16:4735 , piece_id=RTICAA2XJEZPUX23SBGYDKWUO4BDKG35SMUGQ7FYOJX6B7COVTPA, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:40.033419418 (595h49m39.904502463s) [seg_pos={0 0}, upload=522305]: Segment reached optimal threshold of 13 pieces. Dec 30 11:18:40.033736962 (595h49m39.904820008s) [noise=true , address=45.91.116.6:4975 , piece_id=3H2KWCIQITGN4G7BIJMKXTUKUZXDUKR5M4AFGITSHWP4ITCT6H6A, seg_pos={0 0}, upload=522305]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:55024->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 11:18:40.033744296 (595h49m39.904827331s) [noise=true , address=45.91.116.6:4975 , piece_id=3H2KWCIQITGN4G7BIJMKXTUKUZXDUKR5M4AFGITSHWP4ITCT6H6A, seg_pos={0 0}, upload=522305]: Long tail cancellation due to optimal threshold Dec 30 11:18:40.033788580 (595h49m39.904871615s) [noise=true , address=145.79.227.40:4049 , piece_id=JO7HTRW6NI46BEYDBLXDJPQ7RFUCJL5SSTDC6U3ZJB2YO3XLIXOA, seg_pos={0 0}, upload=522305]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 11:18:40.033850739 (595h49m39.904933804s) [noise=true , address=145.79.227.40:4049 , piece_id=JO7HTRW6NI46BEYDBLXDJPQ7RFUCJL5SSTDC6U3ZJB2YO3XLIXOA, seg_pos={0 0}, upload=522305]: Long tail cancellation due to optimal threshold Dec 30 11:18:40.033972892 (595h49m39.905055927s) [noise=true , address=45.91.116.9:4263 , piece_id=CFILRAG744ZFMK7KIRMC2DTIV5JDXA67QC3G6WLOB2AYDQSBYJ3Q, seg_pos={0 0}, upload=522305]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 30 11:18:40.033981037 (595h49m39.905064082s) [noise=true , address=45.91.116.9:4263 , piece_id=CFILRAG744ZFMK7KIRMC2DTIV5JDXA67QC3G6WLOB2AYDQSBYJ3Q, seg_pos={0 0}, upload=522305]: Long tail cancellation due to optimal threshold Dec 30 11:18:40.034038827 (595h49m39.905121862s) [seg_pos={0 0}, upload=522305]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:18:40.034123188 (595h49m39.905206233s) [upload=522305]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:18:40.034128458 (595h49m39.905211493s) [upload=522305]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:18:40.334300412 (595h49m40.205383447s) [upload=522305]: Done uploading segments... Dec 30 11:18:40.334323186 (595h49m40.205406221s) [upload=522305]: Upload finished. err: <nil> Duration: 6.464348878s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X.................X..........X...X.........] duration: 44.899975089s trace: https://jaeger.dev.storj.io/trace/7f3400b3387747 FAIL (2025-12-30 11:20:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, Piece ID: OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA) context deadline exceeded; piecestore: (Node ID: 12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, Piece ID: SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ) context deadline exceeded; piecestore: (Node ID: 1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC, Piece ID: 6ZSKVY2RVZAJBTRBKT4RTVQKMJ2BVQDU53I453YBIBBHCZOQCW4A) context deadline exceeded; piecestore: (Node ID: 12YZZnCo54XknuvoBoVtRudXSnoWdNdrn23Lp6Gou3g451BzUwG, Piece ID: 7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ) context deadline exceeded ======LOGS====== s) [noise=true , address=145.79.227.18:4038 , piece_id=K6PIAQ572FP7PVR5G3BRZWSGVYK7XTXG6Z4KBZZ3G5A732BOLS4A, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.892668738 (595h43m54.763751783s) [noise=true , address=145.79.227.40:4079 , piece_id=43LPLVACHLV3VIUR7YZ4RGRCG65FGVSYM5EOPOSG6W6YUSL6OLPQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893082295 (595h43m54.76416534s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893484291 (595h43m54.764567336s) [noise=true , address=77.90.31.12:4613 , piece_id=OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893955759 (595h43m54.765038794s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.894536003 (595h43m54.765619048s) [noise=true , address=145.79.227.44:4226 , piece_id=JZ5CAXU3XK56URASDAYQHQOCON7SCFEECMSM4NJ7O33M647UWAZQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.894858427 (595h43m54.765941462s) [noise=true , address=64.49.13.155:4920 , piece_id=UE53XZKQ5VW5CDBMS7SF3R2FMA3FVNUQA2RLMFMYIIHPDDSNXTZQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.895172897 (595h43m54.766255942s) [noise=true , address=145.79.227.33:4886 , piece_id=GYC5TKM6I5QRBZJCJVRO35ITEGFZWRXEUTA3V76Y65CZ7ZZEDZSQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.896682591 (595h43m54.767765626s) [noise=true , address=145.79.227.29:4953 , piece_id=VTIGSELICMISVDGASJP4W6Z4IOIALQKRPZEULAODHJL4JQHUU7RA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.897870683 (595h43m54.768953728s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.899025873 (595h43m54.770108918s) [noise=true , address=145.79.227.30:4792 , piece_id=T7WHB25ZUMXYNNPCPLQOKLGUPLTQB3XQJ6ZCE3XRXDOYJWJHXFLA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.900208054 (595h43m54.771291099s) [noise=true , address=77.90.31.18:4827 , piece_id=RXOZKEZUKJA4GXKQUZ6ITP2ZW7TPDDXMF7VINT36HGCQNCDCCB2Q, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.901469476 (595h43m54.772552521s) [noise=true , address=77.90.31.15:4706 , piece_id=7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.902861116 (595h43m54.773944161s) [noise=true , address=145.79.227.17:4987 , piece_id=JNZPAZIFJMCQWJELT26MLRIF46UUIYRYD3EKVFX42VIAXYQU4B5Q, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.904040151 (595h43m54.775123196s) [noise=true , address=77.90.31.16:4743 , piece_id=SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.905225688 (595h43m54.776308723s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:56.312272123 (595h43m56.183355158s) [upload=522223]: Next returned nil segment Dec 30 11:12:56.312288104 (595h43m56.183371139s) [upload=522223]: Waiting for error group managing segments... Dec 30 11:12:56.674443063 (595h43m56.545526118s) [noise=true , address=64.49.13.155:4920 , piece_id=UE53XZKQ5VW5CDBMS7SF3R2FMA3FVNUQA2RLMFMYIIHPDDSNXTZQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.748148810 (595h43m56.619231855s) [noise=true , address=145.79.227.17:4987 , piece_id=JNZPAZIFJMCQWJELT26MLRIF46UUIYRYD3EKVFX42VIAXYQU4B5Q, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.748294467 (595h43m56.619377502s) [noise=true , address=145.79.227.18:4038 , piece_id=K6PIAQ572FP7PVR5G3BRZWSGVYK7XTXG6Z4KBZZ3G5A732BOLS4A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.898059835 (595h43m56.76914288s) [noise=true , address=145.79.227.3:4166 , piece_id=GJBVBRV7YZE3GTXEKBW7LJYW35EXK4SDEVIFNZMETDKCZOXCIWKA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.936973813 (595h43m56.808056848s) [noise=true , address=77.90.31.21:4893 , piece_id=6ZSKVY2RVZAJBTRBKT4RTVQKMJ2BVQDU53I453YBIBBHCZOQCW4A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.961395065 (595h43m56.83247811s) [noise=true , address=77.90.31.18:4827 , piece_id=RXOZKEZUKJA4GXKQUZ6ITP2ZW7TPDDXMF7VINT36HGCQNCDCCB2Q, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.967043209 (595h43m56.838126254s) [noise=true , address=77.90.31.15:4706 , piece_id=7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.059782038 (595h43m56.930865073s) [noise=true , address=77.90.31.16:4743 , piece_id=SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.692145558 (595h43m57.563228633s) [noise=true , address=145.79.227.30:4792 , piece_id=T7WHB25ZUMXYNNPCPLQOKLGUPLTQB3XQJ6ZCE3XRXDOYJWJHXFLA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.705427259 (595h43m57.576510304s) [noise=true , address=145.79.227.33:4886 , piece_id=GYC5TKM6I5QRBZJCJVRO35ITEGFZWRXEUTA3V76Y65CZ7ZZEDZSQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.745163051 (595h43m57.616246096s) [noise=true , address=145.79.227.40:4079 , piece_id=43LPLVACHLV3VIUR7YZ4RGRCG65FGVSYM5EOPOSG6W6YUSL6OLPQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.776805663 (595h43m57.647888708s) [noise=true , address=145.79.227.44:4226 , piece_id=JZ5CAXU3XK56URASDAYQHQOCON7SCFEECMSM4NJ7O33M647UWAZQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.859064686 (595h43m57.730147731s) [noise=true , address=145.79.227.29:4953 , piece_id=VTIGSELICMISVDGASJP4W6Z4IOIALQKRPZEULAODHJL4JQHUU7RA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.882701444 (595h43m57.753784479s) [noise=true , address=77.90.31.12:4613 , piece_id=OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.882722825 (595h43m57.75380587s) [seg_pos={0 0}, upload=522223]: Segment reached optimal threshold of 14 pieces. Dec 30 11:12:57.882954816 (595h43m57.754037851s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.882964956 (595h43m57.754048011s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883110403 (595h43m57.754193448s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.883123828 (595h43m57.754206883s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883237274 (595h43m57.754320339s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection Dec 30 11:12:57.883248406 (595h43m57.754331451s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883390406 (595h43m57.754473451s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.883415915 (595h43m57.75449896s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883463475 (595h43m57.75454652s) [seg_pos={0 0}, upload=522223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:12:57.883549108 (595h43m57.754632153s) [upload=522223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:12:57.883554960 (595h43m57.754638005s) [upload=522223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:12:58.175520738 (595h43m58.046603773s) [upload=522223]: Done uploading segments... Dec 30 11:12:58.175538943 (595h43m58.046621978s) [upload=522223]: Upload finished. err: <nil> Duration: 44.899995257s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [X.X.XXXX.X.XX.X.X.....XX.X........X.X.XX.X.X.X.X...X.XX..XXX] duration: 5.000796026s trace: https://jaeger.dev.storj.io/trace/74d686768daecf5f FAIL (2025-12-30 11:19:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000817296s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X........X.......X..XX..X.X.X..........X....X.......X.....X.] duration: 6.656886811s trace: https://jaeger.dev.storj.io/trace/1a9a678f09614d00 FAIL (2025-12-30 11:19:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: RTICAA2XJEZPUX23SBGYDKWUO4BDKG35SMUGQ7FYOJX6B7COVTPA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 5BGCJES4XINC5HTFQ7ZVHKFBS74C3AZNIW77F7AUU273OW25KA3Q) context deadline exceeded ======LOGS====== Dec 30 11:18:36.555685702 (595h49m36.426768737s) [upload=522305]: Starting upload Dec 30 11:18:36.555777035 (595h49m36.42686009s) [upload=522305]: Done starting upload Dec 30 11:18:36.555904328 (595h49m36.426987373s) [upload=522305]: Uploading segments... Dec 30 11:18:36.555997475 (595h49m36.42708052s) [upload=522305]: Got next segment. Inline: false Dec 30 11:18:36.556039665 (595h49m36.4271227s) [upload=522305]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:18:36.556049073 (595h49m36.427132118s) [upload=522305]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:18:36.804348713 (595h49m36.675431748s) [seg_pos={0 0}, upload=522305]: Begin upload segment... Dec 30 11:18:36.804403778 (595h49m36.675486823s) [seg_pos={0 0}, upload=522305]: Done begin upload segment. Dec 30 11:18:36.804466347 (595h49m36.675549382s) [noise=true , address=45.91.116.8:4048 , piece_id=5BGCJES4XINC5HTFQ7ZVHKFBS74C3AZNIW77F7AUU273OW25KA3Q, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.804489642 (595h49m36.675572677s) [noise=true , address=145.79.227.11:4732 , piece_id=WOQ56KBDNBMDGG527CKLE3TYCWXSQ42B7NMQHGAEUTTKKOGWLMWQ, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.804950720 (595h49m36.676033755s) [noise=true , address=145.79.227.6:4257 , piece_id=VZC2OETBOJ5XDCEZVD7Z3UVN2MEIWD62SUJJMVV742DROS7UKVBA, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.805624893 (595h49m36.676707928s) [noise=true , address=145.79.227.40:4049 , piece_id=JO7HTRW6NI46BEYDBLXDJPQ7RFUCJL5SSTDC6U3ZJB2YO3XLIXOA, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.805977484 (595h49m36.677060519s) [noise=true , address=145.79.227.15:4875 , piece_id=AXY25K6VQHIABA3PVCPA6IFYXUPXBTCPPJRGYBRRKN575T4KLSZA, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.806382325 (595h49m36.677465361s) [noise=true , address=145.79.227.5:4222 , piece_id=6HQW2SPGYSY33W5O7L7ED7ZCQ6OX4WW4BFDJ5QZUESJTEV5UMPOQ, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.806714858 (595h49m36.677797893s) [noise=true , address=77.90.31.16:4735 , piece_id=RTICAA2XJEZPUX23SBGYDKWUO4BDKG35SMUGQ7FYOJX6B7COVTPA, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.807051950 (595h49m36.678134985s) [noise=true , address=45.91.116.6:4975 , piece_id=3H2KWCIQITGN4G7BIJMKXTUKUZXDUKR5M4AFGITSHWP4ITCT6H6A, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.807960430 (595h49m36.679043465s) [noise=true , address=145.79.227.19:4077 , piece_id=SRCJAWKMZ2QMKFG5ARUG5KTWZGJAGG2S5C5NAKYDRJ4UMGPBKKRA, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.808094646 (595h49m36.679177681s) [noise=true , address=77.90.31.19:4844 , piece_id=WILZFHIL4QKSO7NLT2N6I4VLOWJNWXNSM3AMXHXXNHBQC3E6LFHA, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.809220710 (595h49m36.680303755s) [noise=true , address=145.79.227.3:4151 , piece_id=O2N6MCM2EJSSFYR57HE4HYGECADOUENNO25ATUUM2AJIMHQZJE2A, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.810618843 (595h49m36.681701878s) [noise=true , address=45.91.116.12:4226 , piece_id=UYD2I5Z52P5KMZLYMUYOQS6OHFMG34JEJA2BHR6DETUWH6467FDQ, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.812039998 (595h49m36.683123043s) [noise=true , address=145.79.227.47:4266 , piece_id=SE2TB4XQXKZEPU2PSRXTVX6BHHBCSKYJ6A4PAHY2OIU4KFQW2WJA, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.818186933 (595h49m36.689269978s) [noise=true , address=45.91.116.9:4263 , piece_id=CFILRAG744ZFMK7KIRMC2DTIV5JDXA67QC3G6WLOB2AYDQSBYJ3Q, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.820605169 (595h49m36.691688204s) [noise=true , address=145.79.227.9:4365 , piece_id=G6PII4JBEHQUW4JVJKJNR3ZXF6FY5CTAS5HIGIDKJ3KTZJ2WMF6A, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.821581848 (595h49m36.692664883s) [noise=true , address=145.79.227.32:4835 , piece_id=J4OA7BZS2JHHD4VYC65UDVQ64THPLW7C7DGM427KPXANYMXEMHEQ, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:38.526165881 (595h49m38.397248917s) [upload=522305]: Next returned nil segment Dec 30 11:18:38.526181822 (595h49m38.397264867s) [upload=522305]: Waiting for error group managing segments... Dec 30 11:18:38.806812517 (595h49m38.677895562s) [noise=true , address=145.79.227.5:4222 , piece_id=6HQW2SPGYSY33W5O7L7ED7ZCQ6OX4WW4BFDJ5QZUESJTEV5UMPOQ, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.808610099 (595h49m38.679693134s) [noise=true , address=145.79.227.19:4077 , piece_id=SRCJAWKMZ2QMKFG5ARUG5KTWZGJAGG2S5C5NAKYDRJ4UMGPBKKRA, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.810040463 (595h49m38.681123508s) [noise=true , address=145.79.227.15:4875 , piece_id=AXY25K6VQHIABA3PVCPA6IFYXUPXBTCPPJRGYBRRKN575T4KLSZA, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.812714575 (595h49m38.68379762s) [noise=true , address=145.79.227.47:4266 , piece_id=SE2TB4XQXKZEPU2PSRXTVX6BHHBCSKYJ6A4PAHY2OIU4KFQW2WJA, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.814668165 (595h49m38.68575121s) [noise=true , address=145.79.227.6:4257 , piece_id=VZC2OETBOJ5XDCEZVD7Z3UVN2MEIWD62SUJJMVV742DROS7UKVBA, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.815888208 (595h49m38.686971253s) [noise=true , address=145.79.227.11:4732 , piece_id=WOQ56KBDNBMDGG527CKLE3TYCWXSQ42B7NMQHGAEUTTKKOGWLMWQ, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.829989509 (595h49m38.701072554s) [noise=true , address=145.79.227.9:4365 , piece_id=G6PII4JBEHQUW4JVJKJNR3ZXF6FY5CTAS5HIGIDKJ3KTZJ2WMF6A, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.831214852 (595h49m38.702297897s) [noise=true , address=145.79.227.3:4151 , piece_id=O2N6MCM2EJSSFYR57HE4HYGECADOUENNO25ATUUM2AJIMHQZJE2A, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.843511968 (595h49m38.714595013s) [noise=true , address=145.79.227.32:4835 , piece_id=J4OA7BZS2JHHD4VYC65UDVQ64THPLW7C7DGM427KPXANYMXEMHEQ, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:39.240376560 (595h49m39.111459595s) [noise=true , address=45.91.116.8:4048 , piece_id=5BGCJES4XINC5HTFQ7ZVHKFBS74C3AZNIW77F7AUU273OW25KA3Q, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:39.272617302 (595h49m39.143700337s) [noise=true , address=45.91.116.12:4226 , piece_id=UYD2I5Z52P5KMZLYMUYOQS6OHFMG34JEJA2BHR6DETUWH6467FDQ, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:39.407566657 (595h49m39.278649692s) [noise=true , address=77.90.31.19:4844 , piece_id=WILZFHIL4QKSO7NLT2N6I4VLOWJNWXNSM3AMXHXXNHBQC3E6LFHA, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:40.033396193 (595h49m39.904479228s) [noise=true , address=77.90.31.16:4735 , piece_id=RTICAA2XJEZPUX23SBGYDKWUO4BDKG35SMUGQ7FYOJX6B7COVTPA, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:40.033419418 (595h49m39.904502463s) [seg_pos={0 0}, upload=522305]: Segment reached optimal threshold of 13 pieces. Dec 30 11:18:40.033736962 (595h49m39.904820008s) [noise=true , address=45.91.116.6:4975 , piece_id=3H2KWCIQITGN4G7BIJMKXTUKUZXDUKR5M4AFGITSHWP4ITCT6H6A, seg_pos={0 0}, upload=522305]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:55024->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 11:18:40.033744296 (595h49m39.904827331s) [noise=true , address=45.91.116.6:4975 , piece_id=3H2KWCIQITGN4G7BIJMKXTUKUZXDUKR5M4AFGITSHWP4ITCT6H6A, seg_pos={0 0}, upload=522305]: Long tail cancellation due to optimal threshold Dec 30 11:18:40.033788580 (595h49m39.904871615s) [noise=true , address=145.79.227.40:4049 , piece_id=JO7HTRW6NI46BEYDBLXDJPQ7RFUCJL5SSTDC6U3ZJB2YO3XLIXOA, seg_pos={0 0}, upload=522305]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 11:18:40.033850739 (595h49m39.904933804s) [noise=true , address=145.79.227.40:4049 , piece_id=JO7HTRW6NI46BEYDBLXDJPQ7RFUCJL5SSTDC6U3ZJB2YO3XLIXOA, seg_pos={0 0}, upload=522305]: Long tail cancellation due to optimal threshold Dec 30 11:18:40.033972892 (595h49m39.905055927s) [noise=true , address=45.91.116.9:4263 , piece_id=CFILRAG744ZFMK7KIRMC2DTIV5JDXA67QC3G6WLOB2AYDQSBYJ3Q, seg_pos={0 0}, upload=522305]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 30 11:18:40.033981037 (595h49m39.905064082s) [noise=true , address=45.91.116.9:4263 , piece_id=CFILRAG744ZFMK7KIRMC2DTIV5JDXA67QC3G6WLOB2AYDQSBYJ3Q, seg_pos={0 0}, upload=522305]: Long tail cancellation due to optimal threshold Dec 30 11:18:40.034038827 (595h49m39.905121862s) [seg_pos={0 0}, upload=522305]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:18:40.034123188 (595h49m39.905206233s) [upload=522305]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:18:40.034128458 (595h49m39.905211493s) [upload=522305]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:18:40.334300412 (595h49m40.205383447s) [upload=522305]: Done uploading segments... Dec 30 11:18:40.334323186 (595h49m40.205406221s) [upload=522305]: Upload finished. err: <nil> Duration: 6.656909083s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X.................X..........X...X........] duration: 33.274563374s trace: https://jaeger.dev.storj.io/trace/7f3400b3387747 FAIL (2025-12-30 11:19:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, Piece ID: OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA) context deadline exceeded; piecestore: (Node ID: 12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, Piece ID: SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ) context deadline exceeded; piecestore: (Node ID: 1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC, Piece ID: 6ZSKVY2RVZAJBTRBKT4RTVQKMJ2BVQDU53I453YBIBBHCZOQCW4A) context deadline exceeded; piecestore: (Node ID: 12YZZnCo54XknuvoBoVtRudXSnoWdNdrn23Lp6Gou3g451BzUwG, Piece ID: 7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ) context deadline exceeded ======LOGS====== s) [noise=true , address=145.79.227.18:4038 , piece_id=K6PIAQ572FP7PVR5G3BRZWSGVYK7XTXG6Z4KBZZ3G5A732BOLS4A, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.892668738 (595h43m54.763751783s) [noise=true , address=145.79.227.40:4079 , piece_id=43LPLVACHLV3VIUR7YZ4RGRCG65FGVSYM5EOPOSG6W6YUSL6OLPQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893082295 (595h43m54.76416534s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893484291 (595h43m54.764567336s) [noise=true , address=77.90.31.12:4613 , piece_id=OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893955759 (595h43m54.765038794s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.894536003 (595h43m54.765619048s) [noise=true , address=145.79.227.44:4226 , piece_id=JZ5CAXU3XK56URASDAYQHQOCON7SCFEECMSM4NJ7O33M647UWAZQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.894858427 (595h43m54.765941462s) [noise=true , address=64.49.13.155:4920 , piece_id=UE53XZKQ5VW5CDBMS7SF3R2FMA3FVNUQA2RLMFMYIIHPDDSNXTZQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.895172897 (595h43m54.766255942s) [noise=true , address=145.79.227.33:4886 , piece_id=GYC5TKM6I5QRBZJCJVRO35ITEGFZWRXEUTA3V76Y65CZ7ZZEDZSQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.896682591 (595h43m54.767765626s) [noise=true , address=145.79.227.29:4953 , piece_id=VTIGSELICMISVDGASJP4W6Z4IOIALQKRPZEULAODHJL4JQHUU7RA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.897870683 (595h43m54.768953728s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.899025873 (595h43m54.770108918s) [noise=true , address=145.79.227.30:4792 , piece_id=T7WHB25ZUMXYNNPCPLQOKLGUPLTQB3XQJ6ZCE3XRXDOYJWJHXFLA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.900208054 (595h43m54.771291099s) [noise=true , address=77.90.31.18:4827 , piece_id=RXOZKEZUKJA4GXKQUZ6ITP2ZW7TPDDXMF7VINT36HGCQNCDCCB2Q, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.901469476 (595h43m54.772552521s) [noise=true , address=77.90.31.15:4706 , piece_id=7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.902861116 (595h43m54.773944161s) [noise=true , address=145.79.227.17:4987 , piece_id=JNZPAZIFJMCQWJELT26MLRIF46UUIYRYD3EKVFX42VIAXYQU4B5Q, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.904040151 (595h43m54.775123196s) [noise=true , address=77.90.31.16:4743 , piece_id=SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.905225688 (595h43m54.776308723s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:56.312272123 (595h43m56.183355158s) [upload=522223]: Next returned nil segment Dec 30 11:12:56.312288104 (595h43m56.183371139s) [upload=522223]: Waiting for error group managing segments... Dec 30 11:12:56.674443063 (595h43m56.545526118s) [noise=true , address=64.49.13.155:4920 , piece_id=UE53XZKQ5VW5CDBMS7SF3R2FMA3FVNUQA2RLMFMYIIHPDDSNXTZQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.748148810 (595h43m56.619231855s) [noise=true , address=145.79.227.17:4987 , piece_id=JNZPAZIFJMCQWJELT26MLRIF46UUIYRYD3EKVFX42VIAXYQU4B5Q, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.748294467 (595h43m56.619377502s) [noise=true , address=145.79.227.18:4038 , piece_id=K6PIAQ572FP7PVR5G3BRZWSGVYK7XTXG6Z4KBZZ3G5A732BOLS4A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.898059835 (595h43m56.76914288s) [noise=true , address=145.79.227.3:4166 , piece_id=GJBVBRV7YZE3GTXEKBW7LJYW35EXK4SDEVIFNZMETDKCZOXCIWKA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.936973813 (595h43m56.808056848s) [noise=true , address=77.90.31.21:4893 , piece_id=6ZSKVY2RVZAJBTRBKT4RTVQKMJ2BVQDU53I453YBIBBHCZOQCW4A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.961395065 (595h43m56.83247811s) [noise=true , address=77.90.31.18:4827 , piece_id=RXOZKEZUKJA4GXKQUZ6ITP2ZW7TPDDXMF7VINT36HGCQNCDCCB2Q, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.967043209 (595h43m56.838126254s) [noise=true , address=77.90.31.15:4706 , piece_id=7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.059782038 (595h43m56.930865073s) [noise=true , address=77.90.31.16:4743 , piece_id=SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.692145558 (595h43m57.563228633s) [noise=true , address=145.79.227.30:4792 , piece_id=T7WHB25ZUMXYNNPCPLQOKLGUPLTQB3XQJ6ZCE3XRXDOYJWJHXFLA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.705427259 (595h43m57.576510304s) [noise=true , address=145.79.227.33:4886 , piece_id=GYC5TKM6I5QRBZJCJVRO35ITEGFZWRXEUTA3V76Y65CZ7ZZEDZSQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.745163051 (595h43m57.616246096s) [noise=true , address=145.79.227.40:4079 , piece_id=43LPLVACHLV3VIUR7YZ4RGRCG65FGVSYM5EOPOSG6W6YUSL6OLPQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.776805663 (595h43m57.647888708s) [noise=true , address=145.79.227.44:4226 , piece_id=JZ5CAXU3XK56URASDAYQHQOCON7SCFEECMSM4NJ7O33M647UWAZQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.859064686 (595h43m57.730147731s) [noise=true , address=145.79.227.29:4953 , piece_id=VTIGSELICMISVDGASJP4W6Z4IOIALQKRPZEULAODHJL4JQHUU7RA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.882701444 (595h43m57.753784479s) [noise=true , address=77.90.31.12:4613 , piece_id=OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.882722825 (595h43m57.75380587s) [seg_pos={0 0}, upload=522223]: Segment reached optimal threshold of 14 pieces. Dec 30 11:12:57.882954816 (595h43m57.754037851s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.882964956 (595h43m57.754048011s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883110403 (595h43m57.754193448s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.883123828 (595h43m57.754206883s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883237274 (595h43m57.754320339s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection Dec 30 11:12:57.883248406 (595h43m57.754331451s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883390406 (595h43m57.754473451s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.883415915 (595h43m57.75449896s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883463475 (595h43m57.75454652s) [seg_pos={0 0}, upload=522223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:12:57.883549108 (595h43m57.754632153s) [upload=522223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:12:57.883554960 (595h43m57.754638005s) [upload=522223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:12:58.175520738 (595h43m58.046603773s) [upload=522223]: Done uploading segments... Dec 30 11:12:58.175538943 (595h43m58.046621978s) [upload=522223]: Upload finished. err: <nil> Duration: 33.274578071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XX........X.......X..XX..X.X.X..........X....X.......X.....X] duration: 30.005941047s trace: https://jaeger.dev.storj.io/trace/1a9a678f09614d00 FAIL (2025-12-30 11:19:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: RTICAA2XJEZPUX23SBGYDKWUO4BDKG35SMUGQ7FYOJX6B7COVTPA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 5BGCJES4XINC5HTFQ7ZVHKFBS74C3AZNIW77F7AUU273OW25KA3Q) context deadline exceeded ======LOGS====== Dec 30 11:18:36.555685702 (595h49m36.426768737s) [upload=522305]: Starting upload Dec 30 11:18:36.555777035 (595h49m36.42686009s) [upload=522305]: Done starting upload Dec 30 11:18:36.555904328 (595h49m36.426987373s) [upload=522305]: Uploading segments... Dec 30 11:18:36.555997475 (595h49m36.42708052s) [upload=522305]: Got next segment. Inline: false Dec 30 11:18:36.556039665 (595h49m36.4271227s) [upload=522305]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:18:36.556049073 (595h49m36.427132118s) [upload=522305]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:18:36.804348713 (595h49m36.675431748s) [seg_pos={0 0}, upload=522305]: Begin upload segment... Dec 30 11:18:36.804403778 (595h49m36.675486823s) [seg_pos={0 0}, upload=522305]: Done begin upload segment. Dec 30 11:18:36.804466347 (595h49m36.675549382s) [noise=true , address=45.91.116.8:4048 , piece_id=5BGCJES4XINC5HTFQ7ZVHKFBS74C3AZNIW77F7AUU273OW25KA3Q, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.804489642 (595h49m36.675572677s) [noise=true , address=145.79.227.11:4732 , piece_id=WOQ56KBDNBMDGG527CKLE3TYCWXSQ42B7NMQHGAEUTTKKOGWLMWQ, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.804950720 (595h49m36.676033755s) [noise=true , address=145.79.227.6:4257 , piece_id=VZC2OETBOJ5XDCEZVD7Z3UVN2MEIWD62SUJJMVV742DROS7UKVBA, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.805624893 (595h49m36.676707928s) [noise=true , address=145.79.227.40:4049 , piece_id=JO7HTRW6NI46BEYDBLXDJPQ7RFUCJL5SSTDC6U3ZJB2YO3XLIXOA, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.805977484 (595h49m36.677060519s) [noise=true , address=145.79.227.15:4875 , piece_id=AXY25K6VQHIABA3PVCPA6IFYXUPXBTCPPJRGYBRRKN575T4KLSZA, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.806382325 (595h49m36.677465361s) [noise=true , address=145.79.227.5:4222 , piece_id=6HQW2SPGYSY33W5O7L7ED7ZCQ6OX4WW4BFDJ5QZUESJTEV5UMPOQ, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.806714858 (595h49m36.677797893s) [noise=true , address=77.90.31.16:4735 , piece_id=RTICAA2XJEZPUX23SBGYDKWUO4BDKG35SMUGQ7FYOJX6B7COVTPA, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.807051950 (595h49m36.678134985s) [noise=true , address=45.91.116.6:4975 , piece_id=3H2KWCIQITGN4G7BIJMKXTUKUZXDUKR5M4AFGITSHWP4ITCT6H6A, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.807960430 (595h49m36.679043465s) [noise=true , address=145.79.227.19:4077 , piece_id=SRCJAWKMZ2QMKFG5ARUG5KTWZGJAGG2S5C5NAKYDRJ4UMGPBKKRA, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.808094646 (595h49m36.679177681s) [noise=true , address=77.90.31.19:4844 , piece_id=WILZFHIL4QKSO7NLT2N6I4VLOWJNWXNSM3AMXHXXNHBQC3E6LFHA, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.809220710 (595h49m36.680303755s) [noise=true , address=145.79.227.3:4151 , piece_id=O2N6MCM2EJSSFYR57HE4HYGECADOUENNO25ATUUM2AJIMHQZJE2A, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.810618843 (595h49m36.681701878s) [noise=true , address=45.91.116.12:4226 , piece_id=UYD2I5Z52P5KMZLYMUYOQS6OHFMG34JEJA2BHR6DETUWH6467FDQ, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.812039998 (595h49m36.683123043s) [noise=true , address=145.79.227.47:4266 , piece_id=SE2TB4XQXKZEPU2PSRXTVX6BHHBCSKYJ6A4PAHY2OIU4KFQW2WJA, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.818186933 (595h49m36.689269978s) [noise=true , address=45.91.116.9:4263 , piece_id=CFILRAG744ZFMK7KIRMC2DTIV5JDXA67QC3G6WLOB2AYDQSBYJ3Q, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.820605169 (595h49m36.691688204s) [noise=true , address=145.79.227.9:4365 , piece_id=G6PII4JBEHQUW4JVJKJNR3ZXF6FY5CTAS5HIGIDKJ3KTZJ2WMF6A, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:36.821581848 (595h49m36.692664883s) [noise=true , address=145.79.227.32:4835 , piece_id=J4OA7BZS2JHHD4VYC65UDVQ64THPLW7C7DGM427KPXANYMXEMHEQ, seg_pos={0 0}, upload=522305]: Uploading piece... Dec 30 11:18:38.526165881 (595h49m38.397248917s) [upload=522305]: Next returned nil segment Dec 30 11:18:38.526181822 (595h49m38.397264867s) [upload=522305]: Waiting for error group managing segments... Dec 30 11:18:38.806812517 (595h49m38.677895562s) [noise=true , address=145.79.227.5:4222 , piece_id=6HQW2SPGYSY33W5O7L7ED7ZCQ6OX4WW4BFDJ5QZUESJTEV5UMPOQ, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.808610099 (595h49m38.679693134s) [noise=true , address=145.79.227.19:4077 , piece_id=SRCJAWKMZ2QMKFG5ARUG5KTWZGJAGG2S5C5NAKYDRJ4UMGPBKKRA, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.810040463 (595h49m38.681123508s) [noise=true , address=145.79.227.15:4875 , piece_id=AXY25K6VQHIABA3PVCPA6IFYXUPXBTCPPJRGYBRRKN575T4KLSZA, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.812714575 (595h49m38.68379762s) [noise=true , address=145.79.227.47:4266 , piece_id=SE2TB4XQXKZEPU2PSRXTVX6BHHBCSKYJ6A4PAHY2OIU4KFQW2WJA, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.814668165 (595h49m38.68575121s) [noise=true , address=145.79.227.6:4257 , piece_id=VZC2OETBOJ5XDCEZVD7Z3UVN2MEIWD62SUJJMVV742DROS7UKVBA, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.815888208 (595h49m38.686971253s) [noise=true , address=145.79.227.11:4732 , piece_id=WOQ56KBDNBMDGG527CKLE3TYCWXSQ42B7NMQHGAEUTTKKOGWLMWQ, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.829989509 (595h49m38.701072554s) [noise=true , address=145.79.227.9:4365 , piece_id=G6PII4JBEHQUW4JVJKJNR3ZXF6FY5CTAS5HIGIDKJ3KTZJ2WMF6A, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.831214852 (595h49m38.702297897s) [noise=true , address=145.79.227.3:4151 , piece_id=O2N6MCM2EJSSFYR57HE4HYGECADOUENNO25ATUUM2AJIMHQZJE2A, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:38.843511968 (595h49m38.714595013s) [noise=true , address=145.79.227.32:4835 , piece_id=J4OA7BZS2JHHD4VYC65UDVQ64THPLW7C7DGM427KPXANYMXEMHEQ, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:39.240376560 (595h49m39.111459595s) [noise=true , address=45.91.116.8:4048 , piece_id=5BGCJES4XINC5HTFQ7ZVHKFBS74C3AZNIW77F7AUU273OW25KA3Q, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:39.272617302 (595h49m39.143700337s) [noise=true , address=45.91.116.12:4226 , piece_id=UYD2I5Z52P5KMZLYMUYOQS6OHFMG34JEJA2BHR6DETUWH6467FDQ, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:39.407566657 (595h49m39.278649692s) [noise=true , address=77.90.31.19:4844 , piece_id=WILZFHIL4QKSO7NLT2N6I4VLOWJNWXNSM3AMXHXXNHBQC3E6LFHA, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:40.033396193 (595h49m39.904479228s) [noise=true , address=77.90.31.16:4735 , piece_id=RTICAA2XJEZPUX23SBGYDKWUO4BDKG35SMUGQ7FYOJX6B7COVTPA, seg_pos={0 0}, upload=522305]: Done uploading piece. err: <nil> Dec 30 11:18:40.033419418 (595h49m39.904502463s) [seg_pos={0 0}, upload=522305]: Segment reached optimal threshold of 13 pieces. Dec 30 11:18:40.033736962 (595h49m39.904820008s) [noise=true , address=45.91.116.6:4975 , piece_id=3H2KWCIQITGN4G7BIJMKXTUKUZXDUKR5M4AFGITSHWP4ITCT6H6A, seg_pos={0 0}, upload=522305]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:55024->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 11:18:40.033744296 (595h49m39.904827331s) [noise=true , address=45.91.116.6:4975 , piece_id=3H2KWCIQITGN4G7BIJMKXTUKUZXDUKR5M4AFGITSHWP4ITCT6H6A, seg_pos={0 0}, upload=522305]: Long tail cancellation due to optimal threshold Dec 30 11:18:40.033788580 (595h49m39.904871615s) [noise=true , address=145.79.227.40:4049 , piece_id=JO7HTRW6NI46BEYDBLXDJPQ7RFUCJL5SSTDC6U3ZJB2YO3XLIXOA, seg_pos={0 0}, upload=522305]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 11:18:40.033850739 (595h49m39.904933804s) [noise=true , address=145.79.227.40:4049 , piece_id=JO7HTRW6NI46BEYDBLXDJPQ7RFUCJL5SSTDC6U3ZJB2YO3XLIXOA, seg_pos={0 0}, upload=522305]: Long tail cancellation due to optimal threshold Dec 30 11:18:40.033972892 (595h49m39.905055927s) [noise=true , address=45.91.116.9:4263 , piece_id=CFILRAG744ZFMK7KIRMC2DTIV5JDXA67QC3G6WLOB2AYDQSBYJ3Q, seg_pos={0 0}, upload=522305]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 30 11:18:40.033981037 (595h49m39.905064082s) [noise=true , address=45.91.116.9:4263 , piece_id=CFILRAG744ZFMK7KIRMC2DTIV5JDXA67QC3G6WLOB2AYDQSBYJ3Q, seg_pos={0 0}, upload=522305]: Long tail cancellation due to optimal threshold Dec 30 11:18:40.034038827 (595h49m39.905121862s) [seg_pos={0 0}, upload=522305]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:18:40.034123188 (595h49m39.905206233s) [upload=522305]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:18:40.034128458 (595h49m39.905211493s) [upload=522305]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:18:40.334300412 (595h49m40.205383447s) [upload=522305]: Done uploading segments... Dec 30 11:18:40.334323186 (595h49m40.205406221s) [upload=522305]: Upload finished. err: <nil> Duration: 30.005971896s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [.X.X.XXXX.X.XX.X.X.....XX.X........X.X.XX.X.X.X.X...X.XX..XX] duration: 5.001766382s trace: https://jaeger.dev.storj.io/trace/6533ef993b9b0b3e FAIL (2025-12-30 11:19:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001792662s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X.................X..........X...X.......] duration: 19.544580989s trace: https://jaeger.dev.storj.io/trace/7f3400b3387747 FAIL (2025-12-30 11:18:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, Piece ID: OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA) context deadline exceeded; piecestore: (Node ID: 12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, Piece ID: SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ) context deadline exceeded; piecestore: (Node ID: 1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC, Piece ID: 6ZSKVY2RVZAJBTRBKT4RTVQKMJ2BVQDU53I453YBIBBHCZOQCW4A) context deadline exceeded; piecestore: (Node ID: 12YZZnCo54XknuvoBoVtRudXSnoWdNdrn23Lp6Gou3g451BzUwG, Piece ID: 7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ) context deadline exceeded ======LOGS====== s) [noise=true , address=145.79.227.18:4038 , piece_id=K6PIAQ572FP7PVR5G3BRZWSGVYK7XTXG6Z4KBZZ3G5A732BOLS4A, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.892668738 (595h43m54.763751783s) [noise=true , address=145.79.227.40:4079 , piece_id=43LPLVACHLV3VIUR7YZ4RGRCG65FGVSYM5EOPOSG6W6YUSL6OLPQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893082295 (595h43m54.76416534s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893484291 (595h43m54.764567336s) [noise=true , address=77.90.31.12:4613 , piece_id=OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893955759 (595h43m54.765038794s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.894536003 (595h43m54.765619048s) [noise=true , address=145.79.227.44:4226 , piece_id=JZ5CAXU3XK56URASDAYQHQOCON7SCFEECMSM4NJ7O33M647UWAZQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.894858427 (595h43m54.765941462s) [noise=true , address=64.49.13.155:4920 , piece_id=UE53XZKQ5VW5CDBMS7SF3R2FMA3FVNUQA2RLMFMYIIHPDDSNXTZQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.895172897 (595h43m54.766255942s) [noise=true , address=145.79.227.33:4886 , piece_id=GYC5TKM6I5QRBZJCJVRO35ITEGFZWRXEUTA3V76Y65CZ7ZZEDZSQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.896682591 (595h43m54.767765626s) [noise=true , address=145.79.227.29:4953 , piece_id=VTIGSELICMISVDGASJP4W6Z4IOIALQKRPZEULAODHJL4JQHUU7RA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.897870683 (595h43m54.768953728s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.899025873 (595h43m54.770108918s) [noise=true , address=145.79.227.30:4792 , piece_id=T7WHB25ZUMXYNNPCPLQOKLGUPLTQB3XQJ6ZCE3XRXDOYJWJHXFLA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.900208054 (595h43m54.771291099s) [noise=true , address=77.90.31.18:4827 , piece_id=RXOZKEZUKJA4GXKQUZ6ITP2ZW7TPDDXMF7VINT36HGCQNCDCCB2Q, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.901469476 (595h43m54.772552521s) [noise=true , address=77.90.31.15:4706 , piece_id=7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.902861116 (595h43m54.773944161s) [noise=true , address=145.79.227.17:4987 , piece_id=JNZPAZIFJMCQWJELT26MLRIF46UUIYRYD3EKVFX42VIAXYQU4B5Q, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.904040151 (595h43m54.775123196s) [noise=true , address=77.90.31.16:4743 , piece_id=SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.905225688 (595h43m54.776308723s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:56.312272123 (595h43m56.183355158s) [upload=522223]: Next returned nil segment Dec 30 11:12:56.312288104 (595h43m56.183371139s) [upload=522223]: Waiting for error group managing segments... Dec 30 11:12:56.674443063 (595h43m56.545526118s) [noise=true , address=64.49.13.155:4920 , piece_id=UE53XZKQ5VW5CDBMS7SF3R2FMA3FVNUQA2RLMFMYIIHPDDSNXTZQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.748148810 (595h43m56.619231855s) [noise=true , address=145.79.227.17:4987 , piece_id=JNZPAZIFJMCQWJELT26MLRIF46UUIYRYD3EKVFX42VIAXYQU4B5Q, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.748294467 (595h43m56.619377502s) [noise=true , address=145.79.227.18:4038 , piece_id=K6PIAQ572FP7PVR5G3BRZWSGVYK7XTXG6Z4KBZZ3G5A732BOLS4A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.898059835 (595h43m56.76914288s) [noise=true , address=145.79.227.3:4166 , piece_id=GJBVBRV7YZE3GTXEKBW7LJYW35EXK4SDEVIFNZMETDKCZOXCIWKA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.936973813 (595h43m56.808056848s) [noise=true , address=77.90.31.21:4893 , piece_id=6ZSKVY2RVZAJBTRBKT4RTVQKMJ2BVQDU53I453YBIBBHCZOQCW4A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.961395065 (595h43m56.83247811s) [noise=true , address=77.90.31.18:4827 , piece_id=RXOZKEZUKJA4GXKQUZ6ITP2ZW7TPDDXMF7VINT36HGCQNCDCCB2Q, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.967043209 (595h43m56.838126254s) [noise=true , address=77.90.31.15:4706 , piece_id=7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.059782038 (595h43m56.930865073s) [noise=true , address=77.90.31.16:4743 , piece_id=SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.692145558 (595h43m57.563228633s) [noise=true , address=145.79.227.30:4792 , piece_id=T7WHB25ZUMXYNNPCPLQOKLGUPLTQB3XQJ6ZCE3XRXDOYJWJHXFLA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.705427259 (595h43m57.576510304s) [noise=true , address=145.79.227.33:4886 , piece_id=GYC5TKM6I5QRBZJCJVRO35ITEGFZWRXEUTA3V76Y65CZ7ZZEDZSQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.745163051 (595h43m57.616246096s) [noise=true , address=145.79.227.40:4079 , piece_id=43LPLVACHLV3VIUR7YZ4RGRCG65FGVSYM5EOPOSG6W6YUSL6OLPQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.776805663 (595h43m57.647888708s) [noise=true , address=145.79.227.44:4226 , piece_id=JZ5CAXU3XK56URASDAYQHQOCON7SCFEECMSM4NJ7O33M647UWAZQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.859064686 (595h43m57.730147731s) [noise=true , address=145.79.227.29:4953 , piece_id=VTIGSELICMISVDGASJP4W6Z4IOIALQKRPZEULAODHJL4JQHUU7RA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.882701444 (595h43m57.753784479s) [noise=true , address=77.90.31.12:4613 , piece_id=OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.882722825 (595h43m57.75380587s) [seg_pos={0 0}, upload=522223]: Segment reached optimal threshold of 14 pieces. Dec 30 11:12:57.882954816 (595h43m57.754037851s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.882964956 (595h43m57.754048011s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883110403 (595h43m57.754193448s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.883123828 (595h43m57.754206883s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883237274 (595h43m57.754320339s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection Dec 30 11:12:57.883248406 (595h43m57.754331451s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883390406 (595h43m57.754473451s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.883415915 (595h43m57.75449896s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883463475 (595h43m57.75454652s) [seg_pos={0 0}, upload=522223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:12:57.883549108 (595h43m57.754632153s) [upload=522223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:12:57.883554960 (595h43m57.754638005s) [upload=522223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:12:58.175520738 (595h43m58.046603773s) [upload=522223]: Done uploading segments... Dec 30 11:12:58.175538943 (595h43m58.046621978s) [upload=522223]: Upload finished. err: <nil> Duration: 19.544599344s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X.................X..........X...X......] duration: 29.456887664s trace: https://jaeger.dev.storj.io/trace/7f3400b3387747 FAIL (2025-12-30 11:18:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, Piece ID: OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA) context deadline exceeded; piecestore: (Node ID: 12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, Piece ID: SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ) context deadline exceeded; piecestore: (Node ID: 1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC, Piece ID: 6ZSKVY2RVZAJBTRBKT4RTVQKMJ2BVQDU53I453YBIBBHCZOQCW4A) context deadline exceeded; piecestore: (Node ID: 12YZZnCo54XknuvoBoVtRudXSnoWdNdrn23Lp6Gou3g451BzUwG, Piece ID: 7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ) context deadline exceeded ======LOGS====== s) [noise=true , address=145.79.227.18:4038 , piece_id=K6PIAQ572FP7PVR5G3BRZWSGVYK7XTXG6Z4KBZZ3G5A732BOLS4A, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.892668738 (595h43m54.763751783s) [noise=true , address=145.79.227.40:4079 , piece_id=43LPLVACHLV3VIUR7YZ4RGRCG65FGVSYM5EOPOSG6W6YUSL6OLPQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893082295 (595h43m54.76416534s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893484291 (595h43m54.764567336s) [noise=true , address=77.90.31.12:4613 , piece_id=OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893955759 (595h43m54.765038794s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.894536003 (595h43m54.765619048s) [noise=true , address=145.79.227.44:4226 , piece_id=JZ5CAXU3XK56URASDAYQHQOCON7SCFEECMSM4NJ7O33M647UWAZQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.894858427 (595h43m54.765941462s) [noise=true , address=64.49.13.155:4920 , piece_id=UE53XZKQ5VW5CDBMS7SF3R2FMA3FVNUQA2RLMFMYIIHPDDSNXTZQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.895172897 (595h43m54.766255942s) [noise=true , address=145.79.227.33:4886 , piece_id=GYC5TKM6I5QRBZJCJVRO35ITEGFZWRXEUTA3V76Y65CZ7ZZEDZSQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.896682591 (595h43m54.767765626s) [noise=true , address=145.79.227.29:4953 , piece_id=VTIGSELICMISVDGASJP4W6Z4IOIALQKRPZEULAODHJL4JQHUU7RA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.897870683 (595h43m54.768953728s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.899025873 (595h43m54.770108918s) [noise=true , address=145.79.227.30:4792 , piece_id=T7WHB25ZUMXYNNPCPLQOKLGUPLTQB3XQJ6ZCE3XRXDOYJWJHXFLA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.900208054 (595h43m54.771291099s) [noise=true , address=77.90.31.18:4827 , piece_id=RXOZKEZUKJA4GXKQUZ6ITP2ZW7TPDDXMF7VINT36HGCQNCDCCB2Q, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.901469476 (595h43m54.772552521s) [noise=true , address=77.90.31.15:4706 , piece_id=7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.902861116 (595h43m54.773944161s) [noise=true , address=145.79.227.17:4987 , piece_id=JNZPAZIFJMCQWJELT26MLRIF46UUIYRYD3EKVFX42VIAXYQU4B5Q, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.904040151 (595h43m54.775123196s) [noise=true , address=77.90.31.16:4743 , piece_id=SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.905225688 (595h43m54.776308723s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:56.312272123 (595h43m56.183355158s) [upload=522223]: Next returned nil segment Dec 30 11:12:56.312288104 (595h43m56.183371139s) [upload=522223]: Waiting for error group managing segments... Dec 30 11:12:56.674443063 (595h43m56.545526118s) [noise=true , address=64.49.13.155:4920 , piece_id=UE53XZKQ5VW5CDBMS7SF3R2FMA3FVNUQA2RLMFMYIIHPDDSNXTZQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.748148810 (595h43m56.619231855s) [noise=true , address=145.79.227.17:4987 , piece_id=JNZPAZIFJMCQWJELT26MLRIF46UUIYRYD3EKVFX42VIAXYQU4B5Q, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.748294467 (595h43m56.619377502s) [noise=true , address=145.79.227.18:4038 , piece_id=K6PIAQ572FP7PVR5G3BRZWSGVYK7XTXG6Z4KBZZ3G5A732BOLS4A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.898059835 (595h43m56.76914288s) [noise=true , address=145.79.227.3:4166 , piece_id=GJBVBRV7YZE3GTXEKBW7LJYW35EXK4SDEVIFNZMETDKCZOXCIWKA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.936973813 (595h43m56.808056848s) [noise=true , address=77.90.31.21:4893 , piece_id=6ZSKVY2RVZAJBTRBKT4RTVQKMJ2BVQDU53I453YBIBBHCZOQCW4A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.961395065 (595h43m56.83247811s) [noise=true , address=77.90.31.18:4827 , piece_id=RXOZKEZUKJA4GXKQUZ6ITP2ZW7TPDDXMF7VINT36HGCQNCDCCB2Q, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.967043209 (595h43m56.838126254s) [noise=true , address=77.90.31.15:4706 , piece_id=7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.059782038 (595h43m56.930865073s) [noise=true , address=77.90.31.16:4743 , piece_id=SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.692145558 (595h43m57.563228633s) [noise=true , address=145.79.227.30:4792 , piece_id=T7WHB25ZUMXYNNPCPLQOKLGUPLTQB3XQJ6ZCE3XRXDOYJWJHXFLA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.705427259 (595h43m57.576510304s) [noise=true , address=145.79.227.33:4886 , piece_id=GYC5TKM6I5QRBZJCJVRO35ITEGFZWRXEUTA3V76Y65CZ7ZZEDZSQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.745163051 (595h43m57.616246096s) [noise=true , address=145.79.227.40:4079 , piece_id=43LPLVACHLV3VIUR7YZ4RGRCG65FGVSYM5EOPOSG6W6YUSL6OLPQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.776805663 (595h43m57.647888708s) [noise=true , address=145.79.227.44:4226 , piece_id=JZ5CAXU3XK56URASDAYQHQOCON7SCFEECMSM4NJ7O33M647UWAZQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.859064686 (595h43m57.730147731s) [noise=true , address=145.79.227.29:4953 , piece_id=VTIGSELICMISVDGASJP4W6Z4IOIALQKRPZEULAODHJL4JQHUU7RA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.882701444 (595h43m57.753784479s) [noise=true , address=77.90.31.12:4613 , piece_id=OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.882722825 (595h43m57.75380587s) [seg_pos={0 0}, upload=522223]: Segment reached optimal threshold of 14 pieces. Dec 30 11:12:57.882954816 (595h43m57.754037851s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.882964956 (595h43m57.754048011s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883110403 (595h43m57.754193448s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.883123828 (595h43m57.754206883s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883237274 (595h43m57.754320339s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection Dec 30 11:12:57.883248406 (595h43m57.754331451s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883390406 (595h43m57.754473451s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.883415915 (595h43m57.75449896s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883463475 (595h43m57.75454652s) [seg_pos={0 0}, upload=522223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:12:57.883549108 (595h43m57.754632153s) [upload=522223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:12:57.883554960 (595h43m57.754638005s) [upload=522223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:12:58.175520738 (595h43m58.046603773s) [upload=522223]: Done uploading segments... Dec 30 11:12:58.175538943 (595h43m58.046621978s) [upload=522223]: Upload finished. err: <nil> Duration: 29.456904446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XX........X.......X..XX..X.X.X..........X....X.......X.....] duration: 15.782149428s trace: https://jaeger.dev.storj.io/trace/a49828f90b13fd8 FAIL (2025-12-30 11:18:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 11:14:33.461039771 (595h45m33.332122806s) [upload=522248]: Starting upload Dec 30 11:14:33.461095928 (595h45m33.332178973s) [upload=522248]: Done starting upload Dec 30 11:14:33.461295408 (595h45m33.332378443s) [upload=522248]: Uploading segments... Dec 30 11:14:33.461322659 (595h45m33.332405694s) [upload=522248]: Got next segment. Inline: false Dec 30 11:14:33.461350833 (595h45m33.332433868s) [upload=522248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:14:33.461355021 (595h45m33.332438056s) [upload=522248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:14:33.705675815 (595h45m33.57675886s) [seg_pos={0 0}, upload=522248]: Begin upload segment... Dec 30 11:14:33.705758453 (595h45m33.576841488s) [seg_pos={0 0}, upload=522248]: Done begin upload segment. Dec 30 11:14:33.705857572 (595h45m33.576940607s) [noise=true , address=145.79.227.23:4547 , piece_id=QO5SRA4RTQKGIHEGPMLPF2LR2DZXP5CK2YFCLLEM3O2GEC6DL2YA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.705925792 (595h45m33.577008827s) [noise=true , address=145.79.227.27:4691 , piece_id=GOF6BW4ODXGBAQ55TDNJ3PVJI64XG5QMYHQOG52CKIH3KDEK32EQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.706575799 (595h45m33.577658834s) [noise=true , address=145.79.227.6:4259 , piece_id=DVAN4J53QGMGLOH6JISZR5PXYXVPUTEBZZ6DLED2CSOZZISUKFZA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.707044542 (595h45m33.578127587s) [noise=true , address=145.79.227.8:4329 , piece_id=TH5EDBQZHM6ESLMCJ42O6XOSU64PBBNOZTOOX6E5RDDCEDYTGADA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.707390461 (595h45m33.578473506s) [noise=true , address=145.79.227.16:4909 , piece_id=T6ZPOEFKW2GLVIAITLVMD2G2IJYOPKSHOIBCMDDIN35ULBBQ4DHA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.707721300 (595h45m33.578804335s) [noise=true , address=45.91.116.13:4154 , piece_id=DZ5GVQPL46KIQULZEH3FDN5SOU6FCMYDZIKHMT4PFPJA7EJDP2EQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.708023466 (595h45m33.579106511s) [noise=true , address=45.91.116.10:4084 , piece_id=MTWPAPMA3RX5JEXYBH2LVA3WLMYMTMORASURQA2VOGUGNREZZ5WA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.708443075 (595h45m33.57952612s) [noise=true , address=145.79.227.3:4150 , piece_id=5CMFYAYA422V32FMO3BSY2ODQI4HH7AYV3ILRNY3RZNAZWTLCHCA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.708815064 (595h45m33.579898099s) [noise=true , address=145.79.227.18:4007 , piece_id=VV3UB7JX4P2ZR6NLHC4B3H365AFR6EKRVHNC7LOQCKCA4E6JA3JQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.709249871 (595h45m33.580332906s) [noise=true , address=145.79.227.17:4972 , piece_id=O2AAGDAEPPKGT2CNO4UEI7Z2WCVTRCKU2EZ35VDZO5WMMWW2O6GA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.709600409 (595h45m33.580683454s) [noise=true , address=145.79.227.22:4510 , piece_id=Q655IZID2S4J2DAH3FCQCODY46P7JWEJZR32F4IZU6EFGCQTH5JQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.709713263 (595h45m33.580796308s) [noise=true , address=77.90.31.14:4663 , piece_id=7EENYELJEIRDKLCP5IV7577GMSX6NEG6C2AUALMZCC2VRJFLEOVQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.710473391 (595h45m33.581556426s) [noise=true , address=145.79.227.21:4475 , piece_id=XQZCYSQSULXPOHBPJIGOXI6Y5EA4JROGADJBNV67MLRBSIGWJP6Q, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.710786547 (595h45m33.581869592s) [noise=true , address=145.79.227.11:4729 , piece_id=JIVN2ZYMIQ6MJVZEKMDXJP2L5HRKMHFI4QMW7OIG3UGA5I7733FQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.711048306 (595h45m33.582131341s) [noise=true , address=45.91.116.12:4227 , piece_id=T5BZCYEWYKZAEVXQ3LP2WQRSISMQBNAMVHP7OQC2EDGNCU2Y7RBQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.711492452 (595h45m33.582575487s) [noise=true , address=45.91.116.14:4190 , piece_id=UNKLEBU74YRKKF56N44F3BBBBJZYW2KSS7625WODMDFEVBODUTFA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:35.380454405 (595h45m35.25153744s) [upload=522248]: Next returned nil segment Dec 30 11:14:35.380482839 (595h45m35.251565874s) [upload=522248]: Waiting for error group managing segments... Dec 30 11:14:35.687683897 (595h45m35.558766932s) [noise=true , address=145.79.227.22:4510 , piece_id=Q655IZID2S4J2DAH3FCQCODY46P7JWEJZR32F4IZU6EFGCQTH5JQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.688923037 (595h45m35.560006082s) [noise=true , address=145.79.227.11:4729 , piece_id=JIVN2ZYMIQ6MJVZEKMDXJP2L5HRKMHFI4QMW7OIG3UGA5I7733FQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.689502951 (595h45m35.560585986s) [noise=true , address=145.79.227.8:4329 , piece_id=TH5EDBQZHM6ESLMCJ42O6XOSU64PBBNOZTOOX6E5RDDCEDYTGADA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.698663163 (595h45m35.569746198s) [noise=true , address=145.79.227.23:4547 , piece_id=QO5SRA4RTQKGIHEGPMLPF2LR2DZXP5CK2YFCLLEM3O2GEC6DL2YA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.698721875 (595h45m35.56980492s) [noise=true , address=145.79.227.21:4475 , piece_id=XQZCYSQSULXPOHBPJIGOXI6Y5EA4JROGADJBNV67MLRBSIGWJP6Q, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.708251722 (595h45m35.579334757s) [noise=true , address=145.79.227.3:4150 , piece_id=5CMFYAYA422V32FMO3BSY2ODQI4HH7AYV3ILRNY3RZNAZWTLCHCA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.709504758 (595h45m35.580587793s) [noise=true , address=145.79.227.18:4007 , piece_id=VV3UB7JX4P2ZR6NLHC4B3H365AFR6EKRVHNC7LOQCKCA4E6JA3JQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.714229585 (595h45m35.58531262s) [noise=true , address=145.79.227.6:4259 , piece_id=DVAN4J53QGMGLOH6JISZR5PXYXVPUTEBZZ6DLED2CSOZZISUKFZA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.727102366 (595h45m35.598185401s) [noise=true , address=145.79.227.17:4972 , piece_id=O2AAGDAEPPKGT2CNO4UEI7Z2WCVTRCKU2EZ35VDZO5WMMWW2O6GA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.777227693 (595h45m35.648310728s) [noise=true , address=145.79.227.16:4909 , piece_id=T6ZPOEFKW2GLVIAITLVMD2G2IJYOPKSHOIBCMDDIN35ULBBQ4DHA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.786515268 (595h45m35.657598303s) [noise=true , address=145.79.227.27:4691 , piece_id=GOF6BW4ODXGBAQ55TDNJ3PVJI64XG5QMYHQOG52CKIH3KDEK32EQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.988591282 (595h45m35.859674317s) [noise=true , address=77.90.31.14:4663 , piece_id=7EENYELJEIRDKLCP5IV7577GMSX6NEG6C2AUALMZCC2VRJFLEOVQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:36.110014360 (595h45m35.981097395s) [noise=true , address=45.91.116.12:4227 , piece_id=T5BZCYEWYKZAEVXQ3LP2WQRSISMQBNAMVHP7OQC2EDGNCU2Y7RBQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:36.110035300 (595h45m35.981118345s) [seg_pos={0 0}, upload=522248]: Segment reached optimal threshold of 13 pieces. Dec 30 11:14:36.110230792 (595h45m35.981313837s) [noise=true , address=45.91.116.10:4084 , piece_id=MTWPAPMA3RX5JEXYBH2LVA3WLMYMTMORASURQA2VOGUGNREZZ5WA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 11:14:36.110237966 (595h45m35.981321012s) [noise=true , address=45.91.116.10:4084 , piece_id=MTWPAPMA3RX5JEXYBH2LVA3WLMYMTMORASURQA2VOGUGNREZZ5WA, seg_pos={0 0}, upload=522248]: Long tail cancellation due to optimal threshold Dec 30 11:14:36.110515695 (595h45m35.98159873s) [noise=true , address=45.91.116.14:4190 , piece_id=UNKLEBU74YRKKF56N44F3BBBBJZYW2KSS7625WODMDFEVBODUTFA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 11:14:36.110533479 (595h45m35.981616524s) [noise=true , address=45.91.116.14:4190 , piece_id=UNKLEBU74YRKKF56N44F3BBBBJZYW2KSS7625WODMDFEVBODUTFA, seg_pos={0 0}, upload=522248]: Long tail cancellation due to optimal threshold Dec 30 11:14:36.110573946 (595h45m35.981656981s) [noise=true , address=45.91.116.13:4154 , piece_id=DZ5GVQPL46KIQULZEH3FDN5SOU6FCMYDZIKHMT4PFPJA7EJDP2EQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 30 11:14:36.110582261 (595h45m35.981665306s) [noise=true , address=45.91.116.13:4154 , piece_id=DZ5GVQPL46KIQULZEH3FDN5SOU6FCMYDZIKHMT4PFPJA7EJDP2EQ, seg_pos={0 0}, upload=522248]: Long tail cancellation due to optimal threshold Dec 30 11:14:36.110627217 (595h45m35.981710263s) [seg_pos={0 0}, upload=522248]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:14:36.110692742 (595h45m35.981775777s) [upload=522248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:14:36.110698493 (595h45m35.981781528s) [upload=522248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:14:36.398512997 (595h45m36.269596042s) [upload=522248]: Done uploading segments... Dec 30 11:14:36.398591326 (595h45m36.269674371s) [upload=522248]: Upload finished. err: <nil> Duration: 15.782179778s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [..X.X.XXXX.X.XX.X.X.....XX.X........X.X.XX.X.X.X.X...X.XX..X] duration: 5.00202761s trace: https://jaeger.dev.storj.io/trace/4f1909632af441ea FAIL (2025-12-30 11:18:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002051906s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.XX........X.......X..XX..X.X.X..........X....X.......X....] duration: 23.234652459s trace: https://jaeger.dev.storj.io/trace/a49828f90b13fd8 FAIL (2025-12-30 11:17:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 11:14:33.461039771 (595h45m33.332122806s) [upload=522248]: Starting upload Dec 30 11:14:33.461095928 (595h45m33.332178973s) [upload=522248]: Done starting upload Dec 30 11:14:33.461295408 (595h45m33.332378443s) [upload=522248]: Uploading segments... Dec 30 11:14:33.461322659 (595h45m33.332405694s) [upload=522248]: Got next segment. Inline: false Dec 30 11:14:33.461350833 (595h45m33.332433868s) [upload=522248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:14:33.461355021 (595h45m33.332438056s) [upload=522248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:14:33.705675815 (595h45m33.57675886s) [seg_pos={0 0}, upload=522248]: Begin upload segment... Dec 30 11:14:33.705758453 (595h45m33.576841488s) [seg_pos={0 0}, upload=522248]: Done begin upload segment. Dec 30 11:14:33.705857572 (595h45m33.576940607s) [noise=true , address=145.79.227.23:4547 , piece_id=QO5SRA4RTQKGIHEGPMLPF2LR2DZXP5CK2YFCLLEM3O2GEC6DL2YA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.705925792 (595h45m33.577008827s) [noise=true , address=145.79.227.27:4691 , piece_id=GOF6BW4ODXGBAQ55TDNJ3PVJI64XG5QMYHQOG52CKIH3KDEK32EQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.706575799 (595h45m33.577658834s) [noise=true , address=145.79.227.6:4259 , piece_id=DVAN4J53QGMGLOH6JISZR5PXYXVPUTEBZZ6DLED2CSOZZISUKFZA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.707044542 (595h45m33.578127587s) [noise=true , address=145.79.227.8:4329 , piece_id=TH5EDBQZHM6ESLMCJ42O6XOSU64PBBNOZTOOX6E5RDDCEDYTGADA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.707390461 (595h45m33.578473506s) [noise=true , address=145.79.227.16:4909 , piece_id=T6ZPOEFKW2GLVIAITLVMD2G2IJYOPKSHOIBCMDDIN35ULBBQ4DHA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.707721300 (595h45m33.578804335s) [noise=true , address=45.91.116.13:4154 , piece_id=DZ5GVQPL46KIQULZEH3FDN5SOU6FCMYDZIKHMT4PFPJA7EJDP2EQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.708023466 (595h45m33.579106511s) [noise=true , address=45.91.116.10:4084 , piece_id=MTWPAPMA3RX5JEXYBH2LVA3WLMYMTMORASURQA2VOGUGNREZZ5WA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.708443075 (595h45m33.57952612s) [noise=true , address=145.79.227.3:4150 , piece_id=5CMFYAYA422V32FMO3BSY2ODQI4HH7AYV3ILRNY3RZNAZWTLCHCA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.708815064 (595h45m33.579898099s) [noise=true , address=145.79.227.18:4007 , piece_id=VV3UB7JX4P2ZR6NLHC4B3H365AFR6EKRVHNC7LOQCKCA4E6JA3JQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.709249871 (595h45m33.580332906s) [noise=true , address=145.79.227.17:4972 , piece_id=O2AAGDAEPPKGT2CNO4UEI7Z2WCVTRCKU2EZ35VDZO5WMMWW2O6GA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.709600409 (595h45m33.580683454s) [noise=true , address=145.79.227.22:4510 , piece_id=Q655IZID2S4J2DAH3FCQCODY46P7JWEJZR32F4IZU6EFGCQTH5JQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.709713263 (595h45m33.580796308s) [noise=true , address=77.90.31.14:4663 , piece_id=7EENYELJEIRDKLCP5IV7577GMSX6NEG6C2AUALMZCC2VRJFLEOVQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.710473391 (595h45m33.581556426s) [noise=true , address=145.79.227.21:4475 , piece_id=XQZCYSQSULXPOHBPJIGOXI6Y5EA4JROGADJBNV67MLRBSIGWJP6Q, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.710786547 (595h45m33.581869592s) [noise=true , address=145.79.227.11:4729 , piece_id=JIVN2ZYMIQ6MJVZEKMDXJP2L5HRKMHFI4QMW7OIG3UGA5I7733FQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.711048306 (595h45m33.582131341s) [noise=true , address=45.91.116.12:4227 , piece_id=T5BZCYEWYKZAEVXQ3LP2WQRSISMQBNAMVHP7OQC2EDGNCU2Y7RBQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.711492452 (595h45m33.582575487s) [noise=true , address=45.91.116.14:4190 , piece_id=UNKLEBU74YRKKF56N44F3BBBBJZYW2KSS7625WODMDFEVBODUTFA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:35.380454405 (595h45m35.25153744s) [upload=522248]: Next returned nil segment Dec 30 11:14:35.380482839 (595h45m35.251565874s) [upload=522248]: Waiting for error group managing segments... Dec 30 11:14:35.687683897 (595h45m35.558766932s) [noise=true , address=145.79.227.22:4510 , piece_id=Q655IZID2S4J2DAH3FCQCODY46P7JWEJZR32F4IZU6EFGCQTH5JQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.688923037 (595h45m35.560006082s) [noise=true , address=145.79.227.11:4729 , piece_id=JIVN2ZYMIQ6MJVZEKMDXJP2L5HRKMHFI4QMW7OIG3UGA5I7733FQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.689502951 (595h45m35.560585986s) [noise=true , address=145.79.227.8:4329 , piece_id=TH5EDBQZHM6ESLMCJ42O6XOSU64PBBNOZTOOX6E5RDDCEDYTGADA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.698663163 (595h45m35.569746198s) [noise=true , address=145.79.227.23:4547 , piece_id=QO5SRA4RTQKGIHEGPMLPF2LR2DZXP5CK2YFCLLEM3O2GEC6DL2YA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.698721875 (595h45m35.56980492s) [noise=true , address=145.79.227.21:4475 , piece_id=XQZCYSQSULXPOHBPJIGOXI6Y5EA4JROGADJBNV67MLRBSIGWJP6Q, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.708251722 (595h45m35.579334757s) [noise=true , address=145.79.227.3:4150 , piece_id=5CMFYAYA422V32FMO3BSY2ODQI4HH7AYV3ILRNY3RZNAZWTLCHCA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.709504758 (595h45m35.580587793s) [noise=true , address=145.79.227.18:4007 , piece_id=VV3UB7JX4P2ZR6NLHC4B3H365AFR6EKRVHNC7LOQCKCA4E6JA3JQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.714229585 (595h45m35.58531262s) [noise=true , address=145.79.227.6:4259 , piece_id=DVAN4J53QGMGLOH6JISZR5PXYXVPUTEBZZ6DLED2CSOZZISUKFZA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.727102366 (595h45m35.598185401s) [noise=true , address=145.79.227.17:4972 , piece_id=O2AAGDAEPPKGT2CNO4UEI7Z2WCVTRCKU2EZ35VDZO5WMMWW2O6GA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.777227693 (595h45m35.648310728s) [noise=true , address=145.79.227.16:4909 , piece_id=T6ZPOEFKW2GLVIAITLVMD2G2IJYOPKSHOIBCMDDIN35ULBBQ4DHA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.786515268 (595h45m35.657598303s) [noise=true , address=145.79.227.27:4691 , piece_id=GOF6BW4ODXGBAQ55TDNJ3PVJI64XG5QMYHQOG52CKIH3KDEK32EQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.988591282 (595h45m35.859674317s) [noise=true , address=77.90.31.14:4663 , piece_id=7EENYELJEIRDKLCP5IV7577GMSX6NEG6C2AUALMZCC2VRJFLEOVQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:36.110014360 (595h45m35.981097395s) [noise=true , address=45.91.116.12:4227 , piece_id=T5BZCYEWYKZAEVXQ3LP2WQRSISMQBNAMVHP7OQC2EDGNCU2Y7RBQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:36.110035300 (595h45m35.981118345s) [seg_pos={0 0}, upload=522248]: Segment reached optimal threshold of 13 pieces. Dec 30 11:14:36.110230792 (595h45m35.981313837s) [noise=true , address=45.91.116.10:4084 , piece_id=MTWPAPMA3RX5JEXYBH2LVA3WLMYMTMORASURQA2VOGUGNREZZ5WA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 11:14:36.110237966 (595h45m35.981321012s) [noise=true , address=45.91.116.10:4084 , piece_id=MTWPAPMA3RX5JEXYBH2LVA3WLMYMTMORASURQA2VOGUGNREZZ5WA, seg_pos={0 0}, upload=522248]: Long tail cancellation due to optimal threshold Dec 30 11:14:36.110515695 (595h45m35.98159873s) [noise=true , address=45.91.116.14:4190 , piece_id=UNKLEBU74YRKKF56N44F3BBBBJZYW2KSS7625WODMDFEVBODUTFA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 11:14:36.110533479 (595h45m35.981616524s) [noise=true , address=45.91.116.14:4190 , piece_id=UNKLEBU74YRKKF56N44F3BBBBJZYW2KSS7625WODMDFEVBODUTFA, seg_pos={0 0}, upload=522248]: Long tail cancellation due to optimal threshold Dec 30 11:14:36.110573946 (595h45m35.981656981s) [noise=true , address=45.91.116.13:4154 , piece_id=DZ5GVQPL46KIQULZEH3FDN5SOU6FCMYDZIKHMT4PFPJA7EJDP2EQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 30 11:14:36.110582261 (595h45m35.981665306s) [noise=true , address=45.91.116.13:4154 , piece_id=DZ5GVQPL46KIQULZEH3FDN5SOU6FCMYDZIKHMT4PFPJA7EJDP2EQ, seg_pos={0 0}, upload=522248]: Long tail cancellation due to optimal threshold Dec 30 11:14:36.110627217 (595h45m35.981710263s) [seg_pos={0 0}, upload=522248]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:14:36.110692742 (595h45m35.981775777s) [upload=522248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:14:36.110698493 (595h45m35.981781528s) [upload=522248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:14:36.398512997 (595h45m36.269596042s) [upload=522248]: Done uploading segments... Dec 30 11:14:36.398591326 (595h45m36.269674371s) [upload=522248]: Upload finished. err: <nil> Duration: 23.234681185s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [X....................X.................X..........X...X.....] duration: 39.622849935s trace: https://jaeger.dev.storj.io/trace/7f3400b3387747 FAIL (2025-12-30 11:17:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, Piece ID: OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA) context deadline exceeded; piecestore: (Node ID: 12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, Piece ID: SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ) context deadline exceeded; piecestore: (Node ID: 1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC, Piece ID: 6ZSKVY2RVZAJBTRBKT4RTVQKMJ2BVQDU53I453YBIBBHCZOQCW4A) context deadline exceeded; piecestore: (Node ID: 12YZZnCo54XknuvoBoVtRudXSnoWdNdrn23Lp6Gou3g451BzUwG, Piece ID: 7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ) context deadline exceeded ======LOGS====== s) [noise=true , address=145.79.227.18:4038 , piece_id=K6PIAQ572FP7PVR5G3BRZWSGVYK7XTXG6Z4KBZZ3G5A732BOLS4A, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.892668738 (595h43m54.763751783s) [noise=true , address=145.79.227.40:4079 , piece_id=43LPLVACHLV3VIUR7YZ4RGRCG65FGVSYM5EOPOSG6W6YUSL6OLPQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893082295 (595h43m54.76416534s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893484291 (595h43m54.764567336s) [noise=true , address=77.90.31.12:4613 , piece_id=OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893955759 (595h43m54.765038794s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.894536003 (595h43m54.765619048s) [noise=true , address=145.79.227.44:4226 , piece_id=JZ5CAXU3XK56URASDAYQHQOCON7SCFEECMSM4NJ7O33M647UWAZQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.894858427 (595h43m54.765941462s) [noise=true , address=64.49.13.155:4920 , piece_id=UE53XZKQ5VW5CDBMS7SF3R2FMA3FVNUQA2RLMFMYIIHPDDSNXTZQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.895172897 (595h43m54.766255942s) [noise=true , address=145.79.227.33:4886 , piece_id=GYC5TKM6I5QRBZJCJVRO35ITEGFZWRXEUTA3V76Y65CZ7ZZEDZSQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.896682591 (595h43m54.767765626s) [noise=true , address=145.79.227.29:4953 , piece_id=VTIGSELICMISVDGASJP4W6Z4IOIALQKRPZEULAODHJL4JQHUU7RA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.897870683 (595h43m54.768953728s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.899025873 (595h43m54.770108918s) [noise=true , address=145.79.227.30:4792 , piece_id=T7WHB25ZUMXYNNPCPLQOKLGUPLTQB3XQJ6ZCE3XRXDOYJWJHXFLA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.900208054 (595h43m54.771291099s) [noise=true , address=77.90.31.18:4827 , piece_id=RXOZKEZUKJA4GXKQUZ6ITP2ZW7TPDDXMF7VINT36HGCQNCDCCB2Q, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.901469476 (595h43m54.772552521s) [noise=true , address=77.90.31.15:4706 , piece_id=7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.902861116 (595h43m54.773944161s) [noise=true , address=145.79.227.17:4987 , piece_id=JNZPAZIFJMCQWJELT26MLRIF46UUIYRYD3EKVFX42VIAXYQU4B5Q, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.904040151 (595h43m54.775123196s) [noise=true , address=77.90.31.16:4743 , piece_id=SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.905225688 (595h43m54.776308723s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:56.312272123 (595h43m56.183355158s) [upload=522223]: Next returned nil segment Dec 30 11:12:56.312288104 (595h43m56.183371139s) [upload=522223]: Waiting for error group managing segments... Dec 30 11:12:56.674443063 (595h43m56.545526118s) [noise=true , address=64.49.13.155:4920 , piece_id=UE53XZKQ5VW5CDBMS7SF3R2FMA3FVNUQA2RLMFMYIIHPDDSNXTZQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.748148810 (595h43m56.619231855s) [noise=true , address=145.79.227.17:4987 , piece_id=JNZPAZIFJMCQWJELT26MLRIF46UUIYRYD3EKVFX42VIAXYQU4B5Q, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.748294467 (595h43m56.619377502s) [noise=true , address=145.79.227.18:4038 , piece_id=K6PIAQ572FP7PVR5G3BRZWSGVYK7XTXG6Z4KBZZ3G5A732BOLS4A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.898059835 (595h43m56.76914288s) [noise=true , address=145.79.227.3:4166 , piece_id=GJBVBRV7YZE3GTXEKBW7LJYW35EXK4SDEVIFNZMETDKCZOXCIWKA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.936973813 (595h43m56.808056848s) [noise=true , address=77.90.31.21:4893 , piece_id=6ZSKVY2RVZAJBTRBKT4RTVQKMJ2BVQDU53I453YBIBBHCZOQCW4A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.961395065 (595h43m56.83247811s) [noise=true , address=77.90.31.18:4827 , piece_id=RXOZKEZUKJA4GXKQUZ6ITP2ZW7TPDDXMF7VINT36HGCQNCDCCB2Q, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.967043209 (595h43m56.838126254s) [noise=true , address=77.90.31.15:4706 , piece_id=7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.059782038 (595h43m56.930865073s) [noise=true , address=77.90.31.16:4743 , piece_id=SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.692145558 (595h43m57.563228633s) [noise=true , address=145.79.227.30:4792 , piece_id=T7WHB25ZUMXYNNPCPLQOKLGUPLTQB3XQJ6ZCE3XRXDOYJWJHXFLA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.705427259 (595h43m57.576510304s) [noise=true , address=145.79.227.33:4886 , piece_id=GYC5TKM6I5QRBZJCJVRO35ITEGFZWRXEUTA3V76Y65CZ7ZZEDZSQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.745163051 (595h43m57.616246096s) [noise=true , address=145.79.227.40:4079 , piece_id=43LPLVACHLV3VIUR7YZ4RGRCG65FGVSYM5EOPOSG6W6YUSL6OLPQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.776805663 (595h43m57.647888708s) [noise=true , address=145.79.227.44:4226 , piece_id=JZ5CAXU3XK56URASDAYQHQOCON7SCFEECMSM4NJ7O33M647UWAZQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.859064686 (595h43m57.730147731s) [noise=true , address=145.79.227.29:4953 , piece_id=VTIGSELICMISVDGASJP4W6Z4IOIALQKRPZEULAODHJL4JQHUU7RA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.882701444 (595h43m57.753784479s) [noise=true , address=77.90.31.12:4613 , piece_id=OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.882722825 (595h43m57.75380587s) [seg_pos={0 0}, upload=522223]: Segment reached optimal threshold of 14 pieces. Dec 30 11:12:57.882954816 (595h43m57.754037851s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.882964956 (595h43m57.754048011s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883110403 (595h43m57.754193448s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.883123828 (595h43m57.754206883s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883237274 (595h43m57.754320339s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection Dec 30 11:12:57.883248406 (595h43m57.754331451s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883390406 (595h43m57.754473451s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.883415915 (595h43m57.75449896s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883463475 (595h43m57.75454652s) [seg_pos={0 0}, upload=522223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:12:57.883549108 (595h43m57.754632153s) [upload=522223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:12:57.883554960 (595h43m57.754638005s) [upload=522223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:12:58.175520738 (595h43m58.046603773s) [upload=522223]: Done uploading segments... Dec 30 11:12:58.175538943 (595h43m58.046621978s) [upload=522223]: Upload finished. err: <nil> Duration: 39.62288356s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [...X.X.XXXX.X.XX.X.X.....XX.X........X.X.XX.X.X.X.X...X.XX..] duration: 3.417333044s trace: https://jaeger.dev.storj.io/trace/26102834ad6ebfd2 FAIL (2025-12-30 11:17:10 UTC) orbiter warning: context deadline exceeded Duration: 3.417360165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XX.XX........X.......X..XX..X.X.X..........X....X.......X...] duration: 11.099608383s trace: https://jaeger.dev.storj.io/trace/a49828f90b13fd8 FAIL (2025-12-30 11:17:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 11:14:33.461039771 (595h45m33.332122806s) [upload=522248]: Starting upload Dec 30 11:14:33.461095928 (595h45m33.332178973s) [upload=522248]: Done starting upload Dec 30 11:14:33.461295408 (595h45m33.332378443s) [upload=522248]: Uploading segments... Dec 30 11:14:33.461322659 (595h45m33.332405694s) [upload=522248]: Got next segment. Inline: false Dec 30 11:14:33.461350833 (595h45m33.332433868s) [upload=522248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:14:33.461355021 (595h45m33.332438056s) [upload=522248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:14:33.705675815 (595h45m33.57675886s) [seg_pos={0 0}, upload=522248]: Begin upload segment... Dec 30 11:14:33.705758453 (595h45m33.576841488s) [seg_pos={0 0}, upload=522248]: Done begin upload segment. Dec 30 11:14:33.705857572 (595h45m33.576940607s) [noise=true , address=145.79.227.23:4547 , piece_id=QO5SRA4RTQKGIHEGPMLPF2LR2DZXP5CK2YFCLLEM3O2GEC6DL2YA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.705925792 (595h45m33.577008827s) [noise=true , address=145.79.227.27:4691 , piece_id=GOF6BW4ODXGBAQ55TDNJ3PVJI64XG5QMYHQOG52CKIH3KDEK32EQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.706575799 (595h45m33.577658834s) [noise=true , address=145.79.227.6:4259 , piece_id=DVAN4J53QGMGLOH6JISZR5PXYXVPUTEBZZ6DLED2CSOZZISUKFZA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.707044542 (595h45m33.578127587s) [noise=true , address=145.79.227.8:4329 , piece_id=TH5EDBQZHM6ESLMCJ42O6XOSU64PBBNOZTOOX6E5RDDCEDYTGADA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.707390461 (595h45m33.578473506s) [noise=true , address=145.79.227.16:4909 , piece_id=T6ZPOEFKW2GLVIAITLVMD2G2IJYOPKSHOIBCMDDIN35ULBBQ4DHA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.707721300 (595h45m33.578804335s) [noise=true , address=45.91.116.13:4154 , piece_id=DZ5GVQPL46KIQULZEH3FDN5SOU6FCMYDZIKHMT4PFPJA7EJDP2EQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.708023466 (595h45m33.579106511s) [noise=true , address=45.91.116.10:4084 , piece_id=MTWPAPMA3RX5JEXYBH2LVA3WLMYMTMORASURQA2VOGUGNREZZ5WA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.708443075 (595h45m33.57952612s) [noise=true , address=145.79.227.3:4150 , piece_id=5CMFYAYA422V32FMO3BSY2ODQI4HH7AYV3ILRNY3RZNAZWTLCHCA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.708815064 (595h45m33.579898099s) [noise=true , address=145.79.227.18:4007 , piece_id=VV3UB7JX4P2ZR6NLHC4B3H365AFR6EKRVHNC7LOQCKCA4E6JA3JQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.709249871 (595h45m33.580332906s) [noise=true , address=145.79.227.17:4972 , piece_id=O2AAGDAEPPKGT2CNO4UEI7Z2WCVTRCKU2EZ35VDZO5WMMWW2O6GA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.709600409 (595h45m33.580683454s) [noise=true , address=145.79.227.22:4510 , piece_id=Q655IZID2S4J2DAH3FCQCODY46P7JWEJZR32F4IZU6EFGCQTH5JQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.709713263 (595h45m33.580796308s) [noise=true , address=77.90.31.14:4663 , piece_id=7EENYELJEIRDKLCP5IV7577GMSX6NEG6C2AUALMZCC2VRJFLEOVQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.710473391 (595h45m33.581556426s) [noise=true , address=145.79.227.21:4475 , piece_id=XQZCYSQSULXPOHBPJIGOXI6Y5EA4JROGADJBNV67MLRBSIGWJP6Q, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.710786547 (595h45m33.581869592s) [noise=true , address=145.79.227.11:4729 , piece_id=JIVN2ZYMIQ6MJVZEKMDXJP2L5HRKMHFI4QMW7OIG3UGA5I7733FQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.711048306 (595h45m33.582131341s) [noise=true , address=45.91.116.12:4227 , piece_id=T5BZCYEWYKZAEVXQ3LP2WQRSISMQBNAMVHP7OQC2EDGNCU2Y7RBQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.711492452 (595h45m33.582575487s) [noise=true , address=45.91.116.14:4190 , piece_id=UNKLEBU74YRKKF56N44F3BBBBJZYW2KSS7625WODMDFEVBODUTFA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:35.380454405 (595h45m35.25153744s) [upload=522248]: Next returned nil segment Dec 30 11:14:35.380482839 (595h45m35.251565874s) [upload=522248]: Waiting for error group managing segments... Dec 30 11:14:35.687683897 (595h45m35.558766932s) [noise=true , address=145.79.227.22:4510 , piece_id=Q655IZID2S4J2DAH3FCQCODY46P7JWEJZR32F4IZU6EFGCQTH5JQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.688923037 (595h45m35.560006082s) [noise=true , address=145.79.227.11:4729 , piece_id=JIVN2ZYMIQ6MJVZEKMDXJP2L5HRKMHFI4QMW7OIG3UGA5I7733FQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.689502951 (595h45m35.560585986s) [noise=true , address=145.79.227.8:4329 , piece_id=TH5EDBQZHM6ESLMCJ42O6XOSU64PBBNOZTOOX6E5RDDCEDYTGADA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.698663163 (595h45m35.569746198s) [noise=true , address=145.79.227.23:4547 , piece_id=QO5SRA4RTQKGIHEGPMLPF2LR2DZXP5CK2YFCLLEM3O2GEC6DL2YA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.698721875 (595h45m35.56980492s) [noise=true , address=145.79.227.21:4475 , piece_id=XQZCYSQSULXPOHBPJIGOXI6Y5EA4JROGADJBNV67MLRBSIGWJP6Q, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.708251722 (595h45m35.579334757s) [noise=true , address=145.79.227.3:4150 , piece_id=5CMFYAYA422V32FMO3BSY2ODQI4HH7AYV3ILRNY3RZNAZWTLCHCA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.709504758 (595h45m35.580587793s) [noise=true , address=145.79.227.18:4007 , piece_id=VV3UB7JX4P2ZR6NLHC4B3H365AFR6EKRVHNC7LOQCKCA4E6JA3JQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.714229585 (595h45m35.58531262s) [noise=true , address=145.79.227.6:4259 , piece_id=DVAN4J53QGMGLOH6JISZR5PXYXVPUTEBZZ6DLED2CSOZZISUKFZA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.727102366 (595h45m35.598185401s) [noise=true , address=145.79.227.17:4972 , piece_id=O2AAGDAEPPKGT2CNO4UEI7Z2WCVTRCKU2EZ35VDZO5WMMWW2O6GA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.777227693 (595h45m35.648310728s) [noise=true , address=145.79.227.16:4909 , piece_id=T6ZPOEFKW2GLVIAITLVMD2G2IJYOPKSHOIBCMDDIN35ULBBQ4DHA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.786515268 (595h45m35.657598303s) [noise=true , address=145.79.227.27:4691 , piece_id=GOF6BW4ODXGBAQ55TDNJ3PVJI64XG5QMYHQOG52CKIH3KDEK32EQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.988591282 (595h45m35.859674317s) [noise=true , address=77.90.31.14:4663 , piece_id=7EENYELJEIRDKLCP5IV7577GMSX6NEG6C2AUALMZCC2VRJFLEOVQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:36.110014360 (595h45m35.981097395s) [noise=true , address=45.91.116.12:4227 , piece_id=T5BZCYEWYKZAEVXQ3LP2WQRSISMQBNAMVHP7OQC2EDGNCU2Y7RBQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:36.110035300 (595h45m35.981118345s) [seg_pos={0 0}, upload=522248]: Segment reached optimal threshold of 13 pieces. Dec 30 11:14:36.110230792 (595h45m35.981313837s) [noise=true , address=45.91.116.10:4084 , piece_id=MTWPAPMA3RX5JEXYBH2LVA3WLMYMTMORASURQA2VOGUGNREZZ5WA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 11:14:36.110237966 (595h45m35.981321012s) [noise=true , address=45.91.116.10:4084 , piece_id=MTWPAPMA3RX5JEXYBH2LVA3WLMYMTMORASURQA2VOGUGNREZZ5WA, seg_pos={0 0}, upload=522248]: Long tail cancellation due to optimal threshold Dec 30 11:14:36.110515695 (595h45m35.98159873s) [noise=true , address=45.91.116.14:4190 , piece_id=UNKLEBU74YRKKF56N44F3BBBBJZYW2KSS7625WODMDFEVBODUTFA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 11:14:36.110533479 (595h45m35.981616524s) [noise=true , address=45.91.116.14:4190 , piece_id=UNKLEBU74YRKKF56N44F3BBBBJZYW2KSS7625WODMDFEVBODUTFA, seg_pos={0 0}, upload=522248]: Long tail cancellation due to optimal threshold Dec 30 11:14:36.110573946 (595h45m35.981656981s) [noise=true , address=45.91.116.13:4154 , piece_id=DZ5GVQPL46KIQULZEH3FDN5SOU6FCMYDZIKHMT4PFPJA7EJDP2EQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 30 11:14:36.110582261 (595h45m35.981665306s) [noise=true , address=45.91.116.13:4154 , piece_id=DZ5GVQPL46KIQULZEH3FDN5SOU6FCMYDZIKHMT4PFPJA7EJDP2EQ, seg_pos={0 0}, upload=522248]: Long tail cancellation due to optimal threshold Dec 30 11:14:36.110627217 (595h45m35.981710263s) [seg_pos={0 0}, upload=522248]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:14:36.110692742 (595h45m35.981775777s) [upload=522248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:14:36.110698493 (595h45m35.981781528s) [upload=522248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:14:36.398512997 (595h45m36.269596042s) [upload=522248]: Done uploading segments... Dec 30 11:14:36.398591326 (595h45m36.269674371s) [upload=522248]: Upload finished. err: <nil> Duration: 11.099631297s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X....................X.................X..........X...X....] duration: 25.570542917s trace: https://jaeger.dev.storj.io/trace/7f3400b3387747 FAIL (2025-12-30 11:16:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, Piece ID: OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA) context deadline exceeded; piecestore: (Node ID: 12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, Piece ID: SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ) context deadline exceeded; piecestore: (Node ID: 1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC, Piece ID: 6ZSKVY2RVZAJBTRBKT4RTVQKMJ2BVQDU53I453YBIBBHCZOQCW4A) context deadline exceeded; piecestore: (Node ID: 12YZZnCo54XknuvoBoVtRudXSnoWdNdrn23Lp6Gou3g451BzUwG, Piece ID: 7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ) context deadline exceeded ======LOGS====== s) [noise=true , address=145.79.227.18:4038 , piece_id=K6PIAQ572FP7PVR5G3BRZWSGVYK7XTXG6Z4KBZZ3G5A732BOLS4A, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.892668738 (595h43m54.763751783s) [noise=true , address=145.79.227.40:4079 , piece_id=43LPLVACHLV3VIUR7YZ4RGRCG65FGVSYM5EOPOSG6W6YUSL6OLPQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893082295 (595h43m54.76416534s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893484291 (595h43m54.764567336s) [noise=true , address=77.90.31.12:4613 , piece_id=OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893955759 (595h43m54.765038794s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.894536003 (595h43m54.765619048s) [noise=true , address=145.79.227.44:4226 , piece_id=JZ5CAXU3XK56URASDAYQHQOCON7SCFEECMSM4NJ7O33M647UWAZQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.894858427 (595h43m54.765941462s) [noise=true , address=64.49.13.155:4920 , piece_id=UE53XZKQ5VW5CDBMS7SF3R2FMA3FVNUQA2RLMFMYIIHPDDSNXTZQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.895172897 (595h43m54.766255942s) [noise=true , address=145.79.227.33:4886 , piece_id=GYC5TKM6I5QRBZJCJVRO35ITEGFZWRXEUTA3V76Y65CZ7ZZEDZSQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.896682591 (595h43m54.767765626s) [noise=true , address=145.79.227.29:4953 , piece_id=VTIGSELICMISVDGASJP4W6Z4IOIALQKRPZEULAODHJL4JQHUU7RA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.897870683 (595h43m54.768953728s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.899025873 (595h43m54.770108918s) [noise=true , address=145.79.227.30:4792 , piece_id=T7WHB25ZUMXYNNPCPLQOKLGUPLTQB3XQJ6ZCE3XRXDOYJWJHXFLA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.900208054 (595h43m54.771291099s) [noise=true , address=77.90.31.18:4827 , piece_id=RXOZKEZUKJA4GXKQUZ6ITP2ZW7TPDDXMF7VINT36HGCQNCDCCB2Q, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.901469476 (595h43m54.772552521s) [noise=true , address=77.90.31.15:4706 , piece_id=7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.902861116 (595h43m54.773944161s) [noise=true , address=145.79.227.17:4987 , piece_id=JNZPAZIFJMCQWJELT26MLRIF46UUIYRYD3EKVFX42VIAXYQU4B5Q, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.904040151 (595h43m54.775123196s) [noise=true , address=77.90.31.16:4743 , piece_id=SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.905225688 (595h43m54.776308723s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:56.312272123 (595h43m56.183355158s) [upload=522223]: Next returned nil segment Dec 30 11:12:56.312288104 (595h43m56.183371139s) [upload=522223]: Waiting for error group managing segments... Dec 30 11:12:56.674443063 (595h43m56.545526118s) [noise=true , address=64.49.13.155:4920 , piece_id=UE53XZKQ5VW5CDBMS7SF3R2FMA3FVNUQA2RLMFMYIIHPDDSNXTZQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.748148810 (595h43m56.619231855s) [noise=true , address=145.79.227.17:4987 , piece_id=JNZPAZIFJMCQWJELT26MLRIF46UUIYRYD3EKVFX42VIAXYQU4B5Q, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.748294467 (595h43m56.619377502s) [noise=true , address=145.79.227.18:4038 , piece_id=K6PIAQ572FP7PVR5G3BRZWSGVYK7XTXG6Z4KBZZ3G5A732BOLS4A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.898059835 (595h43m56.76914288s) [noise=true , address=145.79.227.3:4166 , piece_id=GJBVBRV7YZE3GTXEKBW7LJYW35EXK4SDEVIFNZMETDKCZOXCIWKA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.936973813 (595h43m56.808056848s) [noise=true , address=77.90.31.21:4893 , piece_id=6ZSKVY2RVZAJBTRBKT4RTVQKMJ2BVQDU53I453YBIBBHCZOQCW4A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.961395065 (595h43m56.83247811s) [noise=true , address=77.90.31.18:4827 , piece_id=RXOZKEZUKJA4GXKQUZ6ITP2ZW7TPDDXMF7VINT36HGCQNCDCCB2Q, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.967043209 (595h43m56.838126254s) [noise=true , address=77.90.31.15:4706 , piece_id=7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.059782038 (595h43m56.930865073s) [noise=true , address=77.90.31.16:4743 , piece_id=SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.692145558 (595h43m57.563228633s) [noise=true , address=145.79.227.30:4792 , piece_id=T7WHB25ZUMXYNNPCPLQOKLGUPLTQB3XQJ6ZCE3XRXDOYJWJHXFLA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.705427259 (595h43m57.576510304s) [noise=true , address=145.79.227.33:4886 , piece_id=GYC5TKM6I5QRBZJCJVRO35ITEGFZWRXEUTA3V76Y65CZ7ZZEDZSQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.745163051 (595h43m57.616246096s) [noise=true , address=145.79.227.40:4079 , piece_id=43LPLVACHLV3VIUR7YZ4RGRCG65FGVSYM5EOPOSG6W6YUSL6OLPQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.776805663 (595h43m57.647888708s) [noise=true , address=145.79.227.44:4226 , piece_id=JZ5CAXU3XK56URASDAYQHQOCON7SCFEECMSM4NJ7O33M647UWAZQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.859064686 (595h43m57.730147731s) [noise=true , address=145.79.227.29:4953 , piece_id=VTIGSELICMISVDGASJP4W6Z4IOIALQKRPZEULAODHJL4JQHUU7RA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.882701444 (595h43m57.753784479s) [noise=true , address=77.90.31.12:4613 , piece_id=OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.882722825 (595h43m57.75380587s) [seg_pos={0 0}, upload=522223]: Segment reached optimal threshold of 14 pieces. Dec 30 11:12:57.882954816 (595h43m57.754037851s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.882964956 (595h43m57.754048011s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883110403 (595h43m57.754193448s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.883123828 (595h43m57.754206883s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883237274 (595h43m57.754320339s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection Dec 30 11:12:57.883248406 (595h43m57.754331451s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883390406 (595h43m57.754473451s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.883415915 (595h43m57.75449896s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883463475 (595h43m57.75454652s) [seg_pos={0 0}, upload=522223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:12:57.883549108 (595h43m57.754632153s) [upload=522223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:12:57.883554960 (595h43m57.754638005s) [upload=522223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:12:58.175520738 (595h43m58.046603773s) [upload=522223]: Done uploading segments... Dec 30 11:12:58.175538943 (595h43m58.046621978s) [upload=522223]: Upload finished. err: <nil> Duration: 25.570563155s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X....................X.................X..........X...X...] duration: 45.121596771s trace: https://jaeger.dev.storj.io/trace/7f3400b3387747 FAIL (2025-12-30 11:16:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, Piece ID: OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA) context deadline exceeded; piecestore: (Node ID: 12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, Piece ID: SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ) context deadline exceeded; piecestore: (Node ID: 1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC, Piece ID: 6ZSKVY2RVZAJBTRBKT4RTVQKMJ2BVQDU53I453YBIBBHCZOQCW4A) context deadline exceeded; piecestore: (Node ID: 12YZZnCo54XknuvoBoVtRudXSnoWdNdrn23Lp6Gou3g451BzUwG, Piece ID: 7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ) context deadline exceeded ======LOGS====== s) [noise=true , address=145.79.227.18:4038 , piece_id=K6PIAQ572FP7PVR5G3BRZWSGVYK7XTXG6Z4KBZZ3G5A732BOLS4A, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.892668738 (595h43m54.763751783s) [noise=true , address=145.79.227.40:4079 , piece_id=43LPLVACHLV3VIUR7YZ4RGRCG65FGVSYM5EOPOSG6W6YUSL6OLPQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893082295 (595h43m54.76416534s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893484291 (595h43m54.764567336s) [noise=true , address=77.90.31.12:4613 , piece_id=OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893955759 (595h43m54.765038794s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.894536003 (595h43m54.765619048s) [noise=true , address=145.79.227.44:4226 , piece_id=JZ5CAXU3XK56URASDAYQHQOCON7SCFEECMSM4NJ7O33M647UWAZQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.894858427 (595h43m54.765941462s) [noise=true , address=64.49.13.155:4920 , piece_id=UE53XZKQ5VW5CDBMS7SF3R2FMA3FVNUQA2RLMFMYIIHPDDSNXTZQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.895172897 (595h43m54.766255942s) [noise=true , address=145.79.227.33:4886 , piece_id=GYC5TKM6I5QRBZJCJVRO35ITEGFZWRXEUTA3V76Y65CZ7ZZEDZSQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.896682591 (595h43m54.767765626s) [noise=true , address=145.79.227.29:4953 , piece_id=VTIGSELICMISVDGASJP4W6Z4IOIALQKRPZEULAODHJL4JQHUU7RA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.897870683 (595h43m54.768953728s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.899025873 (595h43m54.770108918s) [noise=true , address=145.79.227.30:4792 , piece_id=T7WHB25ZUMXYNNPCPLQOKLGUPLTQB3XQJ6ZCE3XRXDOYJWJHXFLA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.900208054 (595h43m54.771291099s) [noise=true , address=77.90.31.18:4827 , piece_id=RXOZKEZUKJA4GXKQUZ6ITP2ZW7TPDDXMF7VINT36HGCQNCDCCB2Q, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.901469476 (595h43m54.772552521s) [noise=true , address=77.90.31.15:4706 , piece_id=7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.902861116 (595h43m54.773944161s) [noise=true , address=145.79.227.17:4987 , piece_id=JNZPAZIFJMCQWJELT26MLRIF46UUIYRYD3EKVFX42VIAXYQU4B5Q, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.904040151 (595h43m54.775123196s) [noise=true , address=77.90.31.16:4743 , piece_id=SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.905225688 (595h43m54.776308723s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:56.312272123 (595h43m56.183355158s) [upload=522223]: Next returned nil segment Dec 30 11:12:56.312288104 (595h43m56.183371139s) [upload=522223]: Waiting for error group managing segments... Dec 30 11:12:56.674443063 (595h43m56.545526118s) [noise=true , address=64.49.13.155:4920 , piece_id=UE53XZKQ5VW5CDBMS7SF3R2FMA3FVNUQA2RLMFMYIIHPDDSNXTZQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.748148810 (595h43m56.619231855s) [noise=true , address=145.79.227.17:4987 , piece_id=JNZPAZIFJMCQWJELT26MLRIF46UUIYRYD3EKVFX42VIAXYQU4B5Q, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.748294467 (595h43m56.619377502s) [noise=true , address=145.79.227.18:4038 , piece_id=K6PIAQ572FP7PVR5G3BRZWSGVYK7XTXG6Z4KBZZ3G5A732BOLS4A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.898059835 (595h43m56.76914288s) [noise=true , address=145.79.227.3:4166 , piece_id=GJBVBRV7YZE3GTXEKBW7LJYW35EXK4SDEVIFNZMETDKCZOXCIWKA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.936973813 (595h43m56.808056848s) [noise=true , address=77.90.31.21:4893 , piece_id=6ZSKVY2RVZAJBTRBKT4RTVQKMJ2BVQDU53I453YBIBBHCZOQCW4A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.961395065 (595h43m56.83247811s) [noise=true , address=77.90.31.18:4827 , piece_id=RXOZKEZUKJA4GXKQUZ6ITP2ZW7TPDDXMF7VINT36HGCQNCDCCB2Q, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.967043209 (595h43m56.838126254s) [noise=true , address=77.90.31.15:4706 , piece_id=7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.059782038 (595h43m56.930865073s) [noise=true , address=77.90.31.16:4743 , piece_id=SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.692145558 (595h43m57.563228633s) [noise=true , address=145.79.227.30:4792 , piece_id=T7WHB25ZUMXYNNPCPLQOKLGUPLTQB3XQJ6ZCE3XRXDOYJWJHXFLA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.705427259 (595h43m57.576510304s) [noise=true , address=145.79.227.33:4886 , piece_id=GYC5TKM6I5QRBZJCJVRO35ITEGFZWRXEUTA3V76Y65CZ7ZZEDZSQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.745163051 (595h43m57.616246096s) [noise=true , address=145.79.227.40:4079 , piece_id=43LPLVACHLV3VIUR7YZ4RGRCG65FGVSYM5EOPOSG6W6YUSL6OLPQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.776805663 (595h43m57.647888708s) [noise=true , address=145.79.227.44:4226 , piece_id=JZ5CAXU3XK56URASDAYQHQOCON7SCFEECMSM4NJ7O33M647UWAZQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.859064686 (595h43m57.730147731s) [noise=true , address=145.79.227.29:4953 , piece_id=VTIGSELICMISVDGASJP4W6Z4IOIALQKRPZEULAODHJL4JQHUU7RA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.882701444 (595h43m57.753784479s) [noise=true , address=77.90.31.12:4613 , piece_id=OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.882722825 (595h43m57.75380587s) [seg_pos={0 0}, upload=522223]: Segment reached optimal threshold of 14 pieces. Dec 30 11:12:57.882954816 (595h43m57.754037851s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.882964956 (595h43m57.754048011s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883110403 (595h43m57.754193448s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.883123828 (595h43m57.754206883s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883237274 (595h43m57.754320339s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection Dec 30 11:12:57.883248406 (595h43m57.754331451s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883390406 (595h43m57.754473451s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.883415915 (595h43m57.75449896s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883463475 (595h43m57.75454652s) [seg_pos={0 0}, upload=522223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:12:57.883549108 (595h43m57.754632153s) [upload=522223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:12:57.883554960 (595h43m57.754638005s) [upload=522223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:12:58.175520738 (595h43m58.046603773s) [upload=522223]: Done uploading segments... Dec 30 11:12:58.175538943 (595h43m58.046621978s) [upload=522223]: Upload finished. err: <nil> Duration: 45.121625576s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.XX.XX........X.......X..XX..X.X.X..........X....X.......X..] duration: 10.028764737s trace: https://jaeger.dev.storj.io/trace/a49828f90b13fd8 FAIL (2025-12-30 11:16:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 11:14:33.461039771 (595h45m33.332122806s) [upload=522248]: Starting upload Dec 30 11:14:33.461095928 (595h45m33.332178973s) [upload=522248]: Done starting upload Dec 30 11:14:33.461295408 (595h45m33.332378443s) [upload=522248]: Uploading segments... Dec 30 11:14:33.461322659 (595h45m33.332405694s) [upload=522248]: Got next segment. Inline: false Dec 30 11:14:33.461350833 (595h45m33.332433868s) [upload=522248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:14:33.461355021 (595h45m33.332438056s) [upload=522248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:14:33.705675815 (595h45m33.57675886s) [seg_pos={0 0}, upload=522248]: Begin upload segment... Dec 30 11:14:33.705758453 (595h45m33.576841488s) [seg_pos={0 0}, upload=522248]: Done begin upload segment. Dec 30 11:14:33.705857572 (595h45m33.576940607s) [noise=true , address=145.79.227.23:4547 , piece_id=QO5SRA4RTQKGIHEGPMLPF2LR2DZXP5CK2YFCLLEM3O2GEC6DL2YA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.705925792 (595h45m33.577008827s) [noise=true , address=145.79.227.27:4691 , piece_id=GOF6BW4ODXGBAQ55TDNJ3PVJI64XG5QMYHQOG52CKIH3KDEK32EQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.706575799 (595h45m33.577658834s) [noise=true , address=145.79.227.6:4259 , piece_id=DVAN4J53QGMGLOH6JISZR5PXYXVPUTEBZZ6DLED2CSOZZISUKFZA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.707044542 (595h45m33.578127587s) [noise=true , address=145.79.227.8:4329 , piece_id=TH5EDBQZHM6ESLMCJ42O6XOSU64PBBNOZTOOX6E5RDDCEDYTGADA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.707390461 (595h45m33.578473506s) [noise=true , address=145.79.227.16:4909 , piece_id=T6ZPOEFKW2GLVIAITLVMD2G2IJYOPKSHOIBCMDDIN35ULBBQ4DHA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.707721300 (595h45m33.578804335s) [noise=true , address=45.91.116.13:4154 , piece_id=DZ5GVQPL46KIQULZEH3FDN5SOU6FCMYDZIKHMT4PFPJA7EJDP2EQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.708023466 (595h45m33.579106511s) [noise=true , address=45.91.116.10:4084 , piece_id=MTWPAPMA3RX5JEXYBH2LVA3WLMYMTMORASURQA2VOGUGNREZZ5WA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.708443075 (595h45m33.57952612s) [noise=true , address=145.79.227.3:4150 , piece_id=5CMFYAYA422V32FMO3BSY2ODQI4HH7AYV3ILRNY3RZNAZWTLCHCA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.708815064 (595h45m33.579898099s) [noise=true , address=145.79.227.18:4007 , piece_id=VV3UB7JX4P2ZR6NLHC4B3H365AFR6EKRVHNC7LOQCKCA4E6JA3JQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.709249871 (595h45m33.580332906s) [noise=true , address=145.79.227.17:4972 , piece_id=O2AAGDAEPPKGT2CNO4UEI7Z2WCVTRCKU2EZ35VDZO5WMMWW2O6GA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.709600409 (595h45m33.580683454s) [noise=true , address=145.79.227.22:4510 , piece_id=Q655IZID2S4J2DAH3FCQCODY46P7JWEJZR32F4IZU6EFGCQTH5JQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.709713263 (595h45m33.580796308s) [noise=true , address=77.90.31.14:4663 , piece_id=7EENYELJEIRDKLCP5IV7577GMSX6NEG6C2AUALMZCC2VRJFLEOVQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.710473391 (595h45m33.581556426s) [noise=true , address=145.79.227.21:4475 , piece_id=XQZCYSQSULXPOHBPJIGOXI6Y5EA4JROGADJBNV67MLRBSIGWJP6Q, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.710786547 (595h45m33.581869592s) [noise=true , address=145.79.227.11:4729 , piece_id=JIVN2ZYMIQ6MJVZEKMDXJP2L5HRKMHFI4QMW7OIG3UGA5I7733FQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.711048306 (595h45m33.582131341s) [noise=true , address=45.91.116.12:4227 , piece_id=T5BZCYEWYKZAEVXQ3LP2WQRSISMQBNAMVHP7OQC2EDGNCU2Y7RBQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.711492452 (595h45m33.582575487s) [noise=true , address=45.91.116.14:4190 , piece_id=UNKLEBU74YRKKF56N44F3BBBBJZYW2KSS7625WODMDFEVBODUTFA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:35.380454405 (595h45m35.25153744s) [upload=522248]: Next returned nil segment Dec 30 11:14:35.380482839 (595h45m35.251565874s) [upload=522248]: Waiting for error group managing segments... Dec 30 11:14:35.687683897 (595h45m35.558766932s) [noise=true , address=145.79.227.22:4510 , piece_id=Q655IZID2S4J2DAH3FCQCODY46P7JWEJZR32F4IZU6EFGCQTH5JQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.688923037 (595h45m35.560006082s) [noise=true , address=145.79.227.11:4729 , piece_id=JIVN2ZYMIQ6MJVZEKMDXJP2L5HRKMHFI4QMW7OIG3UGA5I7733FQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.689502951 (595h45m35.560585986s) [noise=true , address=145.79.227.8:4329 , piece_id=TH5EDBQZHM6ESLMCJ42O6XOSU64PBBNOZTOOX6E5RDDCEDYTGADA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.698663163 (595h45m35.569746198s) [noise=true , address=145.79.227.23:4547 , piece_id=QO5SRA4RTQKGIHEGPMLPF2LR2DZXP5CK2YFCLLEM3O2GEC6DL2YA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.698721875 (595h45m35.56980492s) [noise=true , address=145.79.227.21:4475 , piece_id=XQZCYSQSULXPOHBPJIGOXI6Y5EA4JROGADJBNV67MLRBSIGWJP6Q, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.708251722 (595h45m35.579334757s) [noise=true , address=145.79.227.3:4150 , piece_id=5CMFYAYA422V32FMO3BSY2ODQI4HH7AYV3ILRNY3RZNAZWTLCHCA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.709504758 (595h45m35.580587793s) [noise=true , address=145.79.227.18:4007 , piece_id=VV3UB7JX4P2ZR6NLHC4B3H365AFR6EKRVHNC7LOQCKCA4E6JA3JQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.714229585 (595h45m35.58531262s) [noise=true , address=145.79.227.6:4259 , piece_id=DVAN4J53QGMGLOH6JISZR5PXYXVPUTEBZZ6DLED2CSOZZISUKFZA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.727102366 (595h45m35.598185401s) [noise=true , address=145.79.227.17:4972 , piece_id=O2AAGDAEPPKGT2CNO4UEI7Z2WCVTRCKU2EZ35VDZO5WMMWW2O6GA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.777227693 (595h45m35.648310728s) [noise=true , address=145.79.227.16:4909 , piece_id=T6ZPOEFKW2GLVIAITLVMD2G2IJYOPKSHOIBCMDDIN35ULBBQ4DHA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.786515268 (595h45m35.657598303s) [noise=true , address=145.79.227.27:4691 , piece_id=GOF6BW4ODXGBAQ55TDNJ3PVJI64XG5QMYHQOG52CKIH3KDEK32EQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.988591282 (595h45m35.859674317s) [noise=true , address=77.90.31.14:4663 , piece_id=7EENYELJEIRDKLCP5IV7577GMSX6NEG6C2AUALMZCC2VRJFLEOVQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:36.110014360 (595h45m35.981097395s) [noise=true , address=45.91.116.12:4227 , piece_id=T5BZCYEWYKZAEVXQ3LP2WQRSISMQBNAMVHP7OQC2EDGNCU2Y7RBQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:36.110035300 (595h45m35.981118345s) [seg_pos={0 0}, upload=522248]: Segment reached optimal threshold of 13 pieces. Dec 30 11:14:36.110230792 (595h45m35.981313837s) [noise=true , address=45.91.116.10:4084 , piece_id=MTWPAPMA3RX5JEXYBH2LVA3WLMYMTMORASURQA2VOGUGNREZZ5WA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 11:14:36.110237966 (595h45m35.981321012s) [noise=true , address=45.91.116.10:4084 , piece_id=MTWPAPMA3RX5JEXYBH2LVA3WLMYMTMORASURQA2VOGUGNREZZ5WA, seg_pos={0 0}, upload=522248]: Long tail cancellation due to optimal threshold Dec 30 11:14:36.110515695 (595h45m35.98159873s) [noise=true , address=45.91.116.14:4190 , piece_id=UNKLEBU74YRKKF56N44F3BBBBJZYW2KSS7625WODMDFEVBODUTFA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 11:14:36.110533479 (595h45m35.981616524s) [noise=true , address=45.91.116.14:4190 , piece_id=UNKLEBU74YRKKF56N44F3BBBBJZYW2KSS7625WODMDFEVBODUTFA, seg_pos={0 0}, upload=522248]: Long tail cancellation due to optimal threshold Dec 30 11:14:36.110573946 (595h45m35.981656981s) [noise=true , address=45.91.116.13:4154 , piece_id=DZ5GVQPL46KIQULZEH3FDN5SOU6FCMYDZIKHMT4PFPJA7EJDP2EQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 30 11:14:36.110582261 (595h45m35.981665306s) [noise=true , address=45.91.116.13:4154 , piece_id=DZ5GVQPL46KIQULZEH3FDN5SOU6FCMYDZIKHMT4PFPJA7EJDP2EQ, seg_pos={0 0}, upload=522248]: Long tail cancellation due to optimal threshold Dec 30 11:14:36.110627217 (595h45m35.981710263s) [seg_pos={0 0}, upload=522248]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:14:36.110692742 (595h45m35.981775777s) [upload=522248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:14:36.110698493 (595h45m35.981781528s) [upload=522248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:14:36.398512997 (595h45m36.269596042s) [upload=522248]: Done uploading segments... Dec 30 11:14:36.398591326 (595h45m36.269674371s) [upload=522248]: Upload finished. err: <nil> Duration: 10.028792782s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [....X.X.XXXX.X.XX.X.X.....XX.X........X.X.XX.X.X.X.X...X.XX.] duration: 3.625495439s trace: https://jaeger.dev.storj.io/trace/26102834ad6ebfd2 FAIL (2025-12-30 11:16:07 UTC) orbiter warning: context deadline exceeded Duration: 3.625629844s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X....................X.................X..........X...X..] duration: 35.636974782s trace: https://jaeger.dev.storj.io/trace/7f3400b3387747 FAIL (2025-12-30 11:15:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, Piece ID: OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA) context deadline exceeded; piecestore: (Node ID: 12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, Piece ID: SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ) context deadline exceeded; piecestore: (Node ID: 1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC, Piece ID: 6ZSKVY2RVZAJBTRBKT4RTVQKMJ2BVQDU53I453YBIBBHCZOQCW4A) context deadline exceeded; piecestore: (Node ID: 12YZZnCo54XknuvoBoVtRudXSnoWdNdrn23Lp6Gou3g451BzUwG, Piece ID: 7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ) context deadline exceeded ======LOGS====== s) [noise=true , address=145.79.227.18:4038 , piece_id=K6PIAQ572FP7PVR5G3BRZWSGVYK7XTXG6Z4KBZZ3G5A732BOLS4A, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.892668738 (595h43m54.763751783s) [noise=true , address=145.79.227.40:4079 , piece_id=43LPLVACHLV3VIUR7YZ4RGRCG65FGVSYM5EOPOSG6W6YUSL6OLPQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893082295 (595h43m54.76416534s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893484291 (595h43m54.764567336s) [noise=true , address=77.90.31.12:4613 , piece_id=OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.893955759 (595h43m54.765038794s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.894536003 (595h43m54.765619048s) [noise=true , address=145.79.227.44:4226 , piece_id=JZ5CAXU3XK56URASDAYQHQOCON7SCFEECMSM4NJ7O33M647UWAZQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.894858427 (595h43m54.765941462s) [noise=true , address=64.49.13.155:4920 , piece_id=UE53XZKQ5VW5CDBMS7SF3R2FMA3FVNUQA2RLMFMYIIHPDDSNXTZQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.895172897 (595h43m54.766255942s) [noise=true , address=145.79.227.33:4886 , piece_id=GYC5TKM6I5QRBZJCJVRO35ITEGFZWRXEUTA3V76Y65CZ7ZZEDZSQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.896682591 (595h43m54.767765626s) [noise=true , address=145.79.227.29:4953 , piece_id=VTIGSELICMISVDGASJP4W6Z4IOIALQKRPZEULAODHJL4JQHUU7RA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.897870683 (595h43m54.768953728s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.899025873 (595h43m54.770108918s) [noise=true , address=145.79.227.30:4792 , piece_id=T7WHB25ZUMXYNNPCPLQOKLGUPLTQB3XQJ6ZCE3XRXDOYJWJHXFLA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.900208054 (595h43m54.771291099s) [noise=true , address=77.90.31.18:4827 , piece_id=RXOZKEZUKJA4GXKQUZ6ITP2ZW7TPDDXMF7VINT36HGCQNCDCCB2Q, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.901469476 (595h43m54.772552521s) [noise=true , address=77.90.31.15:4706 , piece_id=7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.902861116 (595h43m54.773944161s) [noise=true , address=145.79.227.17:4987 , piece_id=JNZPAZIFJMCQWJELT26MLRIF46UUIYRYD3EKVFX42VIAXYQU4B5Q, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.904040151 (595h43m54.775123196s) [noise=true , address=77.90.31.16:4743 , piece_id=SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:54.905225688 (595h43m54.776308723s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Uploading piece... Dec 30 11:12:56.312272123 (595h43m56.183355158s) [upload=522223]: Next returned nil segment Dec 30 11:12:56.312288104 (595h43m56.183371139s) [upload=522223]: Waiting for error group managing segments... Dec 30 11:12:56.674443063 (595h43m56.545526118s) [noise=true , address=64.49.13.155:4920 , piece_id=UE53XZKQ5VW5CDBMS7SF3R2FMA3FVNUQA2RLMFMYIIHPDDSNXTZQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.748148810 (595h43m56.619231855s) [noise=true , address=145.79.227.17:4987 , piece_id=JNZPAZIFJMCQWJELT26MLRIF46UUIYRYD3EKVFX42VIAXYQU4B5Q, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.748294467 (595h43m56.619377502s) [noise=true , address=145.79.227.18:4038 , piece_id=K6PIAQ572FP7PVR5G3BRZWSGVYK7XTXG6Z4KBZZ3G5A732BOLS4A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.898059835 (595h43m56.76914288s) [noise=true , address=145.79.227.3:4166 , piece_id=GJBVBRV7YZE3GTXEKBW7LJYW35EXK4SDEVIFNZMETDKCZOXCIWKA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.936973813 (595h43m56.808056848s) [noise=true , address=77.90.31.21:4893 , piece_id=6ZSKVY2RVZAJBTRBKT4RTVQKMJ2BVQDU53I453YBIBBHCZOQCW4A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.961395065 (595h43m56.83247811s) [noise=true , address=77.90.31.18:4827 , piece_id=RXOZKEZUKJA4GXKQUZ6ITP2ZW7TPDDXMF7VINT36HGCQNCDCCB2Q, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:56.967043209 (595h43m56.838126254s) [noise=true , address=77.90.31.15:4706 , piece_id=7WVKHHAQFPRNKKPQOOG44I7CAGGHROASSWM4CVHCVZ2GTIWI5PTQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.059782038 (595h43m56.930865073s) [noise=true , address=77.90.31.16:4743 , piece_id=SUUQDV4FCDXDELPP6VPQWIRBUMXKL33FLNPAO2TFF5FGVWPV7ORQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.692145558 (595h43m57.563228633s) [noise=true , address=145.79.227.30:4792 , piece_id=T7WHB25ZUMXYNNPCPLQOKLGUPLTQB3XQJ6ZCE3XRXDOYJWJHXFLA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.705427259 (595h43m57.576510304s) [noise=true , address=145.79.227.33:4886 , piece_id=GYC5TKM6I5QRBZJCJVRO35ITEGFZWRXEUTA3V76Y65CZ7ZZEDZSQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.745163051 (595h43m57.616246096s) [noise=true , address=145.79.227.40:4079 , piece_id=43LPLVACHLV3VIUR7YZ4RGRCG65FGVSYM5EOPOSG6W6YUSL6OLPQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.776805663 (595h43m57.647888708s) [noise=true , address=145.79.227.44:4226 , piece_id=JZ5CAXU3XK56URASDAYQHQOCON7SCFEECMSM4NJ7O33M647UWAZQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.859064686 (595h43m57.730147731s) [noise=true , address=145.79.227.29:4953 , piece_id=VTIGSELICMISVDGASJP4W6Z4IOIALQKRPZEULAODHJL4JQHUU7RA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.882701444 (595h43m57.753784479s) [noise=true , address=77.90.31.12:4613 , piece_id=OEUT6F4QC3NGITBZQJGRBNLQGVWQY4ZLWAFRXSJ5M7NR7L6LOXQA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: <nil> Dec 30 11:12:57.882722825 (595h43m57.75380587s) [seg_pos={0 0}, upload=522223]: Segment reached optimal threshold of 14 pieces. Dec 30 11:12:57.882954816 (595h43m57.754037851s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.882964956 (595h43m57.754048011s) [noise=true , address=77.90.31.11:4575 , piece_id=M43L2PYFGR3CUKKYDH7CQJS6FFM4W2MULHBT7KF6ANEQMQZM7UFQ, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883110403 (595h43m57.754193448s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.883123828 (595h43m57.754206883s) [noise=true , address=145.79.227.43:4176 , piece_id=6JS4GUSGM5KPFSGOPF22IPWW6644FLRTDJFPAMPPQTEJEP2IDNIQ, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883237274 (595h43m57.754320339s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37214->64.49.13.169:4706: use of closed network connection Dec 30 11:12:57.883248406 (595h43m57.754331451s) [noise=true , address=64.49.13.169:4706 , piece_id=LFUEIKJDKZKCPCQKKRAA5NB2JVOS2AIE7EACHQEXWD4ZZ25OLRVA, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883390406 (595h43m57.754473451s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 30 11:12:57.883415915 (595h43m57.75449896s) [noise=true , address=145.79.227.34:4919 , piece_id=S4HTIMEJ3GEDWDTR5YIH42LVPY3QC2MX6POMXWWAKJWDMRQIVQ3A, seg_pos={0 0}, upload=522223]: Long tail cancellation due to optimal threshold Dec 30 11:12:57.883463475 (595h43m57.75454652s) [seg_pos={0 0}, upload=522223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:12:57.883549108 (595h43m57.754632153s) [upload=522223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:12:57.883554960 (595h43m57.754638005s) [upload=522223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:12:58.175520738 (595h43m58.046603773s) [upload=522223]: Done uploading segments... Dec 30 11:12:58.175538943 (595h43m58.046621978s) [upload=522223]: Upload finished. err: <nil> Duration: 35.636993498s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..XX.XX........X.......X..XX..X.X.X..........X....X.......X.] duration: 5.763629114s trace: https://jaeger.dev.storj.io/trace/a49828f90b13fd8 FAIL (2025-12-30 11:15:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 11:14:33.461039771 (595h45m33.332122806s) [upload=522248]: Starting upload Dec 30 11:14:33.461095928 (595h45m33.332178973s) [upload=522248]: Done starting upload Dec 30 11:14:33.461295408 (595h45m33.332378443s) [upload=522248]: Uploading segments... Dec 30 11:14:33.461322659 (595h45m33.332405694s) [upload=522248]: Got next segment. Inline: false Dec 30 11:14:33.461350833 (595h45m33.332433868s) [upload=522248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:14:33.461355021 (595h45m33.332438056s) [upload=522248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:14:33.705675815 (595h45m33.57675886s) [seg_pos={0 0}, upload=522248]: Begin upload segment... Dec 30 11:14:33.705758453 (595h45m33.576841488s) [seg_pos={0 0}, upload=522248]: Done begin upload segment. Dec 30 11:14:33.705857572 (595h45m33.576940607s) [noise=true , address=145.79.227.23:4547 , piece_id=QO5SRA4RTQKGIHEGPMLPF2LR2DZXP5CK2YFCLLEM3O2GEC6DL2YA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.705925792 (595h45m33.577008827s) [noise=true , address=145.79.227.27:4691 , piece_id=GOF6BW4ODXGBAQ55TDNJ3PVJI64XG5QMYHQOG52CKIH3KDEK32EQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.706575799 (595h45m33.577658834s) [noise=true , address=145.79.227.6:4259 , piece_id=DVAN4J53QGMGLOH6JISZR5PXYXVPUTEBZZ6DLED2CSOZZISUKFZA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.707044542 (595h45m33.578127587s) [noise=true , address=145.79.227.8:4329 , piece_id=TH5EDBQZHM6ESLMCJ42O6XOSU64PBBNOZTOOX6E5RDDCEDYTGADA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.707390461 (595h45m33.578473506s) [noise=true , address=145.79.227.16:4909 , piece_id=T6ZPOEFKW2GLVIAITLVMD2G2IJYOPKSHOIBCMDDIN35ULBBQ4DHA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.707721300 (595h45m33.578804335s) [noise=true , address=45.91.116.13:4154 , piece_id=DZ5GVQPL46KIQULZEH3FDN5SOU6FCMYDZIKHMT4PFPJA7EJDP2EQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.708023466 (595h45m33.579106511s) [noise=true , address=45.91.116.10:4084 , piece_id=MTWPAPMA3RX5JEXYBH2LVA3WLMYMTMORASURQA2VOGUGNREZZ5WA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.708443075 (595h45m33.57952612s) [noise=true , address=145.79.227.3:4150 , piece_id=5CMFYAYA422V32FMO3BSY2ODQI4HH7AYV3ILRNY3RZNAZWTLCHCA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.708815064 (595h45m33.579898099s) [noise=true , address=145.79.227.18:4007 , piece_id=VV3UB7JX4P2ZR6NLHC4B3H365AFR6EKRVHNC7LOQCKCA4E6JA3JQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.709249871 (595h45m33.580332906s) [noise=true , address=145.79.227.17:4972 , piece_id=O2AAGDAEPPKGT2CNO4UEI7Z2WCVTRCKU2EZ35VDZO5WMMWW2O6GA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.709600409 (595h45m33.580683454s) [noise=true , address=145.79.227.22:4510 , piece_id=Q655IZID2S4J2DAH3FCQCODY46P7JWEJZR32F4IZU6EFGCQTH5JQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.709713263 (595h45m33.580796308s) [noise=true , address=77.90.31.14:4663 , piece_id=7EENYELJEIRDKLCP5IV7577GMSX6NEG6C2AUALMZCC2VRJFLEOVQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.710473391 (595h45m33.581556426s) [noise=true , address=145.79.227.21:4475 , piece_id=XQZCYSQSULXPOHBPJIGOXI6Y5EA4JROGADJBNV67MLRBSIGWJP6Q, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.710786547 (595h45m33.581869592s) [noise=true , address=145.79.227.11:4729 , piece_id=JIVN2ZYMIQ6MJVZEKMDXJP2L5HRKMHFI4QMW7OIG3UGA5I7733FQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.711048306 (595h45m33.582131341s) [noise=true , address=45.91.116.12:4227 , piece_id=T5BZCYEWYKZAEVXQ3LP2WQRSISMQBNAMVHP7OQC2EDGNCU2Y7RBQ, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:33.711492452 (595h45m33.582575487s) [noise=true , address=45.91.116.14:4190 , piece_id=UNKLEBU74YRKKF56N44F3BBBBJZYW2KSS7625WODMDFEVBODUTFA, seg_pos={0 0}, upload=522248]: Uploading piece... Dec 30 11:14:35.380454405 (595h45m35.25153744s) [upload=522248]: Next returned nil segment Dec 30 11:14:35.380482839 (595h45m35.251565874s) [upload=522248]: Waiting for error group managing segments... Dec 30 11:14:35.687683897 (595h45m35.558766932s) [noise=true , address=145.79.227.22:4510 , piece_id=Q655IZID2S4J2DAH3FCQCODY46P7JWEJZR32F4IZU6EFGCQTH5JQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.688923037 (595h45m35.560006082s) [noise=true , address=145.79.227.11:4729 , piece_id=JIVN2ZYMIQ6MJVZEKMDXJP2L5HRKMHFI4QMW7OIG3UGA5I7733FQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.689502951 (595h45m35.560585986s) [noise=true , address=145.79.227.8:4329 , piece_id=TH5EDBQZHM6ESLMCJ42O6XOSU64PBBNOZTOOX6E5RDDCEDYTGADA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.698663163 (595h45m35.569746198s) [noise=true , address=145.79.227.23:4547 , piece_id=QO5SRA4RTQKGIHEGPMLPF2LR2DZXP5CK2YFCLLEM3O2GEC6DL2YA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.698721875 (595h45m35.56980492s) [noise=true , address=145.79.227.21:4475 , piece_id=XQZCYSQSULXPOHBPJIGOXI6Y5EA4JROGADJBNV67MLRBSIGWJP6Q, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.708251722 (595h45m35.579334757s) [noise=true , address=145.79.227.3:4150 , piece_id=5CMFYAYA422V32FMO3BSY2ODQI4HH7AYV3ILRNY3RZNAZWTLCHCA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.709504758 (595h45m35.580587793s) [noise=true , address=145.79.227.18:4007 , piece_id=VV3UB7JX4P2ZR6NLHC4B3H365AFR6EKRVHNC7LOQCKCA4E6JA3JQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.714229585 (595h45m35.58531262s) [noise=true , address=145.79.227.6:4259 , piece_id=DVAN4J53QGMGLOH6JISZR5PXYXVPUTEBZZ6DLED2CSOZZISUKFZA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.727102366 (595h45m35.598185401s) [noise=true , address=145.79.227.17:4972 , piece_id=O2AAGDAEPPKGT2CNO4UEI7Z2WCVTRCKU2EZ35VDZO5WMMWW2O6GA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.777227693 (595h45m35.648310728s) [noise=true , address=145.79.227.16:4909 , piece_id=T6ZPOEFKW2GLVIAITLVMD2G2IJYOPKSHOIBCMDDIN35ULBBQ4DHA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.786515268 (595h45m35.657598303s) [noise=true , address=145.79.227.27:4691 , piece_id=GOF6BW4ODXGBAQ55TDNJ3PVJI64XG5QMYHQOG52CKIH3KDEK32EQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:35.988591282 (595h45m35.859674317s) [noise=true , address=77.90.31.14:4663 , piece_id=7EENYELJEIRDKLCP5IV7577GMSX6NEG6C2AUALMZCC2VRJFLEOVQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:36.110014360 (595h45m35.981097395s) [noise=true , address=45.91.116.12:4227 , piece_id=T5BZCYEWYKZAEVXQ3LP2WQRSISMQBNAMVHP7OQC2EDGNCU2Y7RBQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: <nil> Dec 30 11:14:36.110035300 (595h45m35.981118345s) [seg_pos={0 0}, upload=522248]: Segment reached optimal threshold of 13 pieces. Dec 30 11:14:36.110230792 (595h45m35.981313837s) [noise=true , address=45.91.116.10:4084 , piece_id=MTWPAPMA3RX5JEXYBH2LVA3WLMYMTMORASURQA2VOGUGNREZZ5WA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 11:14:36.110237966 (595h45m35.981321012s) [noise=true , address=45.91.116.10:4084 , piece_id=MTWPAPMA3RX5JEXYBH2LVA3WLMYMTMORASURQA2VOGUGNREZZ5WA, seg_pos={0 0}, upload=522248]: Long tail cancellation due to optimal threshold Dec 30 11:14:36.110515695 (595h45m35.98159873s) [noise=true , address=45.91.116.14:4190 , piece_id=UNKLEBU74YRKKF56N44F3BBBBJZYW2KSS7625WODMDFEVBODUTFA, seg_pos={0 0}, upload=522248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 11:14:36.110533479 (595h45m35.981616524s) [noise=true , address=45.91.116.14:4190 , piece_id=UNKLEBU74YRKKF56N44F3BBBBJZYW2KSS7625WODMDFEVBODUTFA, seg_pos={0 0}, upload=522248]: Long tail cancellation due to optimal threshold Dec 30 11:14:36.110573946 (595h45m35.981656981s) [noise=true , address=45.91.116.13:4154 , piece_id=DZ5GVQPL46KIQULZEH3FDN5SOU6FCMYDZIKHMT4PFPJA7EJDP2EQ, seg_pos={0 0}, upload=522248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 30 11:14:36.110582261 (595h45m35.981665306s) [noise=true , address=45.91.116.13:4154 , piece_id=DZ5GVQPL46KIQULZEH3FDN5SOU6FCMYDZIKHMT4PFPJA7EJDP2EQ, seg_pos={0 0}, upload=522248]: Long tail cancellation due to optimal threshold Dec 30 11:14:36.110627217 (595h45m35.981710263s) [seg_pos={0 0}, upload=522248]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:14:36.110692742 (595h45m35.981775777s) [upload=522248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:14:36.110698493 (595h45m35.981781528s) [upload=522248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:14:36.398512997 (595h45m36.269596042s) [upload=522248]: Done uploading segments... Dec 30 11:14:36.398591326 (595h45m36.269674371s) [upload=522248]: Upload finished. err: <nil> Duration: 5.763648421s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)