Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 859826 reports (Page 11773 of 17197)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.47885171s trace: https://jaeger.dev.storj.io/trace/30a119e66e6df16c FAIL (2025-12-07 15:45:44 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.478877768s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X........X......X...] duration: 20.628380344s trace: https://jaeger.dev.storj.io/trace/76649d75ca4b1745 FAIL (2025-12-07 15:45:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ) context deadline exceeded ======LOGS====== Dec 7 15:42:52.075141337 (48h13m51.946224372s) [upload=42393]: Starting upload Dec 7 15:42:52.075200600 (48h13m51.946283635s) [upload=42393]: Done starting upload Dec 7 15:42:52.075355735 (48h13m51.94643877s) [upload=42393]: Uploading segments... Dec 7 15:42:52.075629006 (48h13m51.946712051s) [upload=42393]: Got next segment. Inline: false Dec 7 15:42:52.075673029 (48h13m51.946756074s) [upload=42393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 15:42:52.075679713 (48h13m51.946762758s) [upload=42393]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 15:42:52.646860140 (48h13m52.517943185s) [seg_pos={0 0}, upload=42393]: Begin upload segment... Dec 7 15:42:52.646921907 (48h13m52.518004953s) [seg_pos={0 0}, upload=42393]: Done begin upload segment. Dec 7 15:42:52.646965200 (48h13m52.518048235s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.646982794 (48h13m52.518065829s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.647887376 (48h13m52.518970411s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.648182538 (48h13m52.519265583s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.648286496 (48h13m52.519369541s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.648641683 (48h13m52.519724718s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.648678743 (48h13m52.519761778s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.649046653 (48h13m52.520129698s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.649472815 (48h13m52.52055586s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.649940345 (48h13m52.521023471s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.650032391 (48h13m52.521115436s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.650436259 (48h13m52.521519304s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.650739918 (48h13m52.521822963s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.651153676 (48h13m52.522236721s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.651585758 (48h13m52.522668803s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.651989718 (48h13m52.523072763s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:54.440983928 (48h13m54.312066973s) [upload=42393]: Next returned nil segment Dec 7 15:42:54.440996322 (48h13m54.312079367s) [upload=42393]: Waiting for error group managing segments... Dec 7 15:42:54.795710406 (48h13m54.666793451s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.796230045 (48h13m54.66731309s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.804219258 (48h13m54.675302303s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.804402307 (48h13m54.675485352s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.820310880 (48h13m54.691393915s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.824943301 (48h13m54.696026346s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.828361219 (48h13m54.699444264s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.857739461 (48h13m54.728822506s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.865338421 (48h13m54.736421456s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:55.108668117 (48h13m54.979751152s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:55.204450035 (48h13m55.07553308s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:55.495245134 (48h13m55.366328179s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:55.528102311 (48h13m55.399185356s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:55.528123241 (48h13m55.399206287s) [seg_pos={0 0}, upload=42393]: Segment reached optimal threshold of 13 pieces. Dec 7 15:42:55.528359992 (48h13m55.399443028s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 7 15:42:55.528366525 (48h13m55.39944956s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold Dec 7 15:42:55.528488256 (48h13m55.399571301s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection Dec 7 15:42:55.528505860 (48h13m55.399588895s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold Dec 7 15:42:55.528539404 (48h13m55.399622449s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 7 15:42:55.528548120 (48h13m55.399631155s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold Dec 7 15:42:55.528619466 (48h13m55.399702501s) [seg_pos={0 0}, upload=42393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 15:42:55.528697785 (48h13m55.39978083s) [upload=42393]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 15:42:55.528718635 (48h13m55.39980168s) [upload=42393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 15:42:55.843997748 (48h13m55.715080783s) [upload=42393]: Done uploading segments... Dec 7 15:42:55.844020370 (48h13m55.715103415s) [upload=42393]: Upload finished. err: <nil> Duration: 20.628417324s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.699971085s trace: https://jaeger.dev.storj.io/trace/6aa77812f5688c1c FAIL (2025-12-07 15:45:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments... Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments... Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 5.699998227s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.....X......X.X.................................X.....X...] duration: 13.523312426s trace: https://jaeger.dev.storj.io/trace/646b41ef1d2f671b FAIL (2025-12-07 15:45:30 UTC) orbiter warning: context deadline exceeded Duration: 13.523343315s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.709151173s trace: https://jaeger.dev.storj.io/trace/6aa77812f5688c1c FAIL (2025-12-07 15:45:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments... Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments... Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 5.709173565s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.....X......X.X.................................X.....X..] duration: 15.788879078s trace: https://jaeger.dev.storj.io/trace/646b41ef1d2f671b FAIL (2025-12-07 15:45:00 UTC) orbiter warning: context deadline exceeded Duration: 15.788894617s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [XXXXX..X.XXXXXX..X.X....X....X.X.........XX..X.....XXX..X.X.] duration: 3.470985958s trace: https://jaeger.dev.storj.io/trace/5c097de2d5934d90 FAIL (2025-12-07 15:44:56 UTC) orbiter warning: context deadline exceeded Duration: 3.471018138s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 4.301094265s trace: https://jaeger.dev.storj.io/trace/30a119e66e6df16c FAIL (2025-12-07 15:44:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.301119373s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X.....X......X.X.................................X.....X.] duration: 17.490104218s trace: https://jaeger.dev.storj.io/trace/646b41ef1d2f671b FAIL (2025-12-07 15:44:44 UTC) orbiter warning: context deadline exceeded Duration: 17.490132171s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X........X......X..] duration: 8.301898719s trace: https://jaeger.dev.storj.io/trace/76649d75ca4b1745 FAIL (2025-12-07 15:44:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ) context deadline exceeded ======LOGS====== Dec 7 15:42:52.075141337 (48h13m51.946224372s) [upload=42393]: Starting upload Dec 7 15:42:52.075200600 (48h13m51.946283635s) [upload=42393]: Done starting upload Dec 7 15:42:52.075355735 (48h13m51.94643877s) [upload=42393]: Uploading segments... Dec 7 15:42:52.075629006 (48h13m51.946712051s) [upload=42393]: Got next segment. Inline: false Dec 7 15:42:52.075673029 (48h13m51.946756074s) [upload=42393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 15:42:52.075679713 (48h13m51.946762758s) [upload=42393]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 15:42:52.646860140 (48h13m52.517943185s) [seg_pos={0 0}, upload=42393]: Begin upload segment... Dec 7 15:42:52.646921907 (48h13m52.518004953s) [seg_pos={0 0}, upload=42393]: Done begin upload segment. Dec 7 15:42:52.646965200 (48h13m52.518048235s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.646982794 (48h13m52.518065829s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.647887376 (48h13m52.518970411s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.648182538 (48h13m52.519265583s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.648286496 (48h13m52.519369541s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.648641683 (48h13m52.519724718s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.648678743 (48h13m52.519761778s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.649046653 (48h13m52.520129698s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.649472815 (48h13m52.52055586s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.649940345 (48h13m52.521023471s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.650032391 (48h13m52.521115436s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.650436259 (48h13m52.521519304s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.650739918 (48h13m52.521822963s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.651153676 (48h13m52.522236721s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.651585758 (48h13m52.522668803s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.651989718 (48h13m52.523072763s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:54.440983928 (48h13m54.312066973s) [upload=42393]: Next returned nil segment Dec 7 15:42:54.440996322 (48h13m54.312079367s) [upload=42393]: Waiting for error group managing segments... Dec 7 15:42:54.795710406 (48h13m54.666793451s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.796230045 (48h13m54.66731309s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.804219258 (48h13m54.675302303s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.804402307 (48h13m54.675485352s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.820310880 (48h13m54.691393915s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.824943301 (48h13m54.696026346s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.828361219 (48h13m54.699444264s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.857739461 (48h13m54.728822506s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.865338421 (48h13m54.736421456s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:55.108668117 (48h13m54.979751152s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:55.204450035 (48h13m55.07553308s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:55.495245134 (48h13m55.366328179s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:55.528102311 (48h13m55.399185356s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:55.528123241 (48h13m55.399206287s) [seg_pos={0 0}, upload=42393]: Segment reached optimal threshold of 13 pieces. Dec 7 15:42:55.528359992 (48h13m55.399443028s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 7 15:42:55.528366525 (48h13m55.39944956s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold Dec 7 15:42:55.528488256 (48h13m55.399571301s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection Dec 7 15:42:55.528505860 (48h13m55.399588895s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold Dec 7 15:42:55.528539404 (48h13m55.399622449s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 7 15:42:55.528548120 (48h13m55.399631155s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold Dec 7 15:42:55.528619466 (48h13m55.399702501s) [seg_pos={0 0}, upload=42393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 15:42:55.528697785 (48h13m55.39978083s) [upload=42393]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 15:42:55.528718635 (48h13m55.39980168s) [upload=42393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 15:42:55.843997748 (48h13m55.715080783s) [upload=42393]: Done uploading segments... Dec 7 15:42:55.844020370 (48h13m55.715103415s) [upload=42393]: Upload finished. err: <nil> Duration: 8.301921863s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.73452118s trace: https://jaeger.dev.storj.io/trace/6aa77812f5688c1c FAIL (2025-12-07 15:44:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments... Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments... Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 5.73454713s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 8.131363921s trace: https://jaeger.dev.storj.io/trace/30a119e66e6df16c FAIL (2025-12-07 15:43:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.131391594s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X.....X......X.X.................................X.....X] duration: 30.001473222s trace: https://jaeger.dev.storj.io/trace/646b41ef1d2f671b FAIL (2025-12-07 15:43:57 UTC) orbiter warning: context deadline exceeded Duration: 30.001492289s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.XXXXX..X.XXXXXX..X.X....X....X.X.........XX..X.....XXX..X.X] duration: 5.00106136s trace: https://jaeger.dev.storj.io/trace/5c097de2d5934d90 FAIL (2025-12-07 15:43:48 UTC) orbiter warning: context deadline exceeded Duration: 5.00108666s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X........X......X.] duration: 23.212659125s trace: https://jaeger.dev.storj.io/trace/76649d75ca4b1745 FAIL (2025-12-07 15:43:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ) context deadline exceeded ======LOGS====== Dec 7 15:42:52.075141337 (48h13m51.946224372s) [upload=42393]: Starting upload Dec 7 15:42:52.075200600 (48h13m51.946283635s) [upload=42393]: Done starting upload Dec 7 15:42:52.075355735 (48h13m51.94643877s) [upload=42393]: Uploading segments... Dec 7 15:42:52.075629006 (48h13m51.946712051s) [upload=42393]: Got next segment. Inline: false Dec 7 15:42:52.075673029 (48h13m51.946756074s) [upload=42393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 15:42:52.075679713 (48h13m51.946762758s) [upload=42393]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 15:42:52.646860140 (48h13m52.517943185s) [seg_pos={0 0}, upload=42393]: Begin upload segment... Dec 7 15:42:52.646921907 (48h13m52.518004953s) [seg_pos={0 0}, upload=42393]: Done begin upload segment. Dec 7 15:42:52.646965200 (48h13m52.518048235s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.646982794 (48h13m52.518065829s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.647887376 (48h13m52.518970411s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.648182538 (48h13m52.519265583s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.648286496 (48h13m52.519369541s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.648641683 (48h13m52.519724718s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.648678743 (48h13m52.519761778s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.649046653 (48h13m52.520129698s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.649472815 (48h13m52.52055586s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.649940345 (48h13m52.521023471s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.650032391 (48h13m52.521115436s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.650436259 (48h13m52.521519304s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.650739918 (48h13m52.521822963s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.651153676 (48h13m52.522236721s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.651585758 (48h13m52.522668803s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.651989718 (48h13m52.523072763s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:54.440983928 (48h13m54.312066973s) [upload=42393]: Next returned nil segment Dec 7 15:42:54.440996322 (48h13m54.312079367s) [upload=42393]: Waiting for error group managing segments... Dec 7 15:42:54.795710406 (48h13m54.666793451s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.796230045 (48h13m54.66731309s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.804219258 (48h13m54.675302303s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.804402307 (48h13m54.675485352s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.820310880 (48h13m54.691393915s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.824943301 (48h13m54.696026346s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.828361219 (48h13m54.699444264s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.857739461 (48h13m54.728822506s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.865338421 (48h13m54.736421456s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:55.108668117 (48h13m54.979751152s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:55.204450035 (48h13m55.07553308s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:55.495245134 (48h13m55.366328179s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:55.528102311 (48h13m55.399185356s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:55.528123241 (48h13m55.399206287s) [seg_pos={0 0}, upload=42393]: Segment reached optimal threshold of 13 pieces. Dec 7 15:42:55.528359992 (48h13m55.399443028s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 7 15:42:55.528366525 (48h13m55.39944956s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold Dec 7 15:42:55.528488256 (48h13m55.399571301s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection Dec 7 15:42:55.528505860 (48h13m55.399588895s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold Dec 7 15:42:55.528539404 (48h13m55.399622449s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 7 15:42:55.528548120 (48h13m55.399631155s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold Dec 7 15:42:55.528619466 (48h13m55.399702501s) [seg_pos={0 0}, upload=42393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 15:42:55.528697785 (48h13m55.39978083s) [upload=42393]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 15:42:55.528718635 (48h13m55.39980168s) [upload=42393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 15:42:55.843997748 (48h13m55.715080783s) [upload=42393]: Done uploading segments... Dec 7 15:42:55.844020370 (48h13m55.715103415s) [upload=42393]: Upload finished. err: <nil> Duration: 23.212677641s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.805752538s trace: https://jaeger.dev.storj.io/trace/6aa77812f5688c1c FAIL (2025-12-07 15:43:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments... Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments... Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 5.805778648s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.780010175s trace: https://jaeger.dev.storj.io/trace/30a119e66e6df16c FAIL (2025-12-07 15:43:25 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.780039172s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X........X......X] duration: 30.002940376s trace: https://jaeger.dev.storj.io/trace/76649d75ca4b1745 FAIL (2025-12-07 15:43:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ) context deadline exceeded ======LOGS====== Dec 7 15:42:52.075141337 (48h13m51.946224372s) [upload=42393]: Starting upload Dec 7 15:42:52.075200600 (48h13m51.946283635s) [upload=42393]: Done starting upload Dec 7 15:42:52.075355735 (48h13m51.94643877s) [upload=42393]: Uploading segments... Dec 7 15:42:52.075629006 (48h13m51.946712051s) [upload=42393]: Got next segment. Inline: false Dec 7 15:42:52.075673029 (48h13m51.946756074s) [upload=42393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 15:42:52.075679713 (48h13m51.946762758s) [upload=42393]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 15:42:52.646860140 (48h13m52.517943185s) [seg_pos={0 0}, upload=42393]: Begin upload segment... Dec 7 15:42:52.646921907 (48h13m52.518004953s) [seg_pos={0 0}, upload=42393]: Done begin upload segment. Dec 7 15:42:52.646965200 (48h13m52.518048235s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.646982794 (48h13m52.518065829s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.647887376 (48h13m52.518970411s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.648182538 (48h13m52.519265583s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.648286496 (48h13m52.519369541s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.648641683 (48h13m52.519724718s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.648678743 (48h13m52.519761778s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.649046653 (48h13m52.520129698s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.649472815 (48h13m52.52055586s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.649940345 (48h13m52.521023471s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.650032391 (48h13m52.521115436s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.650436259 (48h13m52.521519304s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.650739918 (48h13m52.521822963s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.651153676 (48h13m52.522236721s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.651585758 (48h13m52.522668803s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:52.651989718 (48h13m52.523072763s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Uploading piece... Dec 7 15:42:54.440983928 (48h13m54.312066973s) [upload=42393]: Next returned nil segment Dec 7 15:42:54.440996322 (48h13m54.312079367s) [upload=42393]: Waiting for error group managing segments... Dec 7 15:42:54.795710406 (48h13m54.666793451s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.796230045 (48h13m54.66731309s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.804219258 (48h13m54.675302303s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.804402307 (48h13m54.675485352s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.820310880 (48h13m54.691393915s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.824943301 (48h13m54.696026346s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.828361219 (48h13m54.699444264s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.857739461 (48h13m54.728822506s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:54.865338421 (48h13m54.736421456s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:55.108668117 (48h13m54.979751152s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:55.204450035 (48h13m55.07553308s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:55.495245134 (48h13m55.366328179s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:55.528102311 (48h13m55.399185356s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil> Dec 7 15:42:55.528123241 (48h13m55.399206287s) [seg_pos={0 0}, upload=42393]: Segment reached optimal threshold of 13 pieces. Dec 7 15:42:55.528359992 (48h13m55.399443028s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 7 15:42:55.528366525 (48h13m55.39944956s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold Dec 7 15:42:55.528488256 (48h13m55.399571301s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection Dec 7 15:42:55.528505860 (48h13m55.399588895s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold Dec 7 15:42:55.528539404 (48h13m55.399622449s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 7 15:42:55.528548120 (48h13m55.399631155s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold Dec 7 15:42:55.528619466 (48h13m55.399702501s) [seg_pos={0 0}, upload=42393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 15:42:55.528697785 (48h13m55.39978083s) [upload=42393]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 15:42:55.528718635 (48h13m55.39980168s) [upload=42393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 15:42:55.843997748 (48h13m55.715080783s) [upload=42393]: Done uploading segments... Dec 7 15:42:55.844020370 (48h13m55.715103415s) [upload=42393]: Upload finished. err: <nil> Duration: 30.002965273s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.706255543s trace: https://jaeger.dev.storj.io/trace/6aa77812f5688c1c FAIL (2025-12-07 15:43:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments... Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments... Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 5.706299516s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.....X......X.X.................................X.....] duration: 14.436280384s trace: https://jaeger.dev.storj.io/trace/57fa7c8cbec72018 FAIL (2025-12-07 15:43:01 UTC) orbiter warning: context deadline exceeded Duration: 14.436311272s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [..XXXXX..X.XXXXXX..X.X....X....X.X.........XX..X.....XXX..X.] duration: 3.63595503s trace: https://jaeger.dev.storj.io/trace/29e20d9e03ae265f FAIL (2025-12-07 15:42:33 UTC) orbiter warning: context deadline exceeded Duration: 3.636013s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 7.873981992s trace: https://jaeger.dev.storj.io/trace/30a119e66e6df16c FAIL (2025-12-07 15:42:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.874038228s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X.....X......X.X.................................X....] duration: 16.132195798s trace: https://jaeger.dev.storj.io/trace/57fa7c8cbec72018 FAIL (2025-12-07 15:42:16 UTC) orbiter warning: context deadline exceeded Duration: 16.132216186s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X........X......] duration: 5.990485372s trace: https://jaeger.dev.storj.io/trace/396da084cc259886 FAIL (2025-12-07 15:42:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 15:37:46.609384273 (48h8m46.480467318s) [upload=42321]: Starting upload Dec 7 15:37:46.609447142 (48h8m46.480530177s) [upload=42321]: Done starting upload Dec 7 15:37:46.609581388 (48h8m46.480664433s) [upload=42321]: Uploading segments... Dec 7 15:37:46.609752243 (48h8m46.480835278s) [upload=42321]: Got next segment. Inline: false Dec 7 15:37:46.609829109 (48h8m46.480912144s) [upload=42321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 15:37:46.609834741 (48h8m46.480917776s) [upload=42321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 15:37:47.125547557 (48h8m46.996630592s) [seg_pos={0 0}, upload=42321]: Begin upload segment... Dec 7 15:37:47.125614395 (48h8m46.99669743s) [seg_pos={0 0}, upload=42321]: Done begin upload segment. Dec 7 15:37:47.125644111 (48h8m46.996727156s) [noise=true , address=145.79.227.41:4086 , piece_id=UAWGBWBMOIAWXIO7BAIIZ7RA536ERIWIHMS7KAUD3E23ALEEWC2Q, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.125697462 (48h8m46.996780507s) [noise=true , address=145.79.227.42:4122 , piece_id=FP3VXDDCVYBBIS4D3EY233ZISVBOFEBLSNAOPX2773SUATZAGSTA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.126407134 (48h8m46.997490179s) [noise=true , address=77.90.31.17:4771 , piece_id=DMZ57PS4PONWG4ENMILDTTI3I2CSQNXI2ZX3PM7Y7RJDI7HEHOLA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.126853805 (48h8m46.99793685s) [noise=true , address=145.79.227.26:4654 , piece_id=EQSMRY5SVYGTITHH3UF5P5SGXBZNDITJY753CD7VY423T66USQSQ, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.127212217 (48h8m46.998295262s) [noise=true , address=145.79.227.22:4510 , piece_id=TW5DNNPLSQZK2ZLQJ6I4JWSX2GRBUZWPBVCARPSK7PHG4QQQYGAA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.127866543 (48h8m46.998949588s) [noise=true , address=145.79.227.20:4438 , piece_id=M5RLXAM6ZXDIIIIFGYU5SRNZ5AI7QRP24IPV4C77C3CYZECSMBRA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.128359091 (48h8m46.999442126s) [noise=true , address=145.79.227.29:4942 , piece_id=ES2FUPLIBQ4U6BDHBC6QRVQG53UIGWS5DPO2DPF6R3FKVPF3WXAQ, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.128780403 (48h8m46.999863448s) [noise=true , address=145.79.227.44:4193 , piece_id=YHKVOHIRGBDBOTJ2T7UGW7B7UI4D7B2OC5HZA6ENTS6MYS3RBS6Q, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.129285576 (48h8m47.000368621s) [noise=true , address=145.79.227.41:4085 , piece_id=YNLMM5NYRB47MBWOUZYPYUINN5HVCB2A5YNW72YFRRQQ6YGACR3Q, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.129727607 (48h8m47.000810652s) [noise=true , address=145.79.227.27:4690 , piece_id=QI5QLCVLP6Z5DNCZZXUFGVAGHWKVC32UA5YOIMO42M5CEKZ4Q2MA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.130221537 (48h8m47.001304582s) [noise=true , address=77.90.31.10:4520 , piece_id=Q32XVBGWDN53N5FKRVU2Z354FJUVORB46PS5WAST5VV4CAKKPSTA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.131412536 (48h8m47.002495591s) [noise=true , address=145.79.227.23:4546 , piece_id=KZYS7BKWUN5D6RZJG4ER3HIFMWAZTR63L3M2RUBPBXZH33YF74TA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.131599662 (48h8m47.002682697s) [noise=true , address=145.79.227.16:4911 , piece_id=FHUIUP32L5L4JDXVSKU66V45L22KGWYWSPBBO4LJ6U4IWNKI3ICA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.133038882 (48h8m47.004121927s) [noise=true , address=145.79.227.25:4619 , piece_id=ALJOALLXBTWEHOELG6YET5HW34EOBKDOA7AL3T2KGLKQ253Z7MXA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.138418296 (48h8m47.009501351s) [noise=true , address=145.79.227.34:4905 , piece_id=ND7JKTGJH3DZBSCC2PQXM5Z57MVWEVUSMIFXCO73W7CC54QJY66A, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.141044165 (48h8m47.01212722s) [noise=true , address=77.90.31.15:4698 , piece_id=Z7NPTLYBD5DZVLVF3JKLJHLRMW7QFWCO7ZXGIKEWCMEM5LRV4ZLA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:48.836748043 (48h8m48.707831078s) [upload=42321]: Next returned nil segment Dec 7 15:37:48.836760767 (48h8m48.707843812s) [upload=42321]: Waiting for error group managing segments... Dec 7 15:37:49.226822230 (48h8m49.097905325s) [noise=true , address=77.90.31.17:4771 , piece_id=DMZ57PS4PONWG4ENMILDTTI3I2CSQNXI2ZX3PM7Y7RJDI7HEHOLA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.229191832 (48h8m49.100274867s) [noise=true , address=77.90.31.10:4520 , piece_id=Q32XVBGWDN53N5FKRVU2Z354FJUVORB46PS5WAST5VV4CAKKPSTA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.269025022 (48h8m49.140108057s) [noise=true , address=145.79.227.27:4690 , piece_id=QI5QLCVLP6Z5DNCZZXUFGVAGHWKVC32UA5YOIMO42M5CEKZ4Q2MA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.270472808 (48h8m49.141555883s) [noise=true , address=145.79.227.42:4122 , piece_id=FP3VXDDCVYBBIS4D3EY233ZISVBOFEBLSNAOPX2773SUATZAGSTA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.271570349 (48h8m49.142653404s) [noise=true , address=145.79.227.16:4911 , piece_id=FHUIUP32L5L4JDXVSKU66V45L22KGWYWSPBBO4LJ6U4IWNKI3ICA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.281506679 (48h8m49.152589714s) [noise=true , address=145.79.227.25:4619 , piece_id=ALJOALLXBTWEHOELG6YET5HW34EOBKDOA7AL3T2KGLKQ253Z7MXA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.288241994 (48h8m49.15932503s) [noise=true , address=145.79.227.34:4905 , piece_id=ND7JKTGJH3DZBSCC2PQXM5Z57MVWEVUSMIFXCO73W7CC54QJY66A, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.292810224 (48h8m49.163893279s) [noise=true , address=145.79.227.41:4085 , piece_id=YNLMM5NYRB47MBWOUZYPYUINN5HVCB2A5YNW72YFRRQQ6YGACR3Q, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.306469012 (48h8m49.177552057s) [noise=true , address=77.90.31.15:4698 , piece_id=Z7NPTLYBD5DZVLVF3JKLJHLRMW7QFWCO7ZXGIKEWCMEM5LRV4ZLA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.331878357 (48h8m49.202961402s) [noise=true , address=145.79.227.29:4942 , piece_id=ES2FUPLIBQ4U6BDHBC6QRVQG53UIGWS5DPO2DPF6R3FKVPF3WXAQ, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.361459515 (48h8m49.23254255s) [noise=true , address=145.79.227.23:4546 , piece_id=KZYS7BKWUN5D6RZJG4ER3HIFMWAZTR63L3M2RUBPBXZH33YF74TA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.372131507 (48h8m49.243214552s) [noise=true , address=145.79.227.41:4086 , piece_id=UAWGBWBMOIAWXIO7BAIIZ7RA536ERIWIHMS7KAUD3E23ALEEWC2Q, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.618376213 (48h8m49.489459258s) [noise=true , address=145.79.227.22:4510 , piece_id=TW5DNNPLSQZK2ZLQJ6I4JWSX2GRBUZWPBVCARPSK7PHG4QQQYGAA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.618396953 (48h8m49.489479998s) [seg_pos={0 0}, upload=42321]: Segment reached optimal threshold of 13 pieces. Dec 7 15:37:49.618588087 (48h8m49.489671122s) [noise=true , address=145.79.227.20:4438 , piece_id=M5RLXAM6ZXDIIIIFGYU5SRNZ5AI7QRP24IPV4C77C3CYZECSMBRA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 7 15:37:49.618595662 (48h8m49.489678697s) [noise=true , address=145.79.227.20:4438 , piece_id=M5RLXAM6ZXDIIIIFGYU5SRNZ5AI7QRP24IPV4C77C3CYZECSMBRA, seg_pos={0 0}, upload=42321]: Long tail cancellation due to optimal threshold Dec 7 15:37:49.618605130 (48h8m49.489688165s) [noise=true , address=145.79.227.26:4654 , piece_id=EQSMRY5SVYGTITHH3UF5P5SGXBZNDITJY753CD7VY423T66USQSQ, seg_pos={0 0}, upload=42321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53114->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53114->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53114->145.79.227.26:4654: use of closed network connection Dec 7 15:37:49.618610640 (48h8m49.489693675s) [noise=true , address=145.79.227.26:4654 , piece_id=EQSMRY5SVYGTITHH3UF5P5SGXBZNDITJY753CD7VY423T66USQSQ, seg_pos={0 0}, upload=42321]: Long tail cancellation due to optimal threshold Dec 7 15:37:49.618719667 (48h8m49.489802702s) [noise=true , address=145.79.227.44:4193 , piece_id=YHKVOHIRGBDBOTJ2T7UGW7B7UI4D7B2OC5HZA6ENTS6MYS3RBS6Q, seg_pos={0 0}, upload=42321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 7 15:37:49.618725498 (48h8m49.489808533s) [noise=true , address=145.79.227.44:4193 , piece_id=YHKVOHIRGBDBOTJ2T7UGW7B7UI4D7B2OC5HZA6ENTS6MYS3RBS6Q, seg_pos={0 0}, upload=42321]: Long tail cancellation due to optimal threshold Dec 7 15:37:49.618742772 (48h8m49.489825807s) [seg_pos={0 0}, upload=42321]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 15:37:49.618805391 (48h8m49.489888436s) [upload=42321]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 15:37:49.618809639 (48h8m49.489892684s) [upload=42321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 15:37:49.936557632 (48h8m49.807640677s) [upload=42321]: Done uploading segments... Dec 7 15:37:49.936575908 (48h8m49.807658953s) [upload=42321]: Upload finished. err: <nil> Duration: 5.990516441s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.681378994s trace: https://jaeger.dev.storj.io/trace/6aa77812f5688c1c FAIL (2025-12-07 15:42:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments... Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments... Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 5.681413078s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.132919806s trace: https://jaeger.dev.storj.io/trace/30a119e66e6df16c FAIL (2025-12-07 15:41:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.1329473s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X........X.....] duration: 5.737744678s trace: https://jaeger.dev.storj.io/trace/396da084cc259886 FAIL (2025-12-07 15:41:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 15:37:46.609384273 (48h8m46.480467318s) [upload=42321]: Starting upload Dec 7 15:37:46.609447142 (48h8m46.480530177s) [upload=42321]: Done starting upload Dec 7 15:37:46.609581388 (48h8m46.480664433s) [upload=42321]: Uploading segments... Dec 7 15:37:46.609752243 (48h8m46.480835278s) [upload=42321]: Got next segment. Inline: false Dec 7 15:37:46.609829109 (48h8m46.480912144s) [upload=42321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 15:37:46.609834741 (48h8m46.480917776s) [upload=42321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 15:37:47.125547557 (48h8m46.996630592s) [seg_pos={0 0}, upload=42321]: Begin upload segment... Dec 7 15:37:47.125614395 (48h8m46.99669743s) [seg_pos={0 0}, upload=42321]: Done begin upload segment. Dec 7 15:37:47.125644111 (48h8m46.996727156s) [noise=true , address=145.79.227.41:4086 , piece_id=UAWGBWBMOIAWXIO7BAIIZ7RA536ERIWIHMS7KAUD3E23ALEEWC2Q, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.125697462 (48h8m46.996780507s) [noise=true , address=145.79.227.42:4122 , piece_id=FP3VXDDCVYBBIS4D3EY233ZISVBOFEBLSNAOPX2773SUATZAGSTA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.126407134 (48h8m46.997490179s) [noise=true , address=77.90.31.17:4771 , piece_id=DMZ57PS4PONWG4ENMILDTTI3I2CSQNXI2ZX3PM7Y7RJDI7HEHOLA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.126853805 (48h8m46.99793685s) [noise=true , address=145.79.227.26:4654 , piece_id=EQSMRY5SVYGTITHH3UF5P5SGXBZNDITJY753CD7VY423T66USQSQ, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.127212217 (48h8m46.998295262s) [noise=true , address=145.79.227.22:4510 , piece_id=TW5DNNPLSQZK2ZLQJ6I4JWSX2GRBUZWPBVCARPSK7PHG4QQQYGAA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.127866543 (48h8m46.998949588s) [noise=true , address=145.79.227.20:4438 , piece_id=M5RLXAM6ZXDIIIIFGYU5SRNZ5AI7QRP24IPV4C77C3CYZECSMBRA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.128359091 (48h8m46.999442126s) [noise=true , address=145.79.227.29:4942 , piece_id=ES2FUPLIBQ4U6BDHBC6QRVQG53UIGWS5DPO2DPF6R3FKVPF3WXAQ, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.128780403 (48h8m46.999863448s) [noise=true , address=145.79.227.44:4193 , piece_id=YHKVOHIRGBDBOTJ2T7UGW7B7UI4D7B2OC5HZA6ENTS6MYS3RBS6Q, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.129285576 (48h8m47.000368621s) [noise=true , address=145.79.227.41:4085 , piece_id=YNLMM5NYRB47MBWOUZYPYUINN5HVCB2A5YNW72YFRRQQ6YGACR3Q, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.129727607 (48h8m47.000810652s) [noise=true , address=145.79.227.27:4690 , piece_id=QI5QLCVLP6Z5DNCZZXUFGVAGHWKVC32UA5YOIMO42M5CEKZ4Q2MA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.130221537 (48h8m47.001304582s) [noise=true , address=77.90.31.10:4520 , piece_id=Q32XVBGWDN53N5FKRVU2Z354FJUVORB46PS5WAST5VV4CAKKPSTA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.131412536 (48h8m47.002495591s) [noise=true , address=145.79.227.23:4546 , piece_id=KZYS7BKWUN5D6RZJG4ER3HIFMWAZTR63L3M2RUBPBXZH33YF74TA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.131599662 (48h8m47.002682697s) [noise=true , address=145.79.227.16:4911 , piece_id=FHUIUP32L5L4JDXVSKU66V45L22KGWYWSPBBO4LJ6U4IWNKI3ICA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.133038882 (48h8m47.004121927s) [noise=true , address=145.79.227.25:4619 , piece_id=ALJOALLXBTWEHOELG6YET5HW34EOBKDOA7AL3T2KGLKQ253Z7MXA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.138418296 (48h8m47.009501351s) [noise=true , address=145.79.227.34:4905 , piece_id=ND7JKTGJH3DZBSCC2PQXM5Z57MVWEVUSMIFXCO73W7CC54QJY66A, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.141044165 (48h8m47.01212722s) [noise=true , address=77.90.31.15:4698 , piece_id=Z7NPTLYBD5DZVLVF3JKLJHLRMW7QFWCO7ZXGIKEWCMEM5LRV4ZLA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:48.836748043 (48h8m48.707831078s) [upload=42321]: Next returned nil segment Dec 7 15:37:48.836760767 (48h8m48.707843812s) [upload=42321]: Waiting for error group managing segments... Dec 7 15:37:49.226822230 (48h8m49.097905325s) [noise=true , address=77.90.31.17:4771 , piece_id=DMZ57PS4PONWG4ENMILDTTI3I2CSQNXI2ZX3PM7Y7RJDI7HEHOLA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.229191832 (48h8m49.100274867s) [noise=true , address=77.90.31.10:4520 , piece_id=Q32XVBGWDN53N5FKRVU2Z354FJUVORB46PS5WAST5VV4CAKKPSTA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.269025022 (48h8m49.140108057s) [noise=true , address=145.79.227.27:4690 , piece_id=QI5QLCVLP6Z5DNCZZXUFGVAGHWKVC32UA5YOIMO42M5CEKZ4Q2MA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.270472808 (48h8m49.141555883s) [noise=true , address=145.79.227.42:4122 , piece_id=FP3VXDDCVYBBIS4D3EY233ZISVBOFEBLSNAOPX2773SUATZAGSTA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.271570349 (48h8m49.142653404s) [noise=true , address=145.79.227.16:4911 , piece_id=FHUIUP32L5L4JDXVSKU66V45L22KGWYWSPBBO4LJ6U4IWNKI3ICA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.281506679 (48h8m49.152589714s) [noise=true , address=145.79.227.25:4619 , piece_id=ALJOALLXBTWEHOELG6YET5HW34EOBKDOA7AL3T2KGLKQ253Z7MXA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.288241994 (48h8m49.15932503s) [noise=true , address=145.79.227.34:4905 , piece_id=ND7JKTGJH3DZBSCC2PQXM5Z57MVWEVUSMIFXCO73W7CC54QJY66A, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.292810224 (48h8m49.163893279s) [noise=true , address=145.79.227.41:4085 , piece_id=YNLMM5NYRB47MBWOUZYPYUINN5HVCB2A5YNW72YFRRQQ6YGACR3Q, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.306469012 (48h8m49.177552057s) [noise=true , address=77.90.31.15:4698 , piece_id=Z7NPTLYBD5DZVLVF3JKLJHLRMW7QFWCO7ZXGIKEWCMEM5LRV4ZLA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.331878357 (48h8m49.202961402s) [noise=true , address=145.79.227.29:4942 , piece_id=ES2FUPLIBQ4U6BDHBC6QRVQG53UIGWS5DPO2DPF6R3FKVPF3WXAQ, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.361459515 (48h8m49.23254255s) [noise=true , address=145.79.227.23:4546 , piece_id=KZYS7BKWUN5D6RZJG4ER3HIFMWAZTR63L3M2RUBPBXZH33YF74TA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.372131507 (48h8m49.243214552s) [noise=true , address=145.79.227.41:4086 , piece_id=UAWGBWBMOIAWXIO7BAIIZ7RA536ERIWIHMS7KAUD3E23ALEEWC2Q, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.618376213 (48h8m49.489459258s) [noise=true , address=145.79.227.22:4510 , piece_id=TW5DNNPLSQZK2ZLQJ6I4JWSX2GRBUZWPBVCARPSK7PHG4QQQYGAA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.618396953 (48h8m49.489479998s) [seg_pos={0 0}, upload=42321]: Segment reached optimal threshold of 13 pieces. Dec 7 15:37:49.618588087 (48h8m49.489671122s) [noise=true , address=145.79.227.20:4438 , piece_id=M5RLXAM6ZXDIIIIFGYU5SRNZ5AI7QRP24IPV4C77C3CYZECSMBRA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 7 15:37:49.618595662 (48h8m49.489678697s) [noise=true , address=145.79.227.20:4438 , piece_id=M5RLXAM6ZXDIIIIFGYU5SRNZ5AI7QRP24IPV4C77C3CYZECSMBRA, seg_pos={0 0}, upload=42321]: Long tail cancellation due to optimal threshold Dec 7 15:37:49.618605130 (48h8m49.489688165s) [noise=true , address=145.79.227.26:4654 , piece_id=EQSMRY5SVYGTITHH3UF5P5SGXBZNDITJY753CD7VY423T66USQSQ, seg_pos={0 0}, upload=42321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53114->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53114->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53114->145.79.227.26:4654: use of closed network connection Dec 7 15:37:49.618610640 (48h8m49.489693675s) [noise=true , address=145.79.227.26:4654 , piece_id=EQSMRY5SVYGTITHH3UF5P5SGXBZNDITJY753CD7VY423T66USQSQ, seg_pos={0 0}, upload=42321]: Long tail cancellation due to optimal threshold Dec 7 15:37:49.618719667 (48h8m49.489802702s) [noise=true , address=145.79.227.44:4193 , piece_id=YHKVOHIRGBDBOTJ2T7UGW7B7UI4D7B2OC5HZA6ENTS6MYS3RBS6Q, seg_pos={0 0}, upload=42321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 7 15:37:49.618725498 (48h8m49.489808533s) [noise=true , address=145.79.227.44:4193 , piece_id=YHKVOHIRGBDBOTJ2T7UGW7B7UI4D7B2OC5HZA6ENTS6MYS3RBS6Q, seg_pos={0 0}, upload=42321]: Long tail cancellation due to optimal threshold Dec 7 15:37:49.618742772 (48h8m49.489825807s) [seg_pos={0 0}, upload=42321]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 15:37:49.618805391 (48h8m49.489888436s) [upload=42321]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 15:37:49.618809639 (48h8m49.489892684s) [upload=42321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 15:37:49.936557632 (48h8m49.807640677s) [upload=42321]: Done uploading segments... Dec 7 15:37:49.936575908 (48h8m49.807658953s) [upload=42321]: Upload finished. err: <nil> Duration: 5.737793s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [...XXXXX..X.XXXXXX..X.X....X....X.X.........XX..X.....XXX..X] duration: 5.000966518s trace: https://jaeger.dev.storj.io/trace/29e20d9e03ae265f FAIL (2025-12-07 15:41:30 UTC) orbiter warning: context deadline exceeded Duration: 5.000988931s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X.....X......X.X.................................X...] duration: 18.41729221s trace: https://jaeger.dev.storj.io/trace/57fa7c8cbec72018 FAIL (2025-12-07 15:41:12 UTC) orbiter warning: context deadline exceeded Duration: 18.417320144s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [....XXXXX..X.XXXXXX..X.X....X....X.X.........XX..X.....XXX..] duration: 3.454090968s trace: https://jaeger.dev.storj.io/trace/19945b440849ff03 FAIL (2025-12-07 15:41:12 UTC) orbiter warning: context deadline exceeded Duration: 3.454118079s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.819140508s trace: https://jaeger.dev.storj.io/trace/6aa77812f5688c1c FAIL (2025-12-07 15:41:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments... Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments... Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 5.819157551s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X....XXXXX..X.XXXXXX..X.X....X....X.X.........XX..X.....XXX.] duration: 3.484376756s trace: https://jaeger.dev.storj.io/trace/19945b440849ff03 FAIL (2025-12-07 15:40:45 UTC) orbiter warning: context deadline exceeded Duration: 3.48440054s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X........X....] duration: 5.701801142s trace: https://jaeger.dev.storj.io/trace/396da084cc259886 FAIL (2025-12-07 15:40:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 15:37:46.609384273 (48h8m46.480467318s) [upload=42321]: Starting upload Dec 7 15:37:46.609447142 (48h8m46.480530177s) [upload=42321]: Done starting upload Dec 7 15:37:46.609581388 (48h8m46.480664433s) [upload=42321]: Uploading segments... Dec 7 15:37:46.609752243 (48h8m46.480835278s) [upload=42321]: Got next segment. Inline: false Dec 7 15:37:46.609829109 (48h8m46.480912144s) [upload=42321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 15:37:46.609834741 (48h8m46.480917776s) [upload=42321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 15:37:47.125547557 (48h8m46.996630592s) [seg_pos={0 0}, upload=42321]: Begin upload segment... Dec 7 15:37:47.125614395 (48h8m46.99669743s) [seg_pos={0 0}, upload=42321]: Done begin upload segment. Dec 7 15:37:47.125644111 (48h8m46.996727156s) [noise=true , address=145.79.227.41:4086 , piece_id=UAWGBWBMOIAWXIO7BAIIZ7RA536ERIWIHMS7KAUD3E23ALEEWC2Q, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.125697462 (48h8m46.996780507s) [noise=true , address=145.79.227.42:4122 , piece_id=FP3VXDDCVYBBIS4D3EY233ZISVBOFEBLSNAOPX2773SUATZAGSTA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.126407134 (48h8m46.997490179s) [noise=true , address=77.90.31.17:4771 , piece_id=DMZ57PS4PONWG4ENMILDTTI3I2CSQNXI2ZX3PM7Y7RJDI7HEHOLA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.126853805 (48h8m46.99793685s) [noise=true , address=145.79.227.26:4654 , piece_id=EQSMRY5SVYGTITHH3UF5P5SGXBZNDITJY753CD7VY423T66USQSQ, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.127212217 (48h8m46.998295262s) [noise=true , address=145.79.227.22:4510 , piece_id=TW5DNNPLSQZK2ZLQJ6I4JWSX2GRBUZWPBVCARPSK7PHG4QQQYGAA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.127866543 (48h8m46.998949588s) [noise=true , address=145.79.227.20:4438 , piece_id=M5RLXAM6ZXDIIIIFGYU5SRNZ5AI7QRP24IPV4C77C3CYZECSMBRA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.128359091 (48h8m46.999442126s) [noise=true , address=145.79.227.29:4942 , piece_id=ES2FUPLIBQ4U6BDHBC6QRVQG53UIGWS5DPO2DPF6R3FKVPF3WXAQ, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.128780403 (48h8m46.999863448s) [noise=true , address=145.79.227.44:4193 , piece_id=YHKVOHIRGBDBOTJ2T7UGW7B7UI4D7B2OC5HZA6ENTS6MYS3RBS6Q, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.129285576 (48h8m47.000368621s) [noise=true , address=145.79.227.41:4085 , piece_id=YNLMM5NYRB47MBWOUZYPYUINN5HVCB2A5YNW72YFRRQQ6YGACR3Q, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.129727607 (48h8m47.000810652s) [noise=true , address=145.79.227.27:4690 , piece_id=QI5QLCVLP6Z5DNCZZXUFGVAGHWKVC32UA5YOIMO42M5CEKZ4Q2MA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.130221537 (48h8m47.001304582s) [noise=true , address=77.90.31.10:4520 , piece_id=Q32XVBGWDN53N5FKRVU2Z354FJUVORB46PS5WAST5VV4CAKKPSTA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.131412536 (48h8m47.002495591s) [noise=true , address=145.79.227.23:4546 , piece_id=KZYS7BKWUN5D6RZJG4ER3HIFMWAZTR63L3M2RUBPBXZH33YF74TA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.131599662 (48h8m47.002682697s) [noise=true , address=145.79.227.16:4911 , piece_id=FHUIUP32L5L4JDXVSKU66V45L22KGWYWSPBBO4LJ6U4IWNKI3ICA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.133038882 (48h8m47.004121927s) [noise=true , address=145.79.227.25:4619 , piece_id=ALJOALLXBTWEHOELG6YET5HW34EOBKDOA7AL3T2KGLKQ253Z7MXA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.138418296 (48h8m47.009501351s) [noise=true , address=145.79.227.34:4905 , piece_id=ND7JKTGJH3DZBSCC2PQXM5Z57MVWEVUSMIFXCO73W7CC54QJY66A, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.141044165 (48h8m47.01212722s) [noise=true , address=77.90.31.15:4698 , piece_id=Z7NPTLYBD5DZVLVF3JKLJHLRMW7QFWCO7ZXGIKEWCMEM5LRV4ZLA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:48.836748043 (48h8m48.707831078s) [upload=42321]: Next returned nil segment Dec 7 15:37:48.836760767 (48h8m48.707843812s) [upload=42321]: Waiting for error group managing segments... Dec 7 15:37:49.226822230 (48h8m49.097905325s) [noise=true , address=77.90.31.17:4771 , piece_id=DMZ57PS4PONWG4ENMILDTTI3I2CSQNXI2ZX3PM7Y7RJDI7HEHOLA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.229191832 (48h8m49.100274867s) [noise=true , address=77.90.31.10:4520 , piece_id=Q32XVBGWDN53N5FKRVU2Z354FJUVORB46PS5WAST5VV4CAKKPSTA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.269025022 (48h8m49.140108057s) [noise=true , address=145.79.227.27:4690 , piece_id=QI5QLCVLP6Z5DNCZZXUFGVAGHWKVC32UA5YOIMO42M5CEKZ4Q2MA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.270472808 (48h8m49.141555883s) [noise=true , address=145.79.227.42:4122 , piece_id=FP3VXDDCVYBBIS4D3EY233ZISVBOFEBLSNAOPX2773SUATZAGSTA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.271570349 (48h8m49.142653404s) [noise=true , address=145.79.227.16:4911 , piece_id=FHUIUP32L5L4JDXVSKU66V45L22KGWYWSPBBO4LJ6U4IWNKI3ICA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.281506679 (48h8m49.152589714s) [noise=true , address=145.79.227.25:4619 , piece_id=ALJOALLXBTWEHOELG6YET5HW34EOBKDOA7AL3T2KGLKQ253Z7MXA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.288241994 (48h8m49.15932503s) [noise=true , address=145.79.227.34:4905 , piece_id=ND7JKTGJH3DZBSCC2PQXM5Z57MVWEVUSMIFXCO73W7CC54QJY66A, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.292810224 (48h8m49.163893279s) [noise=true , address=145.79.227.41:4085 , piece_id=YNLMM5NYRB47MBWOUZYPYUINN5HVCB2A5YNW72YFRRQQ6YGACR3Q, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.306469012 (48h8m49.177552057s) [noise=true , address=77.90.31.15:4698 , piece_id=Z7NPTLYBD5DZVLVF3JKLJHLRMW7QFWCO7ZXGIKEWCMEM5LRV4ZLA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.331878357 (48h8m49.202961402s) [noise=true , address=145.79.227.29:4942 , piece_id=ES2FUPLIBQ4U6BDHBC6QRVQG53UIGWS5DPO2DPF6R3FKVPF3WXAQ, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.361459515 (48h8m49.23254255s) [noise=true , address=145.79.227.23:4546 , piece_id=KZYS7BKWUN5D6RZJG4ER3HIFMWAZTR63L3M2RUBPBXZH33YF74TA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.372131507 (48h8m49.243214552s) [noise=true , address=145.79.227.41:4086 , piece_id=UAWGBWBMOIAWXIO7BAIIZ7RA536ERIWIHMS7KAUD3E23ALEEWC2Q, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.618376213 (48h8m49.489459258s) [noise=true , address=145.79.227.22:4510 , piece_id=TW5DNNPLSQZK2ZLQJ6I4JWSX2GRBUZWPBVCARPSK7PHG4QQQYGAA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.618396953 (48h8m49.489479998s) [seg_pos={0 0}, upload=42321]: Segment reached optimal threshold of 13 pieces. Dec 7 15:37:49.618588087 (48h8m49.489671122s) [noise=true , address=145.79.227.20:4438 , piece_id=M5RLXAM6ZXDIIIIFGYU5SRNZ5AI7QRP24IPV4C77C3CYZECSMBRA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 7 15:37:49.618595662 (48h8m49.489678697s) [noise=true , address=145.79.227.20:4438 , piece_id=M5RLXAM6ZXDIIIIFGYU5SRNZ5AI7QRP24IPV4C77C3CYZECSMBRA, seg_pos={0 0}, upload=42321]: Long tail cancellation due to optimal threshold Dec 7 15:37:49.618605130 (48h8m49.489688165s) [noise=true , address=145.79.227.26:4654 , piece_id=EQSMRY5SVYGTITHH3UF5P5SGXBZNDITJY753CD7VY423T66USQSQ, seg_pos={0 0}, upload=42321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53114->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53114->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53114->145.79.227.26:4654: use of closed network connection Dec 7 15:37:49.618610640 (48h8m49.489693675s) [noise=true , address=145.79.227.26:4654 , piece_id=EQSMRY5SVYGTITHH3UF5P5SGXBZNDITJY753CD7VY423T66USQSQ, seg_pos={0 0}, upload=42321]: Long tail cancellation due to optimal threshold Dec 7 15:37:49.618719667 (48h8m49.489802702s) [noise=true , address=145.79.227.44:4193 , piece_id=YHKVOHIRGBDBOTJ2T7UGW7B7UI4D7B2OC5HZA6ENTS6MYS3RBS6Q, seg_pos={0 0}, upload=42321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 7 15:37:49.618725498 (48h8m49.489808533s) [noise=true , address=145.79.227.44:4193 , piece_id=YHKVOHIRGBDBOTJ2T7UGW7B7UI4D7B2OC5HZA6ENTS6MYS3RBS6Q, seg_pos={0 0}, upload=42321]: Long tail cancellation due to optimal threshold Dec 7 15:37:49.618742772 (48h8m49.489825807s) [seg_pos={0 0}, upload=42321]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 15:37:49.618805391 (48h8m49.489888436s) [upload=42321]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 15:37:49.618809639 (48h8m49.489892684s) [upload=42321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 15:37:49.936557632 (48h8m49.807640677s) [upload=42321]: Done uploading segments... Dec 7 15:37:49.936575908 (48h8m49.807658953s) [upload=42321]: Upload finished. err: <nil> Duration: 5.701843672s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.X....XXXXX..X.XXXXXX..X.X....X....X.X.........XX..X.....XXX] duration: 5.00095808s trace: https://jaeger.dev.storj.io/trace/19945b440849ff03 FAIL (2025-12-07 15:40:32 UTC) orbiter warning: context deadline exceeded Duration: 5.000985622s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.353458892s trace: https://jaeger.dev.storj.io/trace/30a119e66e6df16c FAIL (2025-12-07 15:40:29 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.353488286s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X........X...] duration: 5.969062248s trace: https://jaeger.dev.storj.io/trace/396da084cc259886 FAIL (2025-12-07 15:40:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 15:37:46.609384273 (48h8m46.480467318s) [upload=42321]: Starting upload Dec 7 15:37:46.609447142 (48h8m46.480530177s) [upload=42321]: Done starting upload Dec 7 15:37:46.609581388 (48h8m46.480664433s) [upload=42321]: Uploading segments... Dec 7 15:37:46.609752243 (48h8m46.480835278s) [upload=42321]: Got next segment. Inline: false Dec 7 15:37:46.609829109 (48h8m46.480912144s) [upload=42321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 15:37:46.609834741 (48h8m46.480917776s) [upload=42321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 15:37:47.125547557 (48h8m46.996630592s) [seg_pos={0 0}, upload=42321]: Begin upload segment... Dec 7 15:37:47.125614395 (48h8m46.99669743s) [seg_pos={0 0}, upload=42321]: Done begin upload segment. Dec 7 15:37:47.125644111 (48h8m46.996727156s) [noise=true , address=145.79.227.41:4086 , piece_id=UAWGBWBMOIAWXIO7BAIIZ7RA536ERIWIHMS7KAUD3E23ALEEWC2Q, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.125697462 (48h8m46.996780507s) [noise=true , address=145.79.227.42:4122 , piece_id=FP3VXDDCVYBBIS4D3EY233ZISVBOFEBLSNAOPX2773SUATZAGSTA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.126407134 (48h8m46.997490179s) [noise=true , address=77.90.31.17:4771 , piece_id=DMZ57PS4PONWG4ENMILDTTI3I2CSQNXI2ZX3PM7Y7RJDI7HEHOLA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.126853805 (48h8m46.99793685s) [noise=true , address=145.79.227.26:4654 , piece_id=EQSMRY5SVYGTITHH3UF5P5SGXBZNDITJY753CD7VY423T66USQSQ, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.127212217 (48h8m46.998295262s) [noise=true , address=145.79.227.22:4510 , piece_id=TW5DNNPLSQZK2ZLQJ6I4JWSX2GRBUZWPBVCARPSK7PHG4QQQYGAA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.127866543 (48h8m46.998949588s) [noise=true , address=145.79.227.20:4438 , piece_id=M5RLXAM6ZXDIIIIFGYU5SRNZ5AI7QRP24IPV4C77C3CYZECSMBRA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.128359091 (48h8m46.999442126s) [noise=true , address=145.79.227.29:4942 , piece_id=ES2FUPLIBQ4U6BDHBC6QRVQG53UIGWS5DPO2DPF6R3FKVPF3WXAQ, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.128780403 (48h8m46.999863448s) [noise=true , address=145.79.227.44:4193 , piece_id=YHKVOHIRGBDBOTJ2T7UGW7B7UI4D7B2OC5HZA6ENTS6MYS3RBS6Q, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.129285576 (48h8m47.000368621s) [noise=true , address=145.79.227.41:4085 , piece_id=YNLMM5NYRB47MBWOUZYPYUINN5HVCB2A5YNW72YFRRQQ6YGACR3Q, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.129727607 (48h8m47.000810652s) [noise=true , address=145.79.227.27:4690 , piece_id=QI5QLCVLP6Z5DNCZZXUFGVAGHWKVC32UA5YOIMO42M5CEKZ4Q2MA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.130221537 (48h8m47.001304582s) [noise=true , address=77.90.31.10:4520 , piece_id=Q32XVBGWDN53N5FKRVU2Z354FJUVORB46PS5WAST5VV4CAKKPSTA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.131412536 (48h8m47.002495591s) [noise=true , address=145.79.227.23:4546 , piece_id=KZYS7BKWUN5D6RZJG4ER3HIFMWAZTR63L3M2RUBPBXZH33YF74TA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.131599662 (48h8m47.002682697s) [noise=true , address=145.79.227.16:4911 , piece_id=FHUIUP32L5L4JDXVSKU66V45L22KGWYWSPBBO4LJ6U4IWNKI3ICA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.133038882 (48h8m47.004121927s) [noise=true , address=145.79.227.25:4619 , piece_id=ALJOALLXBTWEHOELG6YET5HW34EOBKDOA7AL3T2KGLKQ253Z7MXA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.138418296 (48h8m47.009501351s) [noise=true , address=145.79.227.34:4905 , piece_id=ND7JKTGJH3DZBSCC2PQXM5Z57MVWEVUSMIFXCO73W7CC54QJY66A, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.141044165 (48h8m47.01212722s) [noise=true , address=77.90.31.15:4698 , piece_id=Z7NPTLYBD5DZVLVF3JKLJHLRMW7QFWCO7ZXGIKEWCMEM5LRV4ZLA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:48.836748043 (48h8m48.707831078s) [upload=42321]: Next returned nil segment Dec 7 15:37:48.836760767 (48h8m48.707843812s) [upload=42321]: Waiting for error group managing segments... Dec 7 15:37:49.226822230 (48h8m49.097905325s) [noise=true , address=77.90.31.17:4771 , piece_id=DMZ57PS4PONWG4ENMILDTTI3I2CSQNXI2ZX3PM7Y7RJDI7HEHOLA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.229191832 (48h8m49.100274867s) [noise=true , address=77.90.31.10:4520 , piece_id=Q32XVBGWDN53N5FKRVU2Z354FJUVORB46PS5WAST5VV4CAKKPSTA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.269025022 (48h8m49.140108057s) [noise=true , address=145.79.227.27:4690 , piece_id=QI5QLCVLP6Z5DNCZZXUFGVAGHWKVC32UA5YOIMO42M5CEKZ4Q2MA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.270472808 (48h8m49.141555883s) [noise=true , address=145.79.227.42:4122 , piece_id=FP3VXDDCVYBBIS4D3EY233ZISVBOFEBLSNAOPX2773SUATZAGSTA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.271570349 (48h8m49.142653404s) [noise=true , address=145.79.227.16:4911 , piece_id=FHUIUP32L5L4JDXVSKU66V45L22KGWYWSPBBO4LJ6U4IWNKI3ICA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.281506679 (48h8m49.152589714s) [noise=true , address=145.79.227.25:4619 , piece_id=ALJOALLXBTWEHOELG6YET5HW34EOBKDOA7AL3T2KGLKQ253Z7MXA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.288241994 (48h8m49.15932503s) [noise=true , address=145.79.227.34:4905 , piece_id=ND7JKTGJH3DZBSCC2PQXM5Z57MVWEVUSMIFXCO73W7CC54QJY66A, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.292810224 (48h8m49.163893279s) [noise=true , address=145.79.227.41:4085 , piece_id=YNLMM5NYRB47MBWOUZYPYUINN5HVCB2A5YNW72YFRRQQ6YGACR3Q, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.306469012 (48h8m49.177552057s) [noise=true , address=77.90.31.15:4698 , piece_id=Z7NPTLYBD5DZVLVF3JKLJHLRMW7QFWCO7ZXGIKEWCMEM5LRV4ZLA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.331878357 (48h8m49.202961402s) [noise=true , address=145.79.227.29:4942 , piece_id=ES2FUPLIBQ4U6BDHBC6QRVQG53UIGWS5DPO2DPF6R3FKVPF3WXAQ, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.361459515 (48h8m49.23254255s) [noise=true , address=145.79.227.23:4546 , piece_id=KZYS7BKWUN5D6RZJG4ER3HIFMWAZTR63L3M2RUBPBXZH33YF74TA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.372131507 (48h8m49.243214552s) [noise=true , address=145.79.227.41:4086 , piece_id=UAWGBWBMOIAWXIO7BAIIZ7RA536ERIWIHMS7KAUD3E23ALEEWC2Q, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.618376213 (48h8m49.489459258s) [noise=true , address=145.79.227.22:4510 , piece_id=TW5DNNPLSQZK2ZLQJ6I4JWSX2GRBUZWPBVCARPSK7PHG4QQQYGAA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.618396953 (48h8m49.489479998s) [seg_pos={0 0}, upload=42321]: Segment reached optimal threshold of 13 pieces. Dec 7 15:37:49.618588087 (48h8m49.489671122s) [noise=true , address=145.79.227.20:4438 , piece_id=M5RLXAM6ZXDIIIIFGYU5SRNZ5AI7QRP24IPV4C77C3CYZECSMBRA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 7 15:37:49.618595662 (48h8m49.489678697s) [noise=true , address=145.79.227.20:4438 , piece_id=M5RLXAM6ZXDIIIIFGYU5SRNZ5AI7QRP24IPV4C77C3CYZECSMBRA, seg_pos={0 0}, upload=42321]: Long tail cancellation due to optimal threshold Dec 7 15:37:49.618605130 (48h8m49.489688165s) [noise=true , address=145.79.227.26:4654 , piece_id=EQSMRY5SVYGTITHH3UF5P5SGXBZNDITJY753CD7VY423T66USQSQ, seg_pos={0 0}, upload=42321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53114->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53114->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53114->145.79.227.26:4654: use of closed network connection Dec 7 15:37:49.618610640 (48h8m49.489693675s) [noise=true , address=145.79.227.26:4654 , piece_id=EQSMRY5SVYGTITHH3UF5P5SGXBZNDITJY753CD7VY423T66USQSQ, seg_pos={0 0}, upload=42321]: Long tail cancellation due to optimal threshold Dec 7 15:37:49.618719667 (48h8m49.489802702s) [noise=true , address=145.79.227.44:4193 , piece_id=YHKVOHIRGBDBOTJ2T7UGW7B7UI4D7B2OC5HZA6ENTS6MYS3RBS6Q, seg_pos={0 0}, upload=42321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 7 15:37:49.618725498 (48h8m49.489808533s) [noise=true , address=145.79.227.44:4193 , piece_id=YHKVOHIRGBDBOTJ2T7UGW7B7UI4D7B2OC5HZA6ENTS6MYS3RBS6Q, seg_pos={0 0}, upload=42321]: Long tail cancellation due to optimal threshold Dec 7 15:37:49.618742772 (48h8m49.489825807s) [seg_pos={0 0}, upload=42321]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 15:37:49.618805391 (48h8m49.489888436s) [upload=42321]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 15:37:49.618809639 (48h8m49.489892684s) [upload=42321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 15:37:49.936557632 (48h8m49.807640677s) [upload=42321]: Done uploading segments... Dec 7 15:37:49.936575908 (48h8m49.807658953s) [upload=42321]: Upload finished. err: <nil> Duration: 5.969091965s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X.....X......X.X.................................X..] duration: 16.031171882s trace: https://jaeger.dev.storj.io/trace/57fa7c8cbec72018 FAIL (2025-12-07 15:40:06 UTC) orbiter warning: context deadline exceeded Duration: 16.031189435s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.782393696s trace: https://jaeger.dev.storj.io/trace/6aa77812f5688c1c FAIL (2025-12-07 15:39:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments... Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments... Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 5.782424295s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X.....X......X.X.................................X.] duration: 20.346796717s trace: https://jaeger.dev.storj.io/trace/57fa7c8cbec72018 FAIL (2025-12-07 15:39:49 UTC) orbiter warning: context deadline exceeded Duration: 20.346822707s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X........X..] duration: 6.548663879s trace: https://jaeger.dev.storj.io/trace/396da084cc259886 FAIL (2025-12-07 15:39:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 15:37:46.609384273 (48h8m46.480467318s) [upload=42321]: Starting upload Dec 7 15:37:46.609447142 (48h8m46.480530177s) [upload=42321]: Done starting upload Dec 7 15:37:46.609581388 (48h8m46.480664433s) [upload=42321]: Uploading segments... Dec 7 15:37:46.609752243 (48h8m46.480835278s) [upload=42321]: Got next segment. Inline: false Dec 7 15:37:46.609829109 (48h8m46.480912144s) [upload=42321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 15:37:46.609834741 (48h8m46.480917776s) [upload=42321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 15:37:47.125547557 (48h8m46.996630592s) [seg_pos={0 0}, upload=42321]: Begin upload segment... Dec 7 15:37:47.125614395 (48h8m46.99669743s) [seg_pos={0 0}, upload=42321]: Done begin upload segment. Dec 7 15:37:47.125644111 (48h8m46.996727156s) [noise=true , address=145.79.227.41:4086 , piece_id=UAWGBWBMOIAWXIO7BAIIZ7RA536ERIWIHMS7KAUD3E23ALEEWC2Q, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.125697462 (48h8m46.996780507s) [noise=true , address=145.79.227.42:4122 , piece_id=FP3VXDDCVYBBIS4D3EY233ZISVBOFEBLSNAOPX2773SUATZAGSTA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.126407134 (48h8m46.997490179s) [noise=true , address=77.90.31.17:4771 , piece_id=DMZ57PS4PONWG4ENMILDTTI3I2CSQNXI2ZX3PM7Y7RJDI7HEHOLA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.126853805 (48h8m46.99793685s) [noise=true , address=145.79.227.26:4654 , piece_id=EQSMRY5SVYGTITHH3UF5P5SGXBZNDITJY753CD7VY423T66USQSQ, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.127212217 (48h8m46.998295262s) [noise=true , address=145.79.227.22:4510 , piece_id=TW5DNNPLSQZK2ZLQJ6I4JWSX2GRBUZWPBVCARPSK7PHG4QQQYGAA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.127866543 (48h8m46.998949588s) [noise=true , address=145.79.227.20:4438 , piece_id=M5RLXAM6ZXDIIIIFGYU5SRNZ5AI7QRP24IPV4C77C3CYZECSMBRA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.128359091 (48h8m46.999442126s) [noise=true , address=145.79.227.29:4942 , piece_id=ES2FUPLIBQ4U6BDHBC6QRVQG53UIGWS5DPO2DPF6R3FKVPF3WXAQ, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.128780403 (48h8m46.999863448s) [noise=true , address=145.79.227.44:4193 , piece_id=YHKVOHIRGBDBOTJ2T7UGW7B7UI4D7B2OC5HZA6ENTS6MYS3RBS6Q, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.129285576 (48h8m47.000368621s) [noise=true , address=145.79.227.41:4085 , piece_id=YNLMM5NYRB47MBWOUZYPYUINN5HVCB2A5YNW72YFRRQQ6YGACR3Q, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.129727607 (48h8m47.000810652s) [noise=true , address=145.79.227.27:4690 , piece_id=QI5QLCVLP6Z5DNCZZXUFGVAGHWKVC32UA5YOIMO42M5CEKZ4Q2MA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.130221537 (48h8m47.001304582s) [noise=true , address=77.90.31.10:4520 , piece_id=Q32XVBGWDN53N5FKRVU2Z354FJUVORB46PS5WAST5VV4CAKKPSTA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.131412536 (48h8m47.002495591s) [noise=true , address=145.79.227.23:4546 , piece_id=KZYS7BKWUN5D6RZJG4ER3HIFMWAZTR63L3M2RUBPBXZH33YF74TA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.131599662 (48h8m47.002682697s) [noise=true , address=145.79.227.16:4911 , piece_id=FHUIUP32L5L4JDXVSKU66V45L22KGWYWSPBBO4LJ6U4IWNKI3ICA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.133038882 (48h8m47.004121927s) [noise=true , address=145.79.227.25:4619 , piece_id=ALJOALLXBTWEHOELG6YET5HW34EOBKDOA7AL3T2KGLKQ253Z7MXA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.138418296 (48h8m47.009501351s) [noise=true , address=145.79.227.34:4905 , piece_id=ND7JKTGJH3DZBSCC2PQXM5Z57MVWEVUSMIFXCO73W7CC54QJY66A, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.141044165 (48h8m47.01212722s) [noise=true , address=77.90.31.15:4698 , piece_id=Z7NPTLYBD5DZVLVF3JKLJHLRMW7QFWCO7ZXGIKEWCMEM5LRV4ZLA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:48.836748043 (48h8m48.707831078s) [upload=42321]: Next returned nil segment Dec 7 15:37:48.836760767 (48h8m48.707843812s) [upload=42321]: Waiting for error group managing segments... Dec 7 15:37:49.226822230 (48h8m49.097905325s) [noise=true , address=77.90.31.17:4771 , piece_id=DMZ57PS4PONWG4ENMILDTTI3I2CSQNXI2ZX3PM7Y7RJDI7HEHOLA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.229191832 (48h8m49.100274867s) [noise=true , address=77.90.31.10:4520 , piece_id=Q32XVBGWDN53N5FKRVU2Z354FJUVORB46PS5WAST5VV4CAKKPSTA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.269025022 (48h8m49.140108057s) [noise=true , address=145.79.227.27:4690 , piece_id=QI5QLCVLP6Z5DNCZZXUFGVAGHWKVC32UA5YOIMO42M5CEKZ4Q2MA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.270472808 (48h8m49.141555883s) [noise=true , address=145.79.227.42:4122 , piece_id=FP3VXDDCVYBBIS4D3EY233ZISVBOFEBLSNAOPX2773SUATZAGSTA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.271570349 (48h8m49.142653404s) [noise=true , address=145.79.227.16:4911 , piece_id=FHUIUP32L5L4JDXVSKU66V45L22KGWYWSPBBO4LJ6U4IWNKI3ICA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.281506679 (48h8m49.152589714s) [noise=true , address=145.79.227.25:4619 , piece_id=ALJOALLXBTWEHOELG6YET5HW34EOBKDOA7AL3T2KGLKQ253Z7MXA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.288241994 (48h8m49.15932503s) [noise=true , address=145.79.227.34:4905 , piece_id=ND7JKTGJH3DZBSCC2PQXM5Z57MVWEVUSMIFXCO73W7CC54QJY66A, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.292810224 (48h8m49.163893279s) [noise=true , address=145.79.227.41:4085 , piece_id=YNLMM5NYRB47MBWOUZYPYUINN5HVCB2A5YNW72YFRRQQ6YGACR3Q, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.306469012 (48h8m49.177552057s) [noise=true , address=77.90.31.15:4698 , piece_id=Z7NPTLYBD5DZVLVF3JKLJHLRMW7QFWCO7ZXGIKEWCMEM5LRV4ZLA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.331878357 (48h8m49.202961402s) [noise=true , address=145.79.227.29:4942 , piece_id=ES2FUPLIBQ4U6BDHBC6QRVQG53UIGWS5DPO2DPF6R3FKVPF3WXAQ, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.361459515 (48h8m49.23254255s) [noise=true , address=145.79.227.23:4546 , piece_id=KZYS7BKWUN5D6RZJG4ER3HIFMWAZTR63L3M2RUBPBXZH33YF74TA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.372131507 (48h8m49.243214552s) [noise=true , address=145.79.227.41:4086 , piece_id=UAWGBWBMOIAWXIO7BAIIZ7RA536ERIWIHMS7KAUD3E23ALEEWC2Q, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.618376213 (48h8m49.489459258s) [noise=true , address=145.79.227.22:4510 , piece_id=TW5DNNPLSQZK2ZLQJ6I4JWSX2GRBUZWPBVCARPSK7PHG4QQQYGAA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.618396953 (48h8m49.489479998s) [seg_pos={0 0}, upload=42321]: Segment reached optimal threshold of 13 pieces. Dec 7 15:37:49.618588087 (48h8m49.489671122s) [noise=true , address=145.79.227.20:4438 , piece_id=M5RLXAM6ZXDIIIIFGYU5SRNZ5AI7QRP24IPV4C77C3CYZECSMBRA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 7 15:37:49.618595662 (48h8m49.489678697s) [noise=true , address=145.79.227.20:4438 , piece_id=M5RLXAM6ZXDIIIIFGYU5SRNZ5AI7QRP24IPV4C77C3CYZECSMBRA, seg_pos={0 0}, upload=42321]: Long tail cancellation due to optimal threshold Dec 7 15:37:49.618605130 (48h8m49.489688165s) [noise=true , address=145.79.227.26:4654 , piece_id=EQSMRY5SVYGTITHH3UF5P5SGXBZNDITJY753CD7VY423T66USQSQ, seg_pos={0 0}, upload=42321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53114->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53114->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53114->145.79.227.26:4654: use of closed network connection Dec 7 15:37:49.618610640 (48h8m49.489693675s) [noise=true , address=145.79.227.26:4654 , piece_id=EQSMRY5SVYGTITHH3UF5P5SGXBZNDITJY753CD7VY423T66USQSQ, seg_pos={0 0}, upload=42321]: Long tail cancellation due to optimal threshold Dec 7 15:37:49.618719667 (48h8m49.489802702s) [noise=true , address=145.79.227.44:4193 , piece_id=YHKVOHIRGBDBOTJ2T7UGW7B7UI4D7B2OC5HZA6ENTS6MYS3RBS6Q, seg_pos={0 0}, upload=42321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 7 15:37:49.618725498 (48h8m49.489808533s) [noise=true , address=145.79.227.44:4193 , piece_id=YHKVOHIRGBDBOTJ2T7UGW7B7UI4D7B2OC5HZA6ENTS6MYS3RBS6Q, seg_pos={0 0}, upload=42321]: Long tail cancellation due to optimal threshold Dec 7 15:37:49.618742772 (48h8m49.489825807s) [seg_pos={0 0}, upload=42321]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 15:37:49.618805391 (48h8m49.489888436s) [upload=42321]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 15:37:49.618809639 (48h8m49.489892684s) [upload=42321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 15:37:49.936557632 (48h8m49.807640677s) [upload=42321]: Done uploading segments... Dec 7 15:37:49.936575908 (48h8m49.807658953s) [upload=42321]: Upload finished. err: <nil> Duration: 6.548693755s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.008158233s trace: https://jaeger.dev.storj.io/trace/30a119e66e6df16c FAIL (2025-12-07 15:39:25 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.008185615s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 4.667823111s trace: https://jaeger.dev.storj.io/trace/6aa77812f5688c1c FAIL (2025-12-07 15:39:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments... Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments... Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 4.66784946s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [..X....XXXXX..X.XXXXXX..X.X....X....X.X.........XX..X.....XX] duration: 5.000922096s trace: https://jaeger.dev.storj.io/trace/4d71784616d066aa FAIL (2025-12-07 15:39:21 UTC) orbiter warning: context deadline exceeded Duration: 5.001036384s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X.....X......X.X.................................X] duration: 30.001693756s trace: https://jaeger.dev.storj.io/trace/57fa7c8cbec72018 FAIL (2025-12-07 15:39:10 UTC) orbiter warning: context deadline exceeded Duration: 30.001719104s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X........X.] duration: 7.879232695s trace: https://jaeger.dev.storj.io/trace/396da084cc259886 FAIL (2025-12-07 15:39:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 15:37:46.609384273 (48h8m46.480467318s) [upload=42321]: Starting upload Dec 7 15:37:46.609447142 (48h8m46.480530177s) [upload=42321]: Done starting upload Dec 7 15:37:46.609581388 (48h8m46.480664433s) [upload=42321]: Uploading segments... Dec 7 15:37:46.609752243 (48h8m46.480835278s) [upload=42321]: Got next segment. Inline: false Dec 7 15:37:46.609829109 (48h8m46.480912144s) [upload=42321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 15:37:46.609834741 (48h8m46.480917776s) [upload=42321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 15:37:47.125547557 (48h8m46.996630592s) [seg_pos={0 0}, upload=42321]: Begin upload segment... Dec 7 15:37:47.125614395 (48h8m46.99669743s) [seg_pos={0 0}, upload=42321]: Done begin upload segment. Dec 7 15:37:47.125644111 (48h8m46.996727156s) [noise=true , address=145.79.227.41:4086 , piece_id=UAWGBWBMOIAWXIO7BAIIZ7RA536ERIWIHMS7KAUD3E23ALEEWC2Q, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.125697462 (48h8m46.996780507s) [noise=true , address=145.79.227.42:4122 , piece_id=FP3VXDDCVYBBIS4D3EY233ZISVBOFEBLSNAOPX2773SUATZAGSTA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.126407134 (48h8m46.997490179s) [noise=true , address=77.90.31.17:4771 , piece_id=DMZ57PS4PONWG4ENMILDTTI3I2CSQNXI2ZX3PM7Y7RJDI7HEHOLA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.126853805 (48h8m46.99793685s) [noise=true , address=145.79.227.26:4654 , piece_id=EQSMRY5SVYGTITHH3UF5P5SGXBZNDITJY753CD7VY423T66USQSQ, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.127212217 (48h8m46.998295262s) [noise=true , address=145.79.227.22:4510 , piece_id=TW5DNNPLSQZK2ZLQJ6I4JWSX2GRBUZWPBVCARPSK7PHG4QQQYGAA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.127866543 (48h8m46.998949588s) [noise=true , address=145.79.227.20:4438 , piece_id=M5RLXAM6ZXDIIIIFGYU5SRNZ5AI7QRP24IPV4C77C3CYZECSMBRA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.128359091 (48h8m46.999442126s) [noise=true , address=145.79.227.29:4942 , piece_id=ES2FUPLIBQ4U6BDHBC6QRVQG53UIGWS5DPO2DPF6R3FKVPF3WXAQ, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.128780403 (48h8m46.999863448s) [noise=true , address=145.79.227.44:4193 , piece_id=YHKVOHIRGBDBOTJ2T7UGW7B7UI4D7B2OC5HZA6ENTS6MYS3RBS6Q, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.129285576 (48h8m47.000368621s) [noise=true , address=145.79.227.41:4085 , piece_id=YNLMM5NYRB47MBWOUZYPYUINN5HVCB2A5YNW72YFRRQQ6YGACR3Q, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.129727607 (48h8m47.000810652s) [noise=true , address=145.79.227.27:4690 , piece_id=QI5QLCVLP6Z5DNCZZXUFGVAGHWKVC32UA5YOIMO42M5CEKZ4Q2MA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.130221537 (48h8m47.001304582s) [noise=true , address=77.90.31.10:4520 , piece_id=Q32XVBGWDN53N5FKRVU2Z354FJUVORB46PS5WAST5VV4CAKKPSTA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.131412536 (48h8m47.002495591s) [noise=true , address=145.79.227.23:4546 , piece_id=KZYS7BKWUN5D6RZJG4ER3HIFMWAZTR63L3M2RUBPBXZH33YF74TA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.131599662 (48h8m47.002682697s) [noise=true , address=145.79.227.16:4911 , piece_id=FHUIUP32L5L4JDXVSKU66V45L22KGWYWSPBBO4LJ6U4IWNKI3ICA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.133038882 (48h8m47.004121927s) [noise=true , address=145.79.227.25:4619 , piece_id=ALJOALLXBTWEHOELG6YET5HW34EOBKDOA7AL3T2KGLKQ253Z7MXA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.138418296 (48h8m47.009501351s) [noise=true , address=145.79.227.34:4905 , piece_id=ND7JKTGJH3DZBSCC2PQXM5Z57MVWEVUSMIFXCO73W7CC54QJY66A, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.141044165 (48h8m47.01212722s) [noise=true , address=77.90.31.15:4698 , piece_id=Z7NPTLYBD5DZVLVF3JKLJHLRMW7QFWCO7ZXGIKEWCMEM5LRV4ZLA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:48.836748043 (48h8m48.707831078s) [upload=42321]: Next returned nil segment Dec 7 15:37:48.836760767 (48h8m48.707843812s) [upload=42321]: Waiting for error group managing segments... Dec 7 15:37:49.226822230 (48h8m49.097905325s) [noise=true , address=77.90.31.17:4771 , piece_id=DMZ57PS4PONWG4ENMILDTTI3I2CSQNXI2ZX3PM7Y7RJDI7HEHOLA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.229191832 (48h8m49.100274867s) [noise=true , address=77.90.31.10:4520 , piece_id=Q32XVBGWDN53N5FKRVU2Z354FJUVORB46PS5WAST5VV4CAKKPSTA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.269025022 (48h8m49.140108057s) [noise=true , address=145.79.227.27:4690 , piece_id=QI5QLCVLP6Z5DNCZZXUFGVAGHWKVC32UA5YOIMO42M5CEKZ4Q2MA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.270472808 (48h8m49.141555883s) [noise=true , address=145.79.227.42:4122 , piece_id=FP3VXDDCVYBBIS4D3EY233ZISVBOFEBLSNAOPX2773SUATZAGSTA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.271570349 (48h8m49.142653404s) [noise=true , address=145.79.227.16:4911 , piece_id=FHUIUP32L5L4JDXVSKU66V45L22KGWYWSPBBO4LJ6U4IWNKI3ICA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.281506679 (48h8m49.152589714s) [noise=true , address=145.79.227.25:4619 , piece_id=ALJOALLXBTWEHOELG6YET5HW34EOBKDOA7AL3T2KGLKQ253Z7MXA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.288241994 (48h8m49.15932503s) [noise=true , address=145.79.227.34:4905 , piece_id=ND7JKTGJH3DZBSCC2PQXM5Z57MVWEVUSMIFXCO73W7CC54QJY66A, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.292810224 (48h8m49.163893279s) [noise=true , address=145.79.227.41:4085 , piece_id=YNLMM5NYRB47MBWOUZYPYUINN5HVCB2A5YNW72YFRRQQ6YGACR3Q, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.306469012 (48h8m49.177552057s) [noise=true , address=77.90.31.15:4698 , piece_id=Z7NPTLYBD5DZVLVF3JKLJHLRMW7QFWCO7ZXGIKEWCMEM5LRV4ZLA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.331878357 (48h8m49.202961402s) [noise=true , address=145.79.227.29:4942 , piece_id=ES2FUPLIBQ4U6BDHBC6QRVQG53UIGWS5DPO2DPF6R3FKVPF3WXAQ, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.361459515 (48h8m49.23254255s) [noise=true , address=145.79.227.23:4546 , piece_id=KZYS7BKWUN5D6RZJG4ER3HIFMWAZTR63L3M2RUBPBXZH33YF74TA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.372131507 (48h8m49.243214552s) [noise=true , address=145.79.227.41:4086 , piece_id=UAWGBWBMOIAWXIO7BAIIZ7RA536ERIWIHMS7KAUD3E23ALEEWC2Q, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.618376213 (48h8m49.489459258s) [noise=true , address=145.79.227.22:4510 , piece_id=TW5DNNPLSQZK2ZLQJ6I4JWSX2GRBUZWPBVCARPSK7PHG4QQQYGAA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.618396953 (48h8m49.489479998s) [seg_pos={0 0}, upload=42321]: Segment reached optimal threshold of 13 pieces. Dec 7 15:37:49.618588087 (48h8m49.489671122s) [noise=true , address=145.79.227.20:4438 , piece_id=M5RLXAM6ZXDIIIIFGYU5SRNZ5AI7QRP24IPV4C77C3CYZECSMBRA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 7 15:37:49.618595662 (48h8m49.489678697s) [noise=true , address=145.79.227.20:4438 , piece_id=M5RLXAM6ZXDIIIIFGYU5SRNZ5AI7QRP24IPV4C77C3CYZECSMBRA, seg_pos={0 0}, upload=42321]: Long tail cancellation due to optimal threshold Dec 7 15:37:49.618605130 (48h8m49.489688165s) [noise=true , address=145.79.227.26:4654 , piece_id=EQSMRY5SVYGTITHH3UF5P5SGXBZNDITJY753CD7VY423T66USQSQ, seg_pos={0 0}, upload=42321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53114->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53114->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53114->145.79.227.26:4654: use of closed network connection Dec 7 15:37:49.618610640 (48h8m49.489693675s) [noise=true , address=145.79.227.26:4654 , piece_id=EQSMRY5SVYGTITHH3UF5P5SGXBZNDITJY753CD7VY423T66USQSQ, seg_pos={0 0}, upload=42321]: Long tail cancellation due to optimal threshold Dec 7 15:37:49.618719667 (48h8m49.489802702s) [noise=true , address=145.79.227.44:4193 , piece_id=YHKVOHIRGBDBOTJ2T7UGW7B7UI4D7B2OC5HZA6ENTS6MYS3RBS6Q, seg_pos={0 0}, upload=42321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 7 15:37:49.618725498 (48h8m49.489808533s) [noise=true , address=145.79.227.44:4193 , piece_id=YHKVOHIRGBDBOTJ2T7UGW7B7UI4D7B2OC5HZA6ENTS6MYS3RBS6Q, seg_pos={0 0}, upload=42321]: Long tail cancellation due to optimal threshold Dec 7 15:37:49.618742772 (48h8m49.489825807s) [seg_pos={0 0}, upload=42321]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 15:37:49.618805391 (48h8m49.489888436s) [upload=42321]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 15:37:49.618809639 (48h8m49.489892684s) [upload=42321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 15:37:49.936557632 (48h8m49.807640677s) [upload=42321]: Done uploading segments... Dec 7 15:37:49.936575908 (48h8m49.807658953s) [upload=42321]: Upload finished. err: <nil> Duration: 7.879262571s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.716624739s trace: https://jaeger.dev.storj.io/trace/6aa77812f5688c1c FAIL (2025-12-07 15:38:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments... Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments... Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 5.716647754s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.....X......X.X.................................] duration: 17.169843829s trace: https://jaeger.dev.storj.io/trace/603f0be1aa1243cb FAIL (2025-12-07 15:38:40 UTC) orbiter warning: context deadline exceeded Duration: 17.169871283s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.954559383s trace: https://jaeger.dev.storj.io/trace/6aa77812f5688c1c FAIL (2025-12-07 15:38:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece... Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments... Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil> Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments... Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 5.954579392s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [...X....XXXXX..X.XXXXXX..X.X....X....X.X.........XX..X.....X] duration: 5.001521463s trace: https://jaeger.dev.storj.io/trace/4925f360e8442e24 FAIL (2025-12-07 15:38:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001540399s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X........X] duration: 30.003788076s trace: https://jaeger.dev.storj.io/trace/396da084cc259886 FAIL (2025-12-07 15:38:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 7 15:37:46.609384273 (48h8m46.480467318s) [upload=42321]: Starting upload Dec 7 15:37:46.609447142 (48h8m46.480530177s) [upload=42321]: Done starting upload Dec 7 15:37:46.609581388 (48h8m46.480664433s) [upload=42321]: Uploading segments... Dec 7 15:37:46.609752243 (48h8m46.480835278s) [upload=42321]: Got next segment. Inline: false Dec 7 15:37:46.609829109 (48h8m46.480912144s) [upload=42321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 7 15:37:46.609834741 (48h8m46.480917776s) [upload=42321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 7 15:37:47.125547557 (48h8m46.996630592s) [seg_pos={0 0}, upload=42321]: Begin upload segment... Dec 7 15:37:47.125614395 (48h8m46.99669743s) [seg_pos={0 0}, upload=42321]: Done begin upload segment. Dec 7 15:37:47.125644111 (48h8m46.996727156s) [noise=true , address=145.79.227.41:4086 , piece_id=UAWGBWBMOIAWXIO7BAIIZ7RA536ERIWIHMS7KAUD3E23ALEEWC2Q, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.125697462 (48h8m46.996780507s) [noise=true , address=145.79.227.42:4122 , piece_id=FP3VXDDCVYBBIS4D3EY233ZISVBOFEBLSNAOPX2773SUATZAGSTA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.126407134 (48h8m46.997490179s) [noise=true , address=77.90.31.17:4771 , piece_id=DMZ57PS4PONWG4ENMILDTTI3I2CSQNXI2ZX3PM7Y7RJDI7HEHOLA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.126853805 (48h8m46.99793685s) [noise=true , address=145.79.227.26:4654 , piece_id=EQSMRY5SVYGTITHH3UF5P5SGXBZNDITJY753CD7VY423T66USQSQ, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.127212217 (48h8m46.998295262s) [noise=true , address=145.79.227.22:4510 , piece_id=TW5DNNPLSQZK2ZLQJ6I4JWSX2GRBUZWPBVCARPSK7PHG4QQQYGAA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.127866543 (48h8m46.998949588s) [noise=true , address=145.79.227.20:4438 , piece_id=M5RLXAM6ZXDIIIIFGYU5SRNZ5AI7QRP24IPV4C77C3CYZECSMBRA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.128359091 (48h8m46.999442126s) [noise=true , address=145.79.227.29:4942 , piece_id=ES2FUPLIBQ4U6BDHBC6QRVQG53UIGWS5DPO2DPF6R3FKVPF3WXAQ, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.128780403 (48h8m46.999863448s) [noise=true , address=145.79.227.44:4193 , piece_id=YHKVOHIRGBDBOTJ2T7UGW7B7UI4D7B2OC5HZA6ENTS6MYS3RBS6Q, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.129285576 (48h8m47.000368621s) [noise=true , address=145.79.227.41:4085 , piece_id=YNLMM5NYRB47MBWOUZYPYUINN5HVCB2A5YNW72YFRRQQ6YGACR3Q, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.129727607 (48h8m47.000810652s) [noise=true , address=145.79.227.27:4690 , piece_id=QI5QLCVLP6Z5DNCZZXUFGVAGHWKVC32UA5YOIMO42M5CEKZ4Q2MA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.130221537 (48h8m47.001304582s) [noise=true , address=77.90.31.10:4520 , piece_id=Q32XVBGWDN53N5FKRVU2Z354FJUVORB46PS5WAST5VV4CAKKPSTA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.131412536 (48h8m47.002495591s) [noise=true , address=145.79.227.23:4546 , piece_id=KZYS7BKWUN5D6RZJG4ER3HIFMWAZTR63L3M2RUBPBXZH33YF74TA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.131599662 (48h8m47.002682697s) [noise=true , address=145.79.227.16:4911 , piece_id=FHUIUP32L5L4JDXVSKU66V45L22KGWYWSPBBO4LJ6U4IWNKI3ICA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.133038882 (48h8m47.004121927s) [noise=true , address=145.79.227.25:4619 , piece_id=ALJOALLXBTWEHOELG6YET5HW34EOBKDOA7AL3T2KGLKQ253Z7MXA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.138418296 (48h8m47.009501351s) [noise=true , address=145.79.227.34:4905 , piece_id=ND7JKTGJH3DZBSCC2PQXM5Z57MVWEVUSMIFXCO73W7CC54QJY66A, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:47.141044165 (48h8m47.01212722s) [noise=true , address=77.90.31.15:4698 , piece_id=Z7NPTLYBD5DZVLVF3JKLJHLRMW7QFWCO7ZXGIKEWCMEM5LRV4ZLA, seg_pos={0 0}, upload=42321]: Uploading piece... Dec 7 15:37:48.836748043 (48h8m48.707831078s) [upload=42321]: Next returned nil segment Dec 7 15:37:48.836760767 (48h8m48.707843812s) [upload=42321]: Waiting for error group managing segments... Dec 7 15:37:49.226822230 (48h8m49.097905325s) [noise=true , address=77.90.31.17:4771 , piece_id=DMZ57PS4PONWG4ENMILDTTI3I2CSQNXI2ZX3PM7Y7RJDI7HEHOLA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.229191832 (48h8m49.100274867s) [noise=true , address=77.90.31.10:4520 , piece_id=Q32XVBGWDN53N5FKRVU2Z354FJUVORB46PS5WAST5VV4CAKKPSTA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.269025022 (48h8m49.140108057s) [noise=true , address=145.79.227.27:4690 , piece_id=QI5QLCVLP6Z5DNCZZXUFGVAGHWKVC32UA5YOIMO42M5CEKZ4Q2MA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.270472808 (48h8m49.141555883s) [noise=true , address=145.79.227.42:4122 , piece_id=FP3VXDDCVYBBIS4D3EY233ZISVBOFEBLSNAOPX2773SUATZAGSTA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.271570349 (48h8m49.142653404s) [noise=true , address=145.79.227.16:4911 , piece_id=FHUIUP32L5L4JDXVSKU66V45L22KGWYWSPBBO4LJ6U4IWNKI3ICA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.281506679 (48h8m49.152589714s) [noise=true , address=145.79.227.25:4619 , piece_id=ALJOALLXBTWEHOELG6YET5HW34EOBKDOA7AL3T2KGLKQ253Z7MXA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.288241994 (48h8m49.15932503s) [noise=true , address=145.79.227.34:4905 , piece_id=ND7JKTGJH3DZBSCC2PQXM5Z57MVWEVUSMIFXCO73W7CC54QJY66A, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.292810224 (48h8m49.163893279s) [noise=true , address=145.79.227.41:4085 , piece_id=YNLMM5NYRB47MBWOUZYPYUINN5HVCB2A5YNW72YFRRQQ6YGACR3Q, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.306469012 (48h8m49.177552057s) [noise=true , address=77.90.31.15:4698 , piece_id=Z7NPTLYBD5DZVLVF3JKLJHLRMW7QFWCO7ZXGIKEWCMEM5LRV4ZLA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.331878357 (48h8m49.202961402s) [noise=true , address=145.79.227.29:4942 , piece_id=ES2FUPLIBQ4U6BDHBC6QRVQG53UIGWS5DPO2DPF6R3FKVPF3WXAQ, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.361459515 (48h8m49.23254255s) [noise=true , address=145.79.227.23:4546 , piece_id=KZYS7BKWUN5D6RZJG4ER3HIFMWAZTR63L3M2RUBPBXZH33YF74TA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.372131507 (48h8m49.243214552s) [noise=true , address=145.79.227.41:4086 , piece_id=UAWGBWBMOIAWXIO7BAIIZ7RA536ERIWIHMS7KAUD3E23ALEEWC2Q, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.618376213 (48h8m49.489459258s) [noise=true , address=145.79.227.22:4510 , piece_id=TW5DNNPLSQZK2ZLQJ6I4JWSX2GRBUZWPBVCARPSK7PHG4QQQYGAA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: <nil> Dec 7 15:37:49.618396953 (48h8m49.489479998s) [seg_pos={0 0}, upload=42321]: Segment reached optimal threshold of 13 pieces. Dec 7 15:37:49.618588087 (48h8m49.489671122s) [noise=true , address=145.79.227.20:4438 , piece_id=M5RLXAM6ZXDIIIIFGYU5SRNZ5AI7QRP24IPV4C77C3CYZECSMBRA, seg_pos={0 0}, upload=42321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 7 15:37:49.618595662 (48h8m49.489678697s) [noise=true , address=145.79.227.20:4438 , piece_id=M5RLXAM6ZXDIIIIFGYU5SRNZ5AI7QRP24IPV4C77C3CYZECSMBRA, seg_pos={0 0}, upload=42321]: Long tail cancellation due to optimal threshold Dec 7 15:37:49.618605130 (48h8m49.489688165s) [noise=true , address=145.79.227.26:4654 , piece_id=EQSMRY5SVYGTITHH3UF5P5SGXBZNDITJY753CD7VY423T66USQSQ, seg_pos={0 0}, upload=42321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53114->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53114->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53114->145.79.227.26:4654: use of closed network connection Dec 7 15:37:49.618610640 (48h8m49.489693675s) [noise=true , address=145.79.227.26:4654 , piece_id=EQSMRY5SVYGTITHH3UF5P5SGXBZNDITJY753CD7VY423T66USQSQ, seg_pos={0 0}, upload=42321]: Long tail cancellation due to optimal threshold Dec 7 15:37:49.618719667 (48h8m49.489802702s) [noise=true , address=145.79.227.44:4193 , piece_id=YHKVOHIRGBDBOTJ2T7UGW7B7UI4D7B2OC5HZA6ENTS6MYS3RBS6Q, seg_pos={0 0}, upload=42321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 7 15:37:49.618725498 (48h8m49.489808533s) [noise=true , address=145.79.227.44:4193 , piece_id=YHKVOHIRGBDBOTJ2T7UGW7B7UI4D7B2OC5HZA6ENTS6MYS3RBS6Q, seg_pos={0 0}, upload=42321]: Long tail cancellation due to optimal threshold Dec 7 15:37:49.618742772 (48h8m49.489825807s) [seg_pos={0 0}, upload=42321]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 7 15:37:49.618805391 (48h8m49.489888436s) [upload=42321]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 7 15:37:49.618809639 (48h8m49.489892684s) [upload=42321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 7 15:37:49.936557632 (48h8m49.807640677s) [upload=42321]: Done uploading segments... Dec 7 15:37:49.936575908 (48h8m49.807658953s) [upload=42321]: Upload finished. err: <nil> Duration: 30.003807723s core uplink us-select-1

1
...
11771
11772
11773
11774
11775
...
17197