Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't input non ascii characters #3799

Open
5 of 10 tasks
sledgehammer999 opened this issue May 16, 2024 · 14 comments · May be fixed by #3951
Open
5 of 10 tasks

Can't input non ascii characters #3799

sledgehammer999 opened this issue May 16, 2024 · 14 comments · May be fixed by #3951

Comments

@sledgehammer999
Copy link

Checklist

  • I have read through the manual page (man fzf)
  • I have searched through the existing issues
  • For bug reports, I have checked if the bug is reproducible in the latest version of fzf

Output of fzf --version

0.52.1 (6432f00)

OS

  • Linux
  • macOS
  • Windows
  • Etc.

Shell

  • bash
  • zsh
  • fish

Problem / Steps to reproduce

Specifically I use bash from git for windows.
OS: Win 10
Terminal:

  1. Windows Terminal (1.20.11271.0)
  2. mintty 3.7.0 (shipped with git for windows)

When I hit CTRL+T and try to input greek characters they appear in the prompt either transliterated to their equivalent ascii (eg α becomes a) or printed as the ? character.
This happens with both terminals.

Both terminals are able to print greek characters (eg ls a folder with greek filenames) and are able to take greek input in their prompts. So the issue is with fzf's prompt.

@Konfekt
Copy link
Contributor

Konfekt commented May 17, 2024

With the same setup, non-ascii characters make Fzf quit; it's intricate

@Konfekt
Copy link
Contributor

Konfekt commented May 17, 2024

I found out that as soon as I removed the --height option from $FZF_DEFAULT_OPTS, fzf accepts non-ascii characters in Git Bash in Windows Terminal; however I could not make fzf work at all in Git bash.

Regarding the --height option, it's unfortunately set by almost all common commands, such as Ctrl-T.
The following lines, say in .bash_profile, remove it wherever possible:

  export FZF_DEFAULT_OPTS="${FZF_DEFAULT_OPTS//--height[=[:space:]][1-9][0-9]?/ }"
  export FZF_CTRL_T_OPTS="${FZF_CTRL_T_OPTS//--height[=[:space:]][1-9][0-9]?/ }"
  export FZF_CTRL_R_OPTS="${FZF_CTRL_R_OPTS//--height[=[:space:]][1-9][0-9]?/ }"

  export FZF_ALT_C_OPTS="${FZF_DEFAULT_OPTS} --preview='tree.com //F {} | tail +3 | head -\$LINES'"

Ideally, of course, the issue with setting --height in Fzf in Git Bash itself could be fixed.

@junegunn
Copy link
Owner

You can just append --no-height at the end, because the last one wins.

export FZF_DEFAULT_OPTS="$FZF_DEFAULT_OPTS --no-height"

@Konfekt
Copy link
Contributor

Konfekt commented May 17, 2024

Thank you, I was not aware of it. This is more robust. Ideally, since the provided bash scripts set their own custom parameters, add a check for Git Bash, say by [ -z ${MSYSTEM+x} ] || ..., and append --no-height.

@sledgehammer999
Copy link
Author

I can confirm that --no-height works for me too. In both Windows Terminal and mintty(from git bash).
BTW echo $FZF_DEFAULT_OPTS is empty so I suppose --height is the default, right?

@Konfekt
Copy link
Contributor

Konfekt commented May 17, 2024

Yes, you can inspect here:

https://github.com/junegunn/fzf/blob/0994d9c881f8380997b96bb0a4a7416b50bc2b0d/shell/key-bindings.bash

__fzf_defaults() {
  # $1: Prepend to FZF_DEFAULT_OPTS_FILE and FZF_DEFAULT_OPTS
  # $2: Append to FZF_DEFAULT_OPTS_FILE and FZF_DEFAULT_OPTS
  echo "--height ${FZF_TMUX_HEIGHT:-40%} --bind=ctrl-z:ignore $1"
  command cat "${FZF_DEFAULT_OPTS_FILE-}" 2> /dev/null
  echo "${FZF_DEFAULT_OPTS-} $2"
}


__fzf_select__() {
  FZF_DEFAULT_COMMAND=${FZF_CTRL_T_COMMAND:-} \
  FZF_DEFAULT_OPTS=$(__fzf_defaults "--reverse --walker=file,dir,follow,hidden --scheme=path" "${FZF_CTRL_T_OPTS-} -m") \

@cpkio
Copy link

cpkio commented Jul 19, 2024

For me unability to input non-latin characters was introduced in 0.54, 0.53 works fine.

@Konfekt
Copy link
Contributor

Konfekt commented Jul 19, 2024

Did you try #3799 (comment) ?

@cpkio
Copy link

cpkio commented Jul 19, 2024

Now I did, with no result. Will stick to 0.53 for a while.

@junegunn
Copy link
Owner

junegunn commented Jul 19, 2024

If you're experiencing a change of behavior in 0.54.0, this might be the cause.

dca2262

See #3847 (comment)

I'll look into it when I get some time.

@junegunn
Copy link
Owner

I can confirm that the LightRenderer for Windows written by @kelleyma49 cannot take Unicode characters.

inHandle, _ := syscall.Open("CONIN$", syscall.O_RDWR, 0)
if err := windows.GetConsoleMode(windows.Handle(inHandle), &r.origStateInput); err != nil {
return err
}
r.inHandle = uintptr(inHandle)

When I type in a Unicode character, it reads it as ? (63).

junegunn added a commit that referenced this issue Jul 25, 2024
This reverts commit dca2262.

> For mouse support on mintty
> Fix #3847

The current implementation LightRenderer for Windows is unable to accept
non-ASCII input unlike the tcell renderer. So even though it supports
mouse on mintty, we shouldn't use it as the default.

* #3799
* #3847
@junegunn
Copy link
Owner

junegunn commented Jul 25, 2024

Ideally, we should fix LightRenderer to support non-ASCII input, but I don't have the Windows expertise to do that, so I'm going to revert the decision I made in 0.54 to prefer LightRenderer over tcell-based renderer for now.

Related:

@kelleyma49 kelleyma49 linked a pull request Aug 4, 2024 that will close this issue
@kelleyma49
Copy link
Contributor

kelleyma49 commented Aug 4, 2024

I believe I have a fix for reading and displaying non-ASCII input. Please see #3951

@chrisant996
Copy link

ReadConsoleW is the way to read Unicode input, accepting it as UTF16 which is the native Windows encoding for Unicode and is available starting in Windows NT and XP (so it's available in all currently supported versions of Windows).

Console Input Buffer has some info on how the input buffer works.

To read the full range of Unicode, surrogate pairs have to be handled. For example, code can't simply convert each UTF16 character individually into UTF8, because surrogate pairs are only a UTF16 encoding concept. Code needs to be aware of when a leading surrogate has been input, and then get the trailing surrogate and then convert them as a string from UTF16 to UTF8. Or convert them to UTF32 and then to UTF8.

Clink has some C++ code for handling surrogate pairs and converting from UTF16 to UTF8 for input here. Review that whole file for more context on reading full Unicode input in a Windows console. (Note that the mouse support doesn't generate proper ANSI escape codes for mount input -- it generates private internal-only custom escape codes for mouse input.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants