how to fix an encoding issue with Chinese characters when piping in a Windows console?

[cesaryuan]

What version of ripgrep are you using?

ripgrep 13.0.0 (rev af6b6c5)
-SIMD -AVX (compiled)
+SIMD +AVX (runtime)

How did you install ripgrep?

Scoop

What operating system are you using ripgrep on?

Windows 10 19045.2846
PowerShell 7.3.4

Describe your bug.

There seems to be an encoding issue when piping the output of a ripgrep search. The chinese characters 测试 (and probably others, those are just the ones I've tested with) are converted to wrong characters(娴嬭瘯) when being piped to anything.

Passing stdout to a file with > is also with this problem.

This is what I used to test.

PS D:\0-Download> echo "测试Chinese Test" | rg -o "测试"
测试
PS D:\0-Download> echo "测试Chinese Test" | rg -o "测试" | echo
娴嬭瘯
PS D:\0-Download> echo "测试Chinese Test" | rg -o "测试" > test.txt
PS D:\0-Download> cat .\test.txt
娴嬭瘯

What are the steps to reproduce the behavior?

PS D:\0-Download> echo "测试Chinese Test" | rg -o "测试"
PS D:\0-Download> echo "测试Chinese Test" | rg -o "测试" | echo
PS D:\0-Download> echo "测试Chinese Test" | rg -o "测试" > test.txt
PS D:\0-Download> cat .\test.txt

What is the actual behavior?

PS D:\0-Download> echo "测试Chinese Test" | rg -o --debug "测试" | echo
DEBUG|grep_regex::literal|crates\regex\src\literal.rs:58: literal prefixes detected: Literals { lits: [Complete(测试)], limit_size: 250, limit_class: 10 }
DEBUG|grep_cli::decompress|crates\cli\src\decompress.rs:482: gzip: could not find executable in PATH
DEBUG|grep_cli::decompress|crates\cli\src\decompress.rs:482: gzip: could not find executable in PATH
DEBUG|grep_cli::decompress|crates\cli\src\decompress.rs:482: bzip2: could not find executable in PATH
DEBUG|grep_cli::decompress|crates\cli\src\decompress.rs:482: bzip2: could not find executable in PATH
DEBUG|grep_cli::decompress|crates\cli\src\decompress.rs:482: lz4: could not find executable in PATH
DEBUG|grep_cli::decompress|crates\cli\src\decompress.rs:482: brotli: could not find executable in PATH
DEBUG|grep_cli::decompress|crates\cli\src\decompress.rs:482: uncompress: could not find executable in PATH
DEBUG|globset|crates\globset\src\lib.rs:421: built glob set; 0 literals, 0 basenames, 5 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes
娴嬭瘯

What is the expected behavior?

PS D:\0-Download> echo "测试Chinese Test" | rg -o "测试" | echo
测试

[cesaryuan]

ripgrep/FAQ.md

Lines 810 to 814 in 0415448

	If you continue to have encoding problems, you can also force the encoding
	that the console will use for printing to UTF-8 with
	`[System.Console]::OutputEncoding = [System.Text.Encoding]::UTF8`. This
	will also reset when PowerShell is restarted, so you can add that line
	to your profile as well if you want to make the setting permanent.

Tried this. Now works!

[BurntSushi]

Nice, glad you got it figured out and thank you for posting the solution!