search result is less than grep because of stopped searching binary file after match

[bigdogs]

hi,

executing command "rg -i word", I found only part of matched lines are outputed, but with grep word -ri, the whole match result are all outputed. After I investigating for a while, I noticed the warnning stopped searching binary file after match (found "\0" byte around offset 948345).

Is this a feature of rg and I should use rg -ia word instead in this case? (Although personally I feel that this is not very reasonable, when I was investigating an issue today, I mistakenly thought that certain events didn't occur because the rg -i command only partially outputted the logs.)

[BurntSushi]

Generally speaking, ripgrep and GNU grep's binary file detection works the same way. They both look for a NUL byte, and if one is found, the search will stop once a match is found. Since you didn't say which grep you're using and since you didn't provide any reproduction, I can't provide a more specific analysis than that. But note it is a non goal for them to behave identically.

Otherwise this is intended behavior. The issue is that if ripgrep didn't treat binary data special, then stuff can be dumped to your terminal and royally mess things up. It's a trade off.

Indeed, if you want to forcefully treat everything as plain text, then -a/--text is the correct path there. The same is also true for grep.

[bigdogs]

The grep is the default version of centos7, maybe it's not following the rule

20230801-143302-1309988855.log: WARNING: stopped searching binary file after match (found "\0" byte around offset 980219)
[[email protected] xs_agent]$ rg processorbase | wc -l
5
[[email protected] xs_agent]$ grep processorbase ./ -rnw | wc -l
547
[[email protected] xs_agent]$ grep --version
grep (GNU grep) 2.20
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Haertel and others, see <http://git.sv.gnu.org/cgit/grep.git/tree/AUTHORS>.
[[email protected] xs_agent]$ rg --version
ripgrep 13.0.0 (rev af6b6c543b)
-SIMD -AVX (compiled)
+SIMD -AVX (runtime)

[BurntSushi]

You haven't provided a reproduction. Until you do that, I can't really do any specific analysis.

And yes, your version of GNU grep is ancient.

[bigdogs]

never mind, Just confirm if it's a expected behavior of rg (though I don't like this feature personly :(

[BurntSushi]

It's a standard feature among all greps, even if the behavior varies a little bit. I already explained that it's protecting you from dumping arbitrary binary data to your terminal. And finally, it's trivial to turn off: alias rg="rg -a".

[bigdogs]

emm.. 1) in my case, there are seval text is invalid (ending with '\0) of my logs (most content are valid utf8). grep consider it normal text and print all matched content. rg will stop if '\0' is found . 2) but for real binary file(e.g xxx.so), grep print Binary file ./lib/xxxso matches, rg print nothing. so I think there is some different feature design choices between "my" grep and rg (btw. alias rg="rg -a" is not a solutoin for my usecase because it will print many corrupted characters in xxx.so` file)

[BurntSushi]

grep should also stop if \0 is found.

Once again, if you can't give me a reproduction I cannot confirm whether it's expected behavior or not. I don't understand why we have to keep doing this dance without a reproduction.

[bigdogs]

:) thinks for the patiant, the reproducable log file private to my company, I will try to make a mini reproduction when I have a time

[BurntSushi]

Thanks