how to search only for files with a specific extension OR no extension at all?

[luckman212]

• macOS 13.0.1
• rg installed via brew

$ rg --version
ripgrep 13.0.0
-SIMD -AVX (compiled)

Not sure if this is possible, but came to ask.

Create 3 files:

for f in '' .sh .py ; do echo foo >foo$f ; done

Try to search for .py and 'no extension'

rg foo -g '*.py' -g '!*.*'

(no results)

...but searched separately, the results are as expected:

$ rg foo -g '*.py'
foo.py:1:foo

$ rg foo -g '!*.*'
foo:1:foo

Is there a way to achieve this in a single command? I tried running with --debug but to be honest I can't parse the results. 🙁

[BurntSushi]

The answer is in the --debug output. It can be a little difficult to parse as it is mostly there to aide programmers, but all you need to do is look for the files you want to search:

$ rg foo -g '*.py' -g '!*.*' --debug 2>&1 | rg foo
DEBUG|rg::args|crates/core/args.rs:543: final argv: ["rg", "--max-columns-preview", "--colors=match:bg:0xff,0x7f,0x00", "--colors=match:fg:white", "--colors=line:none", "--colors=line:fg:magenta", "--colors=path:fg:green", "--type-add=got:*_test.go", "foo", "-g", "*.py", "-g", "!*.*", "--debug"]
DEBUG|grep_regex::literal|crates/regex/src/literal.rs:58: literal prefixes detected: Literals { lits: [Complete(foo)], limit_size: 250, limit_class: 10 }
DEBUG|ignore::walk|crates/ignore/src/walk.rs:1741: ignoring ./foo.sh: Ignore(IgnoreMatch(Override(Glob(Matched(Glob { from: None, original: "!*.*", actual: "**/*.*", is_whitelist: true, is_only_dir: false })))))
DEBUG|ignore::walk|crates/ignore/src/walk.rs:1741: ignoring ./foo: Ignore(IgnoreMatch(Override(Glob(UnmatchedIgnore))))
DEBUG|ignore::walk|crates/ignore/src/walk.rs:1741: ignoring ./foo.py: Ignore(IgnoreMatch(Override(Glob(Matched(Glob { from: None, original: "!*.*", actual: "**/*.*", is_whitelist: true, is_only_dir: false })))))

So foo.sh gets ignored via your !*.* glob, foo.py gets ignored via your !*.* glob and foo gets ignored by virtue of not matching anything. Namely, if you introduce a whitelist glob (which comes in via *.py), then at least one such glob must match in order for ripgrep to include it in its search. (Imagine if that weren't the case. -g '*.py' wouldn't filter your results at all.)

This is behaving as I would expect. It also matches the behavior you are seeking:

Try to search for .py and 'no extension'

Indeed, the results of this would always be empty because "no extension" and ".py" are mutually exclusive. Presumably what you meant to say is "try to search for .py or 'no extension'." This is a little tricky to do. ripgrep doesn't let you formulate arbitrarily complex boolean queries. However, there is a way.

First, we must observe that globs are matched in order. (And this is documented by virtue of ripgrep's globbing semantics being "equivalent to gitignore." And as you might know, gitignore rules can override earlier ones.) The most recent one to match takes precedent. Since *.* matches foo.py, foo.py gets ignored. But you can put the !*.* glob before *.py, in which case, *.py matching will override it:

$ rg foo -g '!*.*' -g '*.py'
foo.py
1:foo

But how to get foo? Well, remember that when you introduce at least one whitelist glob, ripgrep then requires that at least one such glob matches in order to include it in its search. So let's just introduce a glob that matches everything, but give it the lowest precedence:

$ rg foo -g '*' -g '!*.*' -g '*.py'
foo.py
1:foo

foo
1:foo

Veni vidi vici.

[luckman212]

I sincerely appreciate the thorough writeup! This might be a nice example to include in the docs.

[luckman212]

Got caught with another footgun today.

I noticed that with the workaround above, some directories are being excluded. For example:

/path/to/user.workflow.89ABC865-586D-4694-BCDD-DAC0AF44BC46/foo.py

Not sure if there's any workaround for this case...

[BurntSushi]

Yeah you're getting to a point where you probably want to write your filtering logic in a proper programming language. The problem is that !*.* matches user.workflow.89ABC865-586D-4694-BCDD-DAC0AF44BC46 and *.py does not match it, so ripgrep excludes that directory and never sees foo.py at all. To do otherwise would imply that ripgrep would have to traverse all directories even if a user specifically want to filter out a directory. (And some directories can be quite large and take a long time to traverse.)

IMO, your next step is to either write your filtering logic using a real programming language where you have as much expressiveness as you want, or you start getting more specific with your globs. i.e., Instead of a general !*.* as your blacklist, you start blacklisting specific extensions: !*.{foo,bar,baz}.

Another tactic is to split your search into multiple rg commands.

[luckman212]

Thanks for this. I opted for the simple(r) solution of splitting the search into multiple rg commands for now.