feature request: cargo fuzz miri <target> to run all corpus items in miri

[5225225]

The use case here would be fuzzing some unsafe heavy code to find a corpus that should see a wide variety of inputs, then running the fuzzer, feeding each corpus item in turn to the fuzzer. This is useful since the fuzzer can't really notice rust specific UB if it doesn't happen to crash, but that's still UB that the user wants to know about.

This is not asking for fuzzing to be ran using miri (like #311), just running the fuzzer body once per corpus item. This is not exactly hard to do manually (refactor out the fuzzer body into a lib.rs, add a #[test] that reads each corpus item and feeds it to that, then run cargo miri test), but it's a bit of a pain.

[Manishearth]

Oh that's a clever idea. I'm in favor though don't have time to implement it myself.

[fitzgen]

Same

[inahga]

This is not exactly hard to do manually (refactor out the fuzzer body into a lib.rs, add a #[test] that reads each corpus item and feeds it to that, then run cargo miri test), but it's a bit of a pain.

I've had success with this approach and using rstest. Example:

#![cfg_attr(not(any(miri, test)), no_main)]

use libfuzzer_sys::{
    arbitrary::{self, Arbitrary, Unstructured},
    fuzz_target, Corpus,
};

#[derive(Debug, Arbitrary, Clone)]
struct Input {
    foo: usize,
    bar: Vec<u8>,
}

fuzz_target!(|input: Input| { run(input); });

fn run(input: Input) {
    // ...
}

#[cfg(test)]
mod tests {
    use crate::{run, Input};

    #[cfg(miri)]
    use {
        crate::{run, Input},
        libfuzzer_sys::arbitrary::{Arbitrary, Unstructured},
        rstest::rstest,
        std::{fs::File, io::Read, path::PathBuf},
    };

    #[rstest]
    #[cfg(miri)]
    fn miri(#[files("corpus/fuzz_corpus/*")] path: PathBuf) {
        let mut input = File::open(path).unwrap();
        let mut buf = Vec::new();
        input.read_to_end(&mut buf).unwrap();

        let mut unstructured = Unstructured::new(&buf);
        let input = Input::arbitrary(&mut unstructured).unwrap();
        run(input);
    }
}

Then cargo miri nextest run --bin [name of target] pretty much Just Works.

rstest composes well enough with this that I'm not sure it's worth code changes in cargo-fuzz? I suppose fuzz_target! could be extended to auto-generate this kind of code, but perhaps simpler is better. I'm happy to write up something in the fuzz book. Would that be acceptable @5225225 @Manishearth?

n.b. that Miri can have severe slowdowns if the code under test touches globals or thread local storage too much, so in those cases it may be better to shell out and invoke Miri separately for each entry in the corpus. Something to the tune of find corpus/target | parallel FILE={} cargo miri test --bin target miri. In which case the test would take the filename as an environment variable.