Fully reproducible Android images

[mschwaig]

I spent some time taking a closer look at the reproducibility of Robotnix.

I hope it's OK that I summarize my findings here in case someone from the community has ideas about how to resolve or invalidate the remaining issues that I'm seeing.

To perform my analysis I used a modified version of https://github.com/grahamc/r13y.com, which I also presented at NixCon 2022. The TLDR is that you can point this at flakes.

You can find the full results in relation to a mostly unmodified master branch of Robotnix here:
https://cloud.ins.jku.at/index.php/s/NbPNLLKrTiiLf5R

This uses the following config for Robotnix
https://github.com/mschwaig/robotnix-config/tree/0d6ef14ea683d8807cfa9f6049887e419ffda7a1 (vanilla branch)
which in turn uses the following version of Robotnix
https://github.com/danielfullmer/robotnix/tree/d9d22791ef9e3742b87548851baedc3eca511c7d (fix-fod-that-went-bad branch).

To view the results yourself you have to download and unpack the whole folder. I will summarize my findings here.

Of the 30 paths which were not reproducible, 28 have signatures by cache.nixos.org, so they are already built upstream from Robotnix and I have not taken a closer look at them.

Details here.

This can be verified with a command like:

cat reproducibility-log-sha256-XShbduzx2gtpcNjMuGWSu8jFI6RqNyBrtK2PUMfF1jo=.json |
 jq '. | map ( select (.status != "Reproducible") | .drv) | .[]' | xargs -L1 -I'{}' sh -c "nix show-derivation {} | jq -r '.[] | .outputs.out.path'" | xargs -L1 -I'{}' nix path-info --store https://cache.nixos.org --sigs {}

Eventually I would like to add such info to the report and make it possible to skip these paths so that they are not rebuilt at all.

The remaining paths are

• /nix/store/gbgv0y5836kjvdgafx4dx4gyv5ddfkkn-robotnix-aosp_oriole-2022022022.drv and
• /nix/store/4nh8dad5wpxb134iw53ly4nys9bg069i-chromium-100.0.4896.127.drv

but since I have some earlier reports where chromium does not show up, either because it's successfully configured away or because it's reproducible I am not going to look at chromium yet. Instead I am going to look at the AOSP build itself in more detail.

The good news is that the system partition is reproducible, which seems like a big win already.

Half of the differences in the report seem to be related to the order of things inside otatools.zip and the binaries within. Some searches make me think that those binaries look like what Bazil's py_binary produces, but I'm surprised that the ordering within something like that would not be deterministic. Since this does not end up in the running system it's not that bad, but it's annoying, especially when it falls out of the same build step as the actual images.

Looking at the actual partitions inside aosp_oriole-target_files-2022022022.zip there is what looks like a signature using the same signing scheme inside

• dtbo.img
• boot.img
• vendor_boot.img

With

• an additional other difference i have not looked at yet inside boot.img,
• a lot of tiny differences inside userdata.img and
• huge differences inside system_other.img (I don't think that last partition is important on modern Pixel devices, not sure.)
• differences inside vbmeta.img (as far as i know expected because that verified boot related data is actually derived from the contents of the other partitions) and
• what looks like ordering differences for apex_info.pb.

I suspect this should be the docs that describe those signatures https://android.googlesource.com/platform/external/avb/+/master/README.md. However I am also aware that at least when the signing.enable option is set the actual signing happens in another derivation.

I have also written some Nix code to verify the signatures produced in this build step in the style of what I presented at NixCon. I could probably also share what I have in that area but that's still missing most of the actual verification code since it only verifies APK signatures and does not verify that the unsigned artifacts match the signed ones.

To reproduce this analysis yourself can use the commands below, but be aware that it takes some time (maybe days with less than 16 cores) and that tool is in a REALLY clunky state

RUST_LOG=warn nix shell nixpkgs/4a01ca36d6bfc133bc617e661916a81327c9bbc8#cargo -c cargo run -- --flake /home/mschwaig/git/robotnix-config#defaultPackage.x86_64-linux --max-cores 48 --max-cores-per-job 4 check
RUST_LOG=warn nix shell nixpkgs/4a01ca36d6bfc133bc617e661916a81327c9bbc8#cargo -c cargo run -- --flake /home/mschwaig/git/robotnix-config#defaultPackage.x86_64-linux report

with

https://github.com/mschwaig/r13y.com/tree/b376905f8acf865350b6ce86649102c6eb1c172b (use-nix-command branch).