nixos programs.sqlite missing large numbers of packages

[benwbooth]

I'm writing a custom bash command_not_found_handle that does a lookup into /nix/var/nix/profiles/per-user/root/channels/nixos/programs.sqlite to find out which nix package to install for a given program. This is a sqlite database that's generated on the nixos cache servers, and is distributed via the nixos channel.

I'm noticing that a lot of very commonly used programs are missing from both the unstable nixos and the 20.09 nixos versions. For example, the "vim" binary is not found. Looking at the 20.03 version, I can find vim, but the record count in the programs table seems suspiciously low, around ~40,000 program records for all recent versions of programs.sqlite. The nix package database at search.nixos.org claims to have over ~80,000 packages so I would expect at least 80,000 records or more in the programs table.

I noticed a similar issue cropping up with nix-index: nix-community/nix-index#163

In the case of nix-index, it looks like there are mal-formed JSON records being generated at cache.nixos.org, which is causing nix-index to skip the creation of records for many popular packages, such as vim and emacs. I'm not sure if this is the same problem afflicting programs.sqlite, but it may be a possibility.

Is this the right issue tracker for this problem, or is there a more appropriate place to put this? It looks like the cache.nixos.org server scripts that generate JSON files need to be fixed.

Thanks!

[worldofpeace]

Is this the right issue tracker for this problem, or is there a more appropriate place to put this? It looks like the cache.nixos.org server scripts that generate JSON files need to be fixed.

The programs and scripts that do the job are at https://github.com/NixOS/nixos-channel-scripts/blob/7c442a2f67c77344a71e5aae7e4cd2a1554420a9/generate-programs-index.cc. Will have someone move this to that tracker.

[grahamc]

I'm not sure ... it seems to work here;

[grahamc@Petunia:~]$ vim
The program ‘vim’ is currently not installed. It is provided by
several packages. You can install it by typing one of the following:
  nix-env -iA nixos.vim
  nix-env -iA nixos.vimHugeX

this is on 20.09.

[benwbooth]

@grahamc are you using the most recent version of nixos-20.09? I was able to reproduce this again on my nixos machine with 20.09:

$ nix-channel --add https://nixos.org/channels/nixos-20.09 nixos
$ nix-channel --update
$ nix-channel --list
nixos https://nixos.org/channels/nixos-20.09
$ vim
vim: command not found
$ gnvim
The program 'gnvim' is not in your PATH. It is provided by several packages.
You can make it available in an ephemeral shell by typing one of the following:
  nix-shell -p gnvim
  nix-shell -p gnvim-unwrapped

I can also try querying the programs:sqlite database directly:

$ curl https://releases.nixos.org/nixos/20.09/nixos-20.09.3223.38eaa62f283/nixexprs.tar.xz |tar xv 'nixos-*/programs.sqlite'
$ sqlite3 -separator $'\t' nixos-*/programs.sqlite "select * from programs where name like '%vim%'" |cut -f 1 |sort
avimerge
dipy_fit_ivim
dipy_fit_ivim
gnvim
gnvim
govim
gvimacs
metals-vim
metals-vim-lsc
mkenvimage
nvim
nvim-qt
nvimgdiff
pacvim
vimacs
vimacsdiff
vimb
vimcat
vimcat
vimdot
vimdot
vimer
vimpager
vimpager
vimpc
vimwiki_markdown

[wmertens]

@benwbooth It seems to be ok now?

$ sqlite3 /nix/var/nix/profiles/per-user/root/channels/nixos/programs.sqlite 'select count(*) from programs'
104572

Also would like to know your progress on your command-not-found handler :)

[wamserma]

bumping this, as I made a similar observation
It seems that https://releases.nixos.org/?prefix=nixos/22.11/nixos-22.11.1777.cdead16a444/ served a nixexprs.tar.xz with an incomplete programs.sqlite file for some time on 2013-01-21.

This JSON snippet was logged into a git repo via a GitHub action at Sat Jan 21 02:03:52 2023 +0000:

  "cdead16a444a3e5de7bc9b0af8e198b11bb01804": {
    "name": "nixos-22.11.1777.cdead16a444",
    "url": "/nixos/22.11-small/nixos-22.11.1777.cdead16a444/nixexprs.tar.xz",
    "nixexprs_hash": "caa41113175d1a6f30efe30616fca0b53991540ede301d5a2a74ee89715195a2",
    "programs_sqlite_hash": "4c33dce9eca05e5a912ce960eceee609318d415b80ccd3575ccf3a0ebb921a69"
  }

A few hours later I did a nixos-rebuild and programs not found was not working as expected. programs.sqlite had only 300K in size and about 3700 entries. The sha256 of the file on my system matches the value in the JSON above. There might have been an issue in generating programs.sqlite and things might have been redeployed to releases.nixos.org, but we would need someone with access to logs to confirm that.
ping @domenkozar @grahamc @edolstra