Support other package sets

[infinisil]

We want to use the pkgs/by-name convention for other package sets, e.g. NixOS/nixpkgs#299347.

Here's a rough idea how this could look like:

Package set config

Define some Nix config file to determine which package sets exist:

# pkgs/top-level/by-name-sets-config.nix
{
  packageSets = {
    toplevel = {
      attrPath = [ ];
      path = ./pkgs/by-name;
      shardLength = 2;
    };
    mySet = {
      # Any attrpath that will contain the attributes
      attrPath = [ "mySet" ];
      path = ./pkgs/mySet/by-name;
      # No sharding
      shardLength = null;
    };
  };
}

Read the packages in Nix

Create a Nix file to automatically call all of them like the current by-name-overlay.nix:

# pkgs/top-level/by-name-sets.nix
{
  toplevel = callPackage: {
    hello = <...>;
  };
  mySet = callPackage: {
    hi = <...>;
  };
}

For each package set declared, define it by importing by-name-sets.nix, like

mySet = lib.makeScope pkgs.newScope (self:
  (import ./pkgs/top-level/by-name-sets.nix).mySet self.callPackage
) 

Check the packages with the tool

Adjust this tool to read a by-name-sets-config.nix file for determining what it needs to check and how.

[infinisil]

Had a chat today with @DMills27, who's interested in contributing. I took some notes to break down this task:

1. Make by-name-overlay.nix be synced with the tool
   • Move by-name-overlay.nix into nixpkgs-check-by-name
   • Make sure it can accept a config argument in the future
   • Distribute this file in the releases of nixpkgs-check-by-name
     • Not necessary, Github always distributes the source tarball
   • In Nixpkgs, the update-pinned-tool.sh should be updated to also fetch the Nix file from the release.
2. Add a config file with a configurable prefix length
   • Make the tool read a config.json (given as a CLI arg) and change behavior depending on it
   • Make by-name-overlay.nix read a config.json and change behavior depending on
   • See here for how to design the JSON

     {
       "shard_length": 2
     }

   • In the future perhaps a module along with it, not necessary though

     {
       options.shard_length = lib.mkOption {
         type = nullOr int;
         # default = 2;
       }
     }

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/ci-will-soon-enforce-pkgs-by-name-for-new-packages/38098/8

[fgaz]

+1! Here is another by-name package set: NixOS/nixpkgs#344716