Use `nix-eval-jobs` and delete `hydra-eval-jobs` #1421

Ericson2314 · 2024-11-13T21:50:02Z

nix-eval-jobs was forked from hydra-eval-jobs originally, for others to use. That's great! But maintaining two copies is no good. But we use that.

Original author is @delroth, for Lix's Hydra, thank you!

Draft because some tests fail. I am not sure why. Maybe there are fixes on the Lix branch that can be cherry-picked to fix? Not sure!

zowoq · 2024-11-13T23:06:59Z

src/script/hydra-eval-jobset

-                "--flake", $flakeRef,
-                "--gc-roots-dir", getGCRootsDir,
-                "--max-jobs", 1);
+        @cmd = ("nix-eval-jobs", "--flake", $flakeRef . '#hydraJobs');


Uses hydraJobs here and the original hydra-eval-jobs has hydraJobs and checks:

hydra/src/hydra-eval-jobs/hydra-eval-jobs.cc

Lines 148 to 152 in f974891

auto aHydraJobs = vOutputs->attrs()->get(state.symbols.create("hydraJobs"));

if (!aHydraJobs)

aHydraJobs = vOutputs->attrs()->get(state.symbols.create("checks"));

if (!aHydraJobs)

throw Error("flake '%s' does not provide any Hydra jobs or checks", flakeRef);

Ah found more commits that are relevant.

The Lix commits don't address this issue, they only have support for hydraJobs.

I think it would be important to keep the checks supports

Ericson2314 · 2024-11-13T23:47:35Z

I left a comment on nix-community/nix-eval-jobs#330, I think that is causing the failure. CC @Mic92.

Ericson2314 · 2024-11-14T02:51:59Z

nix-community/nix-eval-jobs#336 this I thought would help but is not sufficient hmm

Ericson2314 · 2024-11-14T03:05:52Z

( STDERR )  job  1    {"attr":"caDependingOnCA","attrPath":["caDependingOnCA"],"error":"error: derivation 'caDependingOnCA' does not have valid outputs: error:\n              â¦ while evaluating the output path of a derivation\n                at <nix/derivation-internal.nix>:19:9:\n                  18|       value = commonAttrs // {\n                  19|         outPath = builtins.getAttr outputName strict;\n                    |         ^\n                  20|         drvPath = strict.drvPath;\n\n              error: path '/0rmq7bvk2raajd310spvd416f2jajrabcg6ar706gjbd6b8nmvks' is not in the Nix store"}
( STDERR )  job  1    {UNKNOWN}: asdfasdf at /home/jcericson/src/hydra/master/src/script/hydra-eval-jobset line 412. at /nix/store/m07199rzw9ja8v6ba75r4ak5iy92ps9i-hydra-perl-deps/lib/perl5/site_perl/5.38.2/Catalyst/Model/DBIC/Schema.pm line 526

ah nix-eval-jobs needs more changes.

dasJ · 2024-11-14T10:51:18Z

A relevant thing to consider here is whether we want an official NixOS project to be dependant on a Nix Community project. As of now (afaik), the NixOS GitHub namespace is mostly self-contained. I see these solutions:

Close this PR
Migrate nix-eval-jobs to the NixOS namespace (or even nix repo?)
Just live with the fact

I personally prefer the latter two options over the first one since it would get rid of code duplication and remove a component from Hydra that only a few people are proficient in and that gets a lot less maintenance than nix-eval-jobs.

Maybe that's a question for the new SC?

Mic92 · 2024-11-14T11:10:05Z

Happy to move the project to the NixOS org if I can keep maintaining it there. We would loose aarch64-linux builds because those runners are currently sponsored by namespace in nix-community. I can attach my buildbot-nix installation to this org to compensate.

Mic92 · 2024-11-14T11:17:36Z

Maybe that's a question for the new SC?

That's more a technical question for the Hydra/Nix team, unless we feel to overwhelmed what to do and need to escalate.

Ericson2314 · 2024-11-14T18:24:46Z

OK it builds now (CI should say so in a moment).

Ericson2314 · 2024-11-14T18:29:33Z

Keeping draft just until nix-community/nix-eval-jobs#337 is merged.

Ericson2314 · 2024-11-15T16:17:43Z

OK undrafting because we're back on nix-eval-jobs main.

@dasJ I am fine resolving the nix-community thing later, because Hydra is a funny project --- it's official because history and because hydra.nixos.org, not because it is up to the quality we generally expect for 3rd-party use.

Mindavi

The perl bits look generally ok to me but not my expertise. Maybe put somewhere a bit clearer that the constituents are not supported anymore. That's not used on hydra.nixos.org, right?

Mic92 · 2024-11-16T13:03:54Z

src/script/hydra-eval-jobset

-                foreach my $drvPath (@{$job->{constituents}}) {
-                    my $constituent = $drvPathToId{$drvPath};
-                    if (defined $constituent) {
-                        $db->resultset('AggregateConstituents')->update_or_create({aggregate => $x->{id}, constituent => $constituent->{id}});


@Ericson2314 are we depending on this in hydra.nixos.org?

What is this feature actually about?

According to the commit message we are not.

But good monitoring this.

I find the statement that we weren't using consituents surprising, nixpkgs uses it in both code and documentation:

doc/preface.chapter.md:[this table](https://hydra.nixos.org/job/nixpkgs/trunk/unstable#tabs-constituents) lib/customisation.nix: constituents = map hydraJob (flatten drv.constituents); nixos/doc/manual/release-notes/rl-2305.section.md:- Deprecated `xlibsWrapper` transitional package has been removed in favour of direct use of its constituents: `xorg.libX11`, `freetype` and others. nixos/release-combined.nix: constituents = pkgs.lib.concatLists [ nixos/release-small.nix: constituents = lib.flatten [ pkgs/build-support/release/default.nix: { name, constituents, meta ? { } }: pkgs/build-support/release/default.nix: { inherit constituents meta; pkgs/build-support/release/default.nix: echo $constituents > $out/nix-support/hydra-aggregate-constituents pkgs/build-support/release/default.nix: for i in $constituents; do pkgs/build-support/release/default.nix: - constituents: a list of derivations on which the channel success depends. pkgs/build-support/release/default.nix: constituents = [ foo bar baz ]; pkgs/build-support/release/default.nix: { name, src, constituents ? [], meta ? {}, isNixOS ? true, ... }@args: pkgs/build-support/release/default.nix: echo $constituents > "$out/nix-support/hydra-aggregate-constituents" pkgs/build-support/release/default.nix: for i in $constituents; do pkgs/build-support/rust/build-rust-crate/test/default.nix: constituents = builtins.attrValues tests; pkgs/top-level/release-haskell.nix: # to `releaseTools.aggregate` as constituents. pkgs/top-level/release-haskell.nix: constituents = pkgs/top-level/release-haskell.nix: constituents = accumulateDerivations pkgs/top-level/release-haskell.nix: constituents = accumulateDerivations [ pkgs/top-level/release-haskell.nix: constituents = accumulateDerivations [ pkgs/top-level/release.nix: constituents = [ pkgs/top-level/release.nix: constituents = pkgs/top-level/release-python.nix: constituents = [ README.md:* [Tests for unstable/master](https://hydra.nixos.org/job/nixos/trunk-combined/tested#tabs-constituents) README.md:* [Tests for the NixOS 24.05 release](https://hydra.nixos.org/job/nixos/release-24.05/tested#tabs-constituents)

That sounds like a blocking issue to me

nix-community/nix-eval-jobs@a4441fe This is the commit we'd need to revert of nix-eval-jobs, I guess. Hmm...

Hmm, so there are multiple types of constituent jobs. I am now less sure that we are still using this type.

dasJ

As stated before, I do find this move to be a good idea. Here are some things I noticed while reading the code

dasJ · 2024-11-16T20:55:16Z

flake.nix

@@ -8,14 +8,20 @@
  inputs.nix.inputs.nixpkgs.follows = "nixpkgs";
  inputs.nix.inputs.libgit2.follows = "libgit2";

+  inputs.nix-eval-jobs.url = "github:nix-community/nix-eval-jobs";


Can we somehow get that rev into the footer in src/root/layout.tt? I'm thinking something like Hydra 1.0 (using nix-eval-jobs abcdef and nix-2.18)

dasJ · 2024-11-16T20:56:20Z

src/script/hydra-eval-jobset

-                "--flake", $flakeRef,
-                "--gc-roots-dir", getGCRootsDir,
-                "--max-jobs", 1);
+        @cmd = ("nix-eval-jobs", "--flake", $flakeRef . '#hydraJobs');


I think it would be important to keep the checks supports

dasJ · 2024-11-16T20:57:18Z

src/script/hydra-eval-jobset

-    push @cmd, "--no-allow-import-from-derivation" if $config->{allow_import_from_derivation} // "true" ne "true";
+    push @cmd, ("--gc-roots-dir", getGCRootsDir);
+    push @cmd, ("--max-jobs", 1);
+    push @cmd, "--meta";


Wills there two parameters change anything relevant?

$ git log -G --max-memory-size commit 4b107e6ff36bd89958fba36e0fe0340903e7cd13 Author: Pierre Bourdon <[email protected]> Date: Mon Jul 22 23:16:29 2024 +0200 hydra-eval-jobset: pass --workers and --max-memory-size to n-e-j Lost in the h-e-j -> n-e-j migration, causing evaluation to always be single threaded and limited to 4GiB RAM. Follow the config settings like h-e-j used to do (via C++ code). commit 6f1d68bda483b2ed3b1c1f0a696e8fe300108853 Author: Eelco Dolstra <[email protected]> Date: Wed Feb 19 20:36:52 2020 +0100 Revert "hydra-eval-jobs -> nix eval-hydra-jobs" This reverts commit 345512a6d0743eaac9d451833ecc1a71d12809c9. commit 345512a6d0743eaac9d451833ecc1a71d12809c9 Author: Eelco Dolstra <[email protected]> Date: Sat Feb 15 00:03:58 2020 +0100 hydra-eval-jobs -> nix eval-hydra-jobs

It's purpose seems to be to change a setting that hydra-eval-jobs was changing programmatically before.

src/script/hydra-eval-jobset

dasJ · 2024-11-16T20:59:47Z

src/script/hydra-eval-jobset

-    print STDERR "$stderr";
+            if (defined $err) {
+                print STDERR "$err";
+                undef $err;


Not sure about the API, but is it a good idea to undef this? Maybe set it to an emtpy string instead? Feels like that would die horribly if the process decides to do stdout, then stderr, then stdout again. Not sure why it would, but better safe than sorry

I think every loop it will define them if they need to exists. Perl is weird! Why can't they just return a tuple!

dasJ · 2024-11-16T21:03:40Z

src/script/hydra-eval-jobset

@@ -709,36 +744,20 @@ sub checkJobsetWrapped {
        return;
    }

-    die "Jobset contains a job with an empty name. Make sure the jobset evaluates to an attrset of jobs.\n"


Is this a relevant check? Kind of surprising to see it gone

dasJ · 2024-11-16T21:05:09Z

src/script/hydra-eval-jobset

-                foreach my $drvPath (@{$job->{constituents}}) {
-                    my $constituent = $drvPathToId{$drvPath};
-                    if (defined $constituent) {
-                        $db->resultset('AggregateConstituents')->update_or_create({aggregate => $x->{id}, constituent => $constituent->{id}});


I find the statement that we weren't using consituents surprising, nixpkgs uses it in both code and documentation:

doc/preface.chapter.md:[this table](https://hydra.nixos.org/job/nixpkgs/trunk/unstable#tabs-constituents) lib/customisation.nix: constituents = map hydraJob (flatten drv.constituents); nixos/doc/manual/release-notes/rl-2305.section.md:- Deprecated `xlibsWrapper` transitional package has been removed in favour of direct use of its constituents: `xorg.libX11`, `freetype` and others. nixos/release-combined.nix: constituents = pkgs.lib.concatLists [ nixos/release-small.nix: constituents = lib.flatten [ pkgs/build-support/release/default.nix: { name, constituents, meta ? { } }: pkgs/build-support/release/default.nix: { inherit constituents meta; pkgs/build-support/release/default.nix: echo $constituents > $out/nix-support/hydra-aggregate-constituents pkgs/build-support/release/default.nix: for i in $constituents; do pkgs/build-support/release/default.nix: - constituents: a list of derivations on which the channel success depends. pkgs/build-support/release/default.nix: constituents = [ foo bar baz ]; pkgs/build-support/release/default.nix: { name, src, constituents ? [], meta ? {}, isNixOS ? true, ... }@args: pkgs/build-support/release/default.nix: echo $constituents > "$out/nix-support/hydra-aggregate-constituents" pkgs/build-support/release/default.nix: for i in $constituents; do pkgs/build-support/rust/build-rust-crate/test/default.nix: constituents = builtins.attrValues tests; pkgs/top-level/release-haskell.nix: # to `releaseTools.aggregate` as constituents. pkgs/top-level/release-haskell.nix: constituents = pkgs/top-level/release-haskell.nix: constituents = accumulateDerivations pkgs/top-level/release-haskell.nix: constituents = accumulateDerivations [ pkgs/top-level/release-haskell.nix: constituents = accumulateDerivations [ pkgs/top-level/release.nix: constituents = [ pkgs/top-level/release.nix: constituents = pkgs/top-level/release-python.nix: constituents = [ README.md:* [Tests for unstable/master](https://hydra.nixos.org/job/nixos/trunk-combined/tested#tabs-constituents) README.md:* [Tests for the NixOS 24.05 release](https://hydra.nixos.org/job/nixos/release-24.05/tested#tabs-constituents)

dasJ · 2024-11-17T08:17:45Z

flake.nix

@@ -8,14 +8,20 @@
  inputs.nix.inputs.nixpkgs.follows = "nixpkgs";
  inputs.nix.inputs.libgit2.follows = "libgit2";

+  inputs.nix-eval-jobs.url = "github:nix-community/nix-eval-jobs";
+  inputs.nix-eval-jobs.inputs.nixpkgs.follows = "nixpkgs";


Something I forgot in the last review: I think we should also force the nix version of hydra into the package, otherwise we might depend on 2 nix versions

That's actually a good one and could be a problem if hydra or nix-eval-jobs doesn't keep up with nix releases.

I can maintain release branches for different nix versions again if this is needed.

Ericson2314 · 2024-11-17T21:23:57Z

Thanks for this good review, both of you. In light of these things. I am going to try to flip the order: re-draft this, and get Mesonification done first.

Original commit message: > There are some known regressions regarding local testing setups - since > everything was kinda half written with the expectation that build dir = > source dir (which should not be true anymore). But everything builds and > the test suite runs fine, after several hours spent debugging random > crashes in libpqxx with MALLOC_PERTURB_... I have not experienced regressions with local testing. (cherry picked from commit 4b886d9)

(cherry picked from commit 684cc50)

(cherry picked from commit 6d4ccff)

(cherry picked from commit ed7c587)

nix-eval-jobs streams output, unlike hydra-eval-jobs. Now that we've migrated, we can use this to: 1. Use less RAM by avoiding buffering a whole eval's worth of metadata into a Perl string and an array of JSON objects. 2. Make evals latency a bit lower by allowing the queue runner to start ingesting builds faster. (cherry picked from commit b0e9b4b)

Lost in the h-e-j -> n-e-j migration, causing evaluation to always be single threaded and limited to 4GiB RAM. Follow the config settings like h-e-j used to do (via C++ code). (cherry picked from commit 4b107e6)

The feature cannot easily be ported to nix-eval-jobs since it requires deep integration into the evaluator, and h.n.o doesn't use it. Later more of this will be ripped out. (cherry picked from commit 370a4bf)

Ericson2314 marked this pull request as draft November 13, 2024 21:50

Ericson2314 mentioned this pull request Nov 13, 2024

autotools -> meson #1422

Open

zowoq reviewed Nov 13, 2024

View reviewed changes

Ericson2314 mentioned this pull request Nov 13, 2024

improve compatibility if ca derivations are present nix-community/nix-eval-jobs#330

Merged

Ericson2314 mentioned this pull request Nov 14, 2024

Ensure that we still create map kv pairs for floating CA outputs nix-community/nix-eval-jobs#336

Merged

Ericson2314 mentioned this pull request Nov 14, 2024

PackageInfo::queryOutputs should not trip up on placeholders NixOS/nix#11885

Open

Ericson2314 force-pushed the nix-eval-jobs branch from 498b283 to d0cfbed Compare November 14, 2024 18:21

Ericson2314 mentioned this pull request Nov 14, 2024

Make fallback logic for CA derivations more flexible nix-community/nix-eval-jobs#337

Merged

Ericson2314 marked this pull request as ready for review November 14, 2024 18:28

Ericson2314 marked this pull request as draft November 14, 2024 18:28

Ericson2314 marked this pull request as ready for review November 15, 2024 15:57

Ericson2314 force-pushed the nix-eval-jobs branch from d0cfbed to 8a49ecb Compare November 15, 2024 15:57

Mindavi approved these changes Nov 15, 2024

View reviewed changes

Mic92 reviewed Nov 16, 2024

View reviewed changes

dasJ requested changes Nov 16, 2024

View reviewed changes

dasJ reviewed Nov 17, 2024

View reviewed changes

Ericson2314 marked this pull request as draft November 17, 2024 21:24

delroth added 3 commits November 17, 2024 17:24

flake: add nix-eval-jobs as input

4efb769

(cherry picked from commit 684cc50)

hydra-eval-jobset: use nix-eval-jobs instead of hydra-eval-jobs

afd487d

(cherry picked from commit 6d4ccff)

hydra-eval-jobs: remove, replaced by nix-eval-jobs

d27e7cd

(cherry picked from commit ed7c587)

Ericson2314 force-pushed the nix-eval-jobs branch from 8a49ecb to a3e3fe9 Compare November 17, 2024 23:12

delroth added 3 commits November 17, 2024 18:26

hydra-eval-jobset: pass --workers and --max-memory-size to n-e-j

f2bcece

Lost in the h-e-j -> n-e-j migration, causing evaluation to always be single threaded and limited to 4GiB RAM. Follow the config settings like h-e-j used to do (via C++ code). (cherry picked from commit 4b107e6)

treewide: start removing tests related to constituents

d2227af

The feature cannot easily be ported to nix-eval-jobs since it requires deep integration into the evaluator, and h.n.o doesn't use it. Later more of this will be ripped out. (cherry picked from commit 370a4bf)

Ericson2314 force-pushed the nix-eval-jobs branch from a3e3fe9 to d2227af Compare November 17, 2024 23:26

	auto aHydraJobs = vOutputs->attrs()->get(state.symbols.create("hydraJobs"));
	if (!aHydraJobs)
	aHydraJobs = vOutputs->attrs()->get(state.symbols.create("checks"));
	if (!aHydraJobs)
	throw Error("flake '%s' does not provide any Hydra jobs or checks", flakeRef);

Use nix-eval-jobs and delete hydra-eval-jobs #1421

Are you sure you want to change the base?

Use nix-eval-jobs and delete hydra-eval-jobs #1421

Conversation

Ericson2314 commented Nov 13, 2024

zowoq Nov 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zowoq Nov 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ericson2314 commented Nov 13, 2024

Ericson2314 commented Nov 14, 2024

Ericson2314 commented Nov 14, 2024

dasJ commented Nov 14, 2024 • edited Loading

Mic92 commented Nov 14, 2024 • edited Loading

Mic92 commented Nov 14, 2024

Ericson2314 commented Nov 14, 2024 • edited Loading

Ericson2314 commented Nov 14, 2024

Ericson2314 commented Nov 15, 2024

Mindavi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ericson2314 Nov 16, 2024 • edited Loading

Choose a reason for hiding this comment

dasJ Nov 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dasJ left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ericson2314 Nov 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dasJ Nov 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ericson2314 commented Nov 17, 2024

Use `nix-eval-jobs` and delete `hydra-eval-jobs` #1421

Use `nix-eval-jobs` and delete `hydra-eval-jobs` #1421

zowoq Nov 13, 2024 •

edited

Loading

zowoq Nov 14, 2024 •

edited

Loading

dasJ commented Nov 14, 2024 •

edited

Loading

Mic92 commented Nov 14, 2024 •

edited

Loading

Ericson2314 commented Nov 14, 2024 •

edited

Loading

Ericson2314 Nov 16, 2024 •

edited

Loading

dasJ Nov 16, 2024 •

edited

Loading

dasJ left a comment •

edited

Loading

Ericson2314 Nov 17, 2024 •

edited

Loading

dasJ Nov 16, 2024 •

edited

Loading