Skip to content

Commit

Permalink
Stream files from store instead of buffering them
Browse files Browse the repository at this point in the history
When an artifact is requested from hydra the output is first copied
from the nix store into memory and then sent as a response, delaying
the download and taking up significant amounts of memory.

As reported in NixOS#1357

Instead of calling a command and blocking while reading in the entire
output, this adds read_into_socket(). the function takes a
command, starting a subprocess with that command, returning a file
descriptor attached to stdout.
This file descriptor is then by responsebuilder of Catalyst to steam
the output directly
  • Loading branch information
71rd authored and yu-re-ka committed Aug 13, 2024
1 parent f1b552e commit 459aa0a
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 1 deletion.
2 changes: 1 addition & 1 deletion src/lib/Hydra/Controller/Build.pm
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,7 @@ sub serveFile {
# XSS hole.
$c->response->header('Content-Security-Policy' => 'sandbox allow-scripts');

$c->stash->{'plain'} = { data => grab(cmd => ["nix", "--experimental-features", "nix-command",
$c->stash->{'plain'} = { data => readIntoSocket(cmd => ["nix", "--experimental-features", "nix-command",
"store", "cat", "--store", getStoreUri(), "$path"]) };

# Detect MIME type.
Expand Down
12 changes: 12 additions & 0 deletions src/lib/Hydra/Helper/Nix.pm
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ our @EXPORT = qw(
jobsetOverview
jobsetOverview_
pathIsInsidePrefix
readIntoSocket
readNixFile
registerRoot
restartBuilds
Expand Down Expand Up @@ -406,6 +407,17 @@ sub pathIsInsidePrefix {
return $cur;
}

sub readIntoSocket{
my (%args) = @_;
my $sock;

eval {
my $x= join(" ", @{$args{cmd}});
open($sock, "-|", $x) or die q(failed to open socket from command:\n $x);
};

return $sock;
}



Expand Down

0 comments on commit 459aa0a

Please sign in to comment.