Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add build-redirect endpoint #1290

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Conversation

mkenigs
Copy link

@mkenigs mkenigs commented Jun 22, 2023

This adds a /build-redirect/$basename endpoint that redirects to the first build of /nix/store/$basename. The first build is determined by taking the minimum of build ID.

Currently, the only way to find the build for a store path is using the /search endpoint, but a request to that endpoint is so slow it will timeout:

$ time curl -H "Accept: application/json" https://hydra.nixos.org/search?query=/nix/store/v02pl5dhayp8jnz8ahdvg5vi71s8xc6g-hello-2.12.1
{"error":"DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR:  canceling statement due to statement timeout [for Statement \"SELECT me.id, me.finished, me.timestamp, me.jobset_id, me.job, me.nixname, me.description, me.drvpath, me.system, me.license, me.homepage, me.maintainers, me.maxsilent, me.timeout, me.ischannel, me.iscurrent, me.priority, me.globalpriority, me.starttime, me.stoptime, me.iscachedbuild, me.buildstatus, me.size, me.closuresize, me.releasename, me.keep, me.notificationpendingsince FROM builds me LEFT JOIN buildoutputs buildoutputs ON buildoutputs.build = me.id WHERE ( buildoutputs.path ILIKE ? ) ORDER BY id desc LIMIT ?\" with ParamValues: 1='%/nix/store/v02pl5dhayp8jnz8ahdvg5vi71s8xc6g-hello-2.12.1%', 2='10'] at /nix/store/sw39pc6gwfwadr0f3nz6bzsf63qwpcwg-hydra-0.1.20221124.0118770/libexec/hydra/lib/Hydra/Controller/Root.pm line 486\n"}
real	0m22.935s
user	0m0.070s
sys	0m0.019s

The new endpoint uses an index on BuildOutputs.path. Here is an explain of the SQL query executed by the new endpoint:

hydra=# explain analyze SELECT MIN( me.build ) FROM buildoutputs me WHERE ( path = '...store path...' );
                                                             QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=12.50..12.51 rows=1 width=4) (actual time=0.024..0.025 rows=1 loops=1)
   ->  Bitmap Heap Scan on buildoutputs me  (cost=4.03..12.49 rows=4 width=4) (actual time=0.019..0.020 rows=2 loops=1)
         Recheck Cond: (path = '...store path...'::text)
         Heap Blocks: exact=1
         ->  Bitmap Index Scan on indexbuildoutputspath  (cost=0.00..4.03 rows=4 width=0) (actual time=0.010..0.010 rows=2 loops=1)
               Index Cond: (path = '...store path...'::text)
 Planning Time: 0.134 ms
 Execution Time: 0.060 ms
(8 rows)

This adds a `/build-redirect/$basename` endpoint that redirects to the
first build of `/nix/store/$basename`. The first build is determined by
taking the minimum of build ID.

Currently, the only way to find the build for a store path is using the
`/search` endpoint, but a request to that endpoint is so slow it will
timeout:

```
$ time curl -H "Accept: application/json" https://hydra.nixos.org/search?query=/nix/store/v02pl5dhayp8jnz8ahdvg5vi71s8xc6g-hello-2.12.1
{"error":"DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR:  canceling statement due to statement timeout [for Statement \"SELECT me.id, me.finished, me.timestamp, me.jobset_id, me.job, me.nixname, me.description, me.drvpath, me.system, me.license, me.homepage, me.maintainers, me.maxsilent, me.timeout, me.ischannel, me.iscurrent, me.priority, me.globalpriority, me.starttime, me.stoptime, me.iscachedbuild, me.buildstatus, me.size, me.closuresize, me.releasename, me.keep, me.notificationpendingsince FROM builds me LEFT JOIN buildoutputs buildoutputs ON buildoutputs.build = me.id WHERE ( buildoutputs.path ILIKE ? ) ORDER BY id desc LIMIT ?\" with ParamValues: 1='%/nix/store/v02pl5dhayp8jnz8ahdvg5vi71s8xc6g-hello-2.12.1%', 2='10'] at /nix/store/sw39pc6gwfwadr0f3nz6bzsf63qwpcwg-hydra-0.1.20221124.0118770/libexec/hydra/lib/Hydra/Controller/Root.pm line 486\n"}
real	0m22.935s
user	0m0.070s
sys	0m0.019s
```

The new endpoint uses an index on BuildOutputs.path. Here is an explain
of the SQL query executed by the new endpoint:

```
hydra=# explain analyze SELECT MIN( me.build ) FROM buildoutputs me WHERE ( path = '...store path...' );
                                                             QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=12.50..12.51 rows=1 width=4) (actual time=0.024..0.025 rows=1 loops=1)
   ->  Bitmap Heap Scan on buildoutputs me  (cost=4.03..12.49 rows=4 width=4) (actual time=0.019..0.020 rows=2 loops=1)
         Recheck Cond: (path = '...store path...'::text)
         Heap Blocks: exact=1
         ->  Bitmap Index Scan on indexbuildoutputspath  (cost=0.00..4.03 rows=4 width=0) (actual time=0.010..0.010 rows=2 loops=1)
               Index Cond: (path = '...store path...'::text)
 Planning Time: 0.134 ms
 Execution Time: 0.060 ms
(8 rows)
```
@mkenigs
Copy link
Author

mkenigs commented Jul 24, 2023

@grahamc @edolstra does the general direction of this patch look like something that could be merged at some point?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant