Improve 'ghcup compile ghc' interface

[hasufell]

The main issues are:

1. passing additional flags to build configure...
2. but also passing additional flags to bindist configure
3. and also passing additional flags to hadrian itself

Currently we support the first case via -- longopts. But that can only cover one use case.

This is particularly important for cross builds, which may need different than default settings.

@hsyl20

[hasufell]

Should probably also make hadrian default or detect whether a make build system exists. --hadrian flag should only be required if there's both make and hadrian.

[chreekat]

Should probably also make hadrian default or detect whether a make build system exists. --hadrian flag should only be required if there's both make and hadrian.

Is it possible to say "Use make for <9.6 and Hadrian for >= 9.6"?

And fwiw I think 9.4 was the only series that officially supported both, although I also think Hadrian was pretty immature then.

[hasufell]

Is it possible to say "Use make for <9.6 and Hadrian for >= 9.6"?

Currently, my idea is just to look for the hadrian files and if they're not there, assume make:

https://github.com/haskell/ghcup-hs/pull/854/files#diff-3b9b75e85b8e396244d24bf04ef933cae33b4d20df6f5889304bfb36696c43d6R957

The choice can be made explicit with --hadrian | --make.

[chreekat]

I don't know that it's valuable to make it an option to users if 9.4 is the only version that officially supported both. <9.4, --hadrian might work but will be buggy (and the bugs will never get fixed); >=9.6, --make will fail.

[hasufell]

Yes, but GHC has made it extra hard to detect which version you're actually compiling: https://gitlab.haskell.org/ghc/ghc/-/issues/22322

The fix was backported, but I have no clear overview at the moment what versions work and which don't.

As such I'm moving away from logic that depends on version numbers. If you try to compile 9.6.1 with make, that may fail. But maybe not, because you're using custom patched bindists, a different branch etc (remember you can already supply patches before the build... that's explicitly supported).

Dynamic detection seems more flexible.

What we probably want is a more descriptive error in case you try to build 9.6.1 with make: "The make build system has been removed since 9.6. You probably want to omit '--make'."

[hasufell]

And I'll probably support the VERSION file, in case it exists.

[hasufell]

What's missing now is the question on how to support additional arguments to ./configure and hadrian ....

If we compile, we have two configure phases:

1. configure before compilation
2. configure of the bindist

It's not clear whether they support the exact same arguments. Or if you'd even want that.

For hadrian I think we're mostly good with the --flavour=foo switch, since many things are now flavour transformers.

[0rphee]

EDIT: Sorry, i was running this with github actions and it seems github actions doesn't display a part of the logs. I'll look into it

Hi, thanks again for so awesome tool, and your time.

(i think my issues are related to this ticket) I'm trying to build a cross-compiler from host x86_64-macos to target aarch64-macos with ghcup compile ghc

I've tried multiple combinations of ghc target versions (8.10.7, 9.0.2, 9.2.8, 9.4.5, 9.6.2) (all with bootstrapping version 9.4.5), and additional flags (particularly --jobs), without any success. I am mostly asking if I should keep trying to build using ghcup or I just should try directly from ghc source.

Here I leave some logs w/errors while trying to compile.
ghcup compile ghc -v 9.6.2 -b 9.4.5 "" -x aarch64-apple-darwin with [ Error ] [�]8;;https://errors.haskell.org/messages/GHCup-00841�\GHCup-00841�]8;;�\] Process "make" with arguments [] failed with exit code 2.
ghcup compile ghc -v 9.2.8 -b 9.4.5 "" -x aarch64-apple-darwin with

[ Error ] [�]8;;https://errors.haskell.org/messages/GHCup-00841�\GHCup-00841�]8;;�\] Process "sh" with arguments ["./configure",
[ ...   ]                              "--target=aarch64-apple-darwin",
[ ...   ]                              "--prefix=/Users/runner/.ghcup/ghc/aarch64-apple-darwin-9.2.8",
[ ...   ]                              ""] failed with exit code 1. 

ghcup compile ghc -v 9.0.2 -b 9.4.5 "" -x aarch64-apple-darwin with

[ Error ] [�]8;;https://errors.haskell.org/messages/GHCup-00841�\GHCup-00841�]8;;�\] Process "sh" with arguments ["./configure",
[ ...   ]                              "--target=aarch64-apple-darwin",
[ ...   ]                              "--prefix=/Users/runner/.ghcup/ghc/aarch64-apple-darwin-9.0.2",
[ ...   ]                              ""] failed with exit code 1.

ghcup compile ghc -v 8.10.7 -b 9.4.5 "" -x aarch64-apple-darwin with

[ Error ] [�]8;;https://errors.haskell.org/messages/GHCup-00841�\GHCup-00841�]8;;�\] Process "sh" with arguments ["./configure",
[ ...   ]                              "--target=aarch64-apple-darwin",
[ ...   ]                              "--prefix=/Users/runner/.ghcup/ghc/aarch64-apple-darwin-8.10.7",
[ ...   ]                              ""] failed with exit code 1.

The error 00841 is:

A process exited prematurely [GHCup-00841]

This error can happen for a variety of reasons and may indicate that GHCup needs to improve error handling/messages. Consider [raising an issue](https://github.com/haskell/ghcup-hs/issues).

One common source of errors is a missing or non-working C toolchain, leading to errors when trying to install GHC:

(i didn't post examples with -j flag, but the error was something like "make no parse -j8". if its necessary ill look it up

[dfordivam]

Regarding the hadrian flags, I figured out that the "binary-dist" target does work with the '--docs=none'.

We already have a way to specify -j and --flavour flags.

In addition the user might want to specify these

--docs[=TARGET]             Strip down docs targets (none, no-haddocks, no-sphinx[-{html, pdfs, man}].
--bignum[=BACKEND]          Select ghc-bignum backend: native, gmp (default), check-gmp, ffi.

But it would be problematic if the user specifies these

-o[BUILD_ROOT], --build-root[=BUILD_ROOT] Where to store build artifacts. (Default _build).
-c, --configure             Deprecated: Run the boot and configure scripts.

Flags other than these are likely not relevant for compilation via ghcup.

So perhaps it would be simpler if ghcup also supports --docs and --bignum flags, which are passed on to hadrian.

All hadrian flags (from ghc 9.10.1)

Usage: hadrian [options] [target] ...

Standard options:
  --abbrev=FULL=SHORT         Use abbreviation in status messages.
  --allow-redefine-rules      Allow redefining built-in rules
  --no-allow-redefine-rules   Forbid redefining built-in rules (default)
  --no-build                  Don't build anything.
  -C DIRECTORY, --directory=DIRECTORY
                              Change to DIRECTORY before doing anything.
  --color, --colour           Colorize the output.
  --no-color, --no-colour     Don't colorize the output.
  --compact[=yes|no|auto]     Use a compact Bazel/Buck style output.
  -d[FILE], --debug[=FILE]    Print lots of debugging information.
  --demo                      Run in demo mode.
  --digest                    Files change when digest changes.
  --digest-and                Files change when modtime and digest change.
  --digest-and-input          Files change on modtime (and digest for inputs).
  --digest-or                 Files change when modtime or digest change.
  --digest-not                Files change when modtime changes.
  --exception                 Throw exceptions directly.
  --flush=N                   Flush metadata every N seconds.
  --never-flush               Never explicitly flush metadata.
  -h, --help                  Print this message and exit.
  -j[N], --jobs[=N]           Allow N jobs/threads at once [default CPUs].
  --keep-going                Keep going when some targets can't be made.
  -l, --lint                  Perform limited validation after the run.
  --lint-watch=PATTERN        Error if any of the patterns are created (expensive).
  --lint-fsatrace[=DIR]       Use fsatrace to do validation [in current dir].
  --lint-ignore=PATTERN       Ignore any lint errors in these patterns.
  --no-lint                   Turn off --lint.
  --live[=FILE]               List the files that are live [to live.txt].
  -m PREFIX, --metadata=PREFIX
                              Prefix for storing metadata files.
  --numeric-version           Print just the version number and exit.
  --skip-commands             Try and avoid running external programs.
  -B[PATTERN], --rebuild[=PATTERN]
                              If required, these files will rebuild even if nothing has changed.
  --no-rebuild[=PATTERN]      If required, these files will rebuild only if things have changed (default).
  --skip[=PATTERN]            Don't rebuild matching files this run.
  -r[FILE], --report[=FILE], --profile[=FILE]
                              Write out profiling information [to report.html].
  --no-reports                Turn off --report.
  --rule-version=VERSION      Version of the build rules.
  --no-rule-version           Ignore the build rules version.
  --share[=DIRECTORY]         Shared cache location.
  --share-list                List the shared cache files.
  --share-sanity              Sanity check the shared cache files.
  --share-remove[=SUBSTRING]  Remove the shared cache keys.
  --share-copy                Copy files into the cache.
  --share-symlink             Symlink files into the cache.
  -s, --silent                Don't print anything.
  --sleep                     Sleep for a second before building.
  -S, --no-keep-going, --stop
                              Turns off -k.
  --storage                   Write a storage log.
  -p[N], --progress[=N]       Show progress messages [every N secs, default 5].
  --no-progress               Don't show progress messages.
  -q, --quiet                 Print less (pass repeatedly for even less).
  --no-time                   Don't print build time.
  --timings                   Print phase timings.
  -V, --verbose, --trace      Print more (pass repeatedly for even more).
  -v, --version               Print the version number and exit.
  -w, --print-directory       Print the current directory.
  --no-print-directory        Turn off -w, even if it was turned on implicitly.

Extra options:
  -c, --configure             Deprecated: Run the boot and configure scripts.
  -o BUILD_ROOT, --build-root=BUILD_ROOT
                              Where to store build artifacts. (Default _build).
  --flavour[=FLAVOUR]         Build flavour (Default, Devel1, Devel2, Perf, Prof, Quick or Quickest).
  --freeze1                   Freeze Stage1 GHC.
  --freeze2                   Freeze Stage2 GHC.
  --hash-unit-ids             Include package hashes in unit ids.
  --skip-depends              Skip rebuilding dependency information.
  --bignum[=BACKEND]          Select ghc-bignum backend: native, gmp (default), check-gmp, ffi.
  --progress-info=STYLE       Progress info style (None, Brief, Normal or Unicorn).
  --docs=TARGET               Strip down docs targets (none, no-haddocks, no-sphinx[-{html, pdfs, man}].
  -k, --keep-test-files       Keep all the files generated when running the testsuite.
  --test-compiler=TEST_COMPILER
                              Use given compiler [Default=stage2].
  --test-config-file=CONFIG_FILE
                              configuration file for testsuite. Default=testsuite/config/ghc
  --config=EXTRA_TEST_CONFIG  Configurations to run test, in key=value format.
  --summary-junit[=TEST_SUMMARY_JUNIT]
                              Output testsuite summary in JUnit format.
  --summary-metrics[=METRICS_FILE]
                              Output testsuite performance metrics summary.
  --only[=TESTS]              Test cases to run.
  --only-perf                 Only run performance tests.
  --skip-perf                 Skip performance tests.
  --test-root-dirs[=DIR1:[DIR2:...:DIRn]]
                              Test root directories to look at (all by default).
  --test-speed=SPEED          fast, slow or normal. Normal by default
  --summary[=TEST_SUMMARY]    Where to output the test summary file.
  --test-verbose[=TEST_VERBOSE]
                              A verbosity value between 0 and 5. 0 is silent, 4 and higher activates extra output.
  --test-way=TEST_WAY         only run these ways
  --broken-test=TEST_NAME     consider these tests to be broken
  -a, --test-accept           Accept new output of tests
  --test-have-intree-files    Run the in-tree tests even with an out of tree compiler
  --prefix[=PATH]             Destination path for the bindist 'install' rule
  --complete-setting[=SETTING]
                              Setting key to autocomplete, for the 'autocomplete' target.
  --haddock-for-hackage       Generate documentation suitable for upload to a hackage server.

[dfordivam]

For ghc-bignum, there is one flavour transformer native_bignum
https://gitlab.haskell.org/ghc/ghc/-/blob/master/hadrian/doc/flavours.md#flavour-transformers

[dfordivam]

• Regarding passing additional flags to bindist configure

The following flags are already handled by GHCup: --prefix, --target, --disable-ld-override

Flags other than these may not be necessary to be specified by most of users.

Note that for cross-compilation, most of the flags have to be specified at the build configure.
ref: https://gitlab.haskell.org/ghc/ghc/-/wikis/building/cross-compiling#configuring-the-build

• Regarding passing additional flags to hadrian

As I have mentioned above, and discussed here --bignum is likely not necessary, and just adding support of --docs should suffice.

Ref: All bindist configure flags (from ghc 9.10.1)

Usage: ./configure [OPTION]... [VAR=VALUE]...

To assign environment variables (e.g., CC, CFLAGS...), specify them as
VAR=VALUE.  See below for descriptions of some of the useful variables.

Defaults for the options are specified in brackets.

Configuration:
  -h, --help              display this help and exit
      --help=short        display options specific to this package
      --help=recursive    display the short help of all the included packages
  -V, --version           display version information and exit
  -q, --quiet, --silent   do not print `checking ...' messages
      --cache-file=FILE   cache test results in FILE [disabled]
  -C, --config-cache      alias for `--cache-file=config.cache'
  -n, --no-create         do not create output files
      --srcdir=DIR        find the sources in DIR [configure dir or `..']

Installation directories:
  --prefix=PREFIX         install architecture-independent files in PREFIX
                          [/usr/local]
  --exec-prefix=EPREFIX   install architecture-dependent files in EPREFIX
                          [PREFIX]

By default, `make install' will install all the files in
`/usr/local/bin', `/usr/local/lib' etc.  You can specify
an installation prefix other than `/usr/local' using `--prefix',
for instance `--prefix=$HOME'.

For better control, use the options below.

Fine tuning of the installation directories:
  --bindir=DIR            user executables [EPREFIX/bin]
  --sbindir=DIR           system admin executables [EPREFIX/sbin]
  --libexecdir=DIR        program executables [EPREFIX/libexec]
  --sysconfdir=DIR        read-only single-machine data [PREFIX/etc]
  --sharedstatedir=DIR    modifiable architecture-independent data [PREFIX/com]
  --localstatedir=DIR     modifiable single-machine data [PREFIX/var]
  --runstatedir=DIR       modifiable per-process data [LOCALSTATEDIR/run]
  --libdir=DIR            object code libraries [EPREFIX/lib]
  --includedir=DIR        C header files [PREFIX/include]
  --oldincludedir=DIR     C header files for non-gcc [/usr/include]
  --datarootdir=DIR       read-only arch.-independent data root [PREFIX/share]
  --datadir=DIR           read-only architecture-independent data [DATAROOTDIR]
  --infodir=DIR           info documentation [DATAROOTDIR/info]
  --localedir=DIR         locale-dependent data [DATAROOTDIR/locale]
  --mandir=DIR            man documentation [DATAROOTDIR/man]
  --docdir=DIR            documentation root [DATAROOTDIR/doc/ghc-9.10.1]
  --htmldir=DIR           html documentation [DOCDIR]
  --dvidir=DIR            dvi documentation [DOCDIR]
  --pdfdir=DIR            pdf documentation [DOCDIR]
  --psdir=DIR             ps documentation [DOCDIR]

System types:
  --build=BUILD     configure for building on BUILD [guessed]
  --host=HOST       cross-compile to build programs to run on HOST [BUILD]
  --target=TARGET   configure for building compilers for TARGET [HOST]

Optional Features:
  --disable-option-checking  ignore unrecognized --enable/--with options
  --disable-FEATURE       do not include FEATURE (same as --enable-FEATURE=no)
  --enable-FEATURE[=ARG]  include FEATURE [ARG=yes]
  --enable-distro-toolchain
                          Do not use bundled Windows toolchain binaries.
  --disable-ld-override   Prevent GHC from overriding the default linker used
                          by gcc. If ld-override is enabled GHC will try to
                          tell gcc to use whichever linker is selected by the
                          LD environment variable. [default=override enabled]

Optional Packages:
  --with-PACKAGE[=ARG]    use PACKAGE [ARG=yes]
  --without-PACKAGE       do not use PACKAGE (same as --with-PACKAGE=no)
  --with-hs-cpp=ARG       Path to the Haskell (C) preprocessor for Haskell
                          files [default=autodetect]
  --with-hs-cpp-flags=ARG Flags to the Haskell (C) preprocessor for Haskell
                          files [default=autodetect]
  --with-cpp=ARG          Path to the (C) preprocessor [default=autodetect].
                          If you set --with-cpp=CC, ensure -E is included in
                          --with-cpp-flags
  --with-cpp-flags=ARG    Flags to the (C) preprocessor [default=autodetect]

Some influential environment variables:
  CC          C compiler command
  CFLAGS      C compiler flags
  LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries in a
              nonstandard directory <lib dir>
  LIBS        libraries to pass to the linker, e.g. -l<library>
  CPPFLAGS    (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if
              you have headers in a nonstandard directory <include dir>
  CXX         C++ compiler command
  CXXFLAGS    C++ compiler flags
  CPP         C preprocessor
  LLC         Use as the path to LLVM's llc [default=autodetect]
  OPT         Use as the path to LLVM's opt [default=autodetect]
  LLVMAS      Use as the path to LLVM's assembler (typically clang)
              [default=autodetect]

Use these variables to override the choices made by `configure' or to help
it to find libraries and programs with nonstandard names/locations.

[dfordivam]

With reference to #1082, perhaps GHCup should throw an error if user specifies --config with hadrian, and tell the user to specify --make if they wish to use it. (or default to --make if it is available and user has specified --config) (done via #1085)

And as the --make has been deprecated from 9.6 onward, the only way the user would be able to do custom builds would be to specify hadrian/UserSettings.hs file as described here.
Should GHCup support specifying this file via CLI option?