Allow cross-compilation with Hadrian #844

hsyl20 · 2023-07-04T14:08:54Z

Try to fix cross-compilation with Hadrian for the JS backend (cf #838).

Currently failing with:

emconfigure /home/hsyl20/projects/ghcup-hs/dist-newstyle/build/x86_64-linux/ghc-9.4.5/ghcup-0.1.19.4/x/ghcup/build/ghcup/ghcup compile ghc --version=9.6.2 --bootstrap-ghc=9.4.3 --cross-target=javascript-unknown-ghcjs -j8 --hadrian --flavour=default+native_bignum -v --keep errors

...

[ Info  ] Merging file tree from "/home/hsyl20/.ghcup/tmp/ghcup-d9469921e6dd5041/home/hsyl20/.ghcup/ghc/javascript-unknown-ghcjs-9.6.2" to "/home/hsyl20/.ghcup/ghc/javascript-unknown-ghcjs-9.6.2"
[ Error ] [GHCup-00080] Failed to merge file tree from /home/hsyl20/.ghcup/tmp/ghcup-d9469921e6dd5041/home/hsyl20/.ghcup/ghc/javascript-unknown-ghcjs-9.6.2 to /home/hsyl20/.ghcup/ghc/javascript-unknown-ghcjs-9.6.2
[ ...   ] exception was: user error (mergeFileTree: DB file /home/hsyl20/.ghcup/db/ghc/9.6.2 already exists!)
[ ...   ] ...you may need to delete /home/hsyl20/.ghcup/ghc/javascript-unknown-ghcjs-9.6.2 manually. Make sure it's gone.
emconfigure: error: '/home/hsyl20/projects/ghcup-hs/dist-newstyle/build/x86_64-linux/ghc-9.4.5/ghcup-0.1.19.4/x/ghcup/build/ghcup/ghcup compile ghc --version=9.6.2 --bootstrap-ghc=9.4.3 --cross-target=javascript-unknown-ghcjs -j8 --hadrian --flavour=default+native_bignum -v --keep errors' failed (returned 9)

Shouldn't dbs be distinguished for cross-compilers?

hasufell · 2023-07-04T16:47:27Z

Shouldn't dbs be distinguished for cross-compilers?

Yeah, that might be a regression that needs to be fixed.

hasufell · 2023-07-04T16:58:01Z

@hsyl20 try this: #845

hsyl20 · 2023-07-05T07:39:51Z

@hasufell #845 fixes the installation!

Two notes:

ghcup uses the binary-dist Hadrian target which takes a lot of time compressing the bindist with xz and then unpacks the result. It could be useful to use the binary-dist-dir target which directly produces the unpacked bindist directory (or is the compressed bindist cached for some reason?)
Setting the cross compiler as default does:

❯ /home/hsyl20/projects/ghcup-hs/dist-newstyle/build/x86_64-linux/ghc-9.4.5/ghcup-0.1.19.4/x/ghcup/build/ghcup/ghcup set ghc javascript-unknown-ghcjs-9.6.2 -v
[ Debug ] Identified Platform as: Linux UnknownLinux
[ Debug ] last access was 284.383929082s ago, cache interval is 300s
[ Debug ] Decoding yaml at: /home/hsyl20/.ghcup/cache/ghcup-0.0.7.yaml
[ Debug ] rm -f /home/hsyl20/.ghcup/bin/javascript-unknown-ghcjs-ghc-pkg
[ Debug ] rm -f /home/hsyl20/.ghcup/bin/javascript-unknown-ghcjs-hp2ps
[ Debug ] rm -f /home/hsyl20/.ghcup/bin/javascript-unknown-ghcjs-ghci
[ Debug ] rm -f /home/hsyl20/.ghcup/bin/javascript-unknown-ghcjs-ghc
[ Debug ] rm -f /home/hsyl20/.ghcup/bin/javascript-unknown-ghcjs-hsc2hs
[ Debug ] rm -f /home/hsyl20/.ghcup/bin/haddock-ghc
[ Debug ] rm -f /home/hsyl20/.ghcup/bin/javascript-unknown-ghcjs-ghc-pkg
[ Debug ] ln -s ../ghc/javascript-unknown-ghcjs-9.6.2/bin/javascript-unknown-ghcjs-ghc-pkg-9.6.2 /home/hsyl20/.ghcup/bin/javascript-unknown-ghcjs-ghc-pkg
[ Debug ] rm -f /home/hsyl20/.ghcup/bin/javascript-unknown-ghcjs-hp2ps
[ Debug ] ln -s ../ghc/javascript-unknown-ghcjs-9.6.2/bin/javascript-unknown-ghcjs-hp2ps-ghc-9.6.2 /home/hsyl20/.ghcup/bin/javascript-unknown-ghcjs-hp2ps
[ Debug ] rm -f /home/hsyl20/.ghcup/bin/javascript-unknown-ghcjs-ghci
[ Debug ] ln -s ../ghc/javascript-unknown-ghcjs-9.6.2/bin/javascript-unknown-ghcjs-ghci-9.6.2 /home/hsyl20/.ghcup/bin/javascript-unknown-ghcjs-ghci
[ Debug ] rm -f /home/hsyl20/.ghcup/bin/javascript-unknown-ghcjs-ghc
[ Debug ] ln -s ../ghc/javascript-unknown-ghcjs-9.6.2/bin/javascript-unknown-ghcjs-ghc-9.6.2 /home/hsyl20/.ghcup/bin/javascript-unknown-ghcjs-ghc
[ Debug ] rm -f /home/hsyl20/.ghcup/bin/javascript-unknown-ghcjs-hsc2hs
[ Debug ] ln -s ../ghc/javascript-unknown-ghcjs-9.6.2/bin/javascript-unknown-ghcjs-hsc2hs-ghc-9.6.2 /home/hsyl20/.ghcup/bin/javascript-unknown-ghcjs-hsc2hs
[ Info  ] GHC 9.6.2 successfully set as default version for cross target javascript-unknown-ghcjs

So we have to use javascript-unknown-ghcjs-ghc instead of ghc. Could we remove the prefixes? It would make cabal work directly without being passed additional flags. (This could be done independently of this PR and #845.)

hasufell · 2023-07-05T07:50:36Z

So we have to use javascript-unknown-ghcjs-ghc instead of ghc

ghc would clash with the non-cross default ghc, so I don't see how.

Does ~/.ghcup/ghc/javascript-unknown-ghcjs-9.6.2/bin/ contain an unqualified ghc?

If so, then ghcup run or ghcup whereis could be used to expose those to PATH. But they may not work for cross just yet.

hsyl20 · 2023-07-05T08:40:24Z

Does ~/.ghcup/ghc/javascript-unknown-ghcjs-9.6.2/bin/ contain an unqualified ghc?
If so, then ghcup run or ghcup whereis could be used to expose those to PATH. But they may not work for cross just yet.

No, they are qualified:

❯ ls ~/.ghcup/ghc/javascript-unknown-ghcjs-9.6.2/bin/
javascript-unknown-ghcjs-ghc	     javascript-unknown-ghcjs-ghc-pkg-9.6.2
javascript-unknown-ghcjs-ghc-9.6.2   javascript-unknown-ghcjs-hp2ps
javascript-unknown-ghcjs-ghci	     javascript-unknown-ghcjs-hp2ps-ghc-9.6.2
javascript-unknown-ghcjs-ghci-9.6.2  javascript-unknown-ghcjs-hsc2hs
javascript-unknown-ghcjs-ghc-pkg     javascript-unknown-ghcjs-hsc2hs-ghc-9.6.2

Using cabal build -w javascript-unknown-ghcjs-ghc-9.6.2 is good enough. I'll fix the remaining lint issues in this PR and it should be good to merge.

hasufell · 2023-07-05T08:46:31Z

ghcup uses the binary-dist Hadrian target which takes a lot of time compressing the bindist with xz and then unpacks the result. It could be useful to use the binary-dist-dir target which directly produces the unpacked bindist directory

I found it useful to produce a bindist always and put it into cache dir, in case you want to reinstall for whatever reason.

We could have a cli switch to skip it or so.

hsyl20 · 2023-07-05T10:41:08Z

I found it useful to produce a bindist always and put it into cache dir, in case you want to reinstall for whatever reason.
We could have a cli switch to skip it or so.

It's quite a corner case, so let's not do it until there is more demand.

Could you help me with CI? It seems to choke on the empty host name provided by:

cabal-cache sync-to-archive --host-name-override= ...

hasufell · 2023-07-06T12:40:30Z

lib/GHCup/GHC.hs

@@ -1015,8 +1015,6 @@ compileGHC targetGhc ov bstrap jobs mbuildConfig patches aargs buildFlavour hadr
                             m
                             (Maybe FilePath)  -- ^ output path of bindist, None for cross
  compileHadrianBindist tver workdir ghcdir = do
-    lEM $ execWithGhcEnv "python3" ["./boot"] (Just workdir) "ghc-bootstrap"


Are you sure about this? We support compiling for old GHCs as well, both from git and from the src tarball. Do none of them need the boot?

From git we would have to ./boot. But apparently compileMakeBindist doesn't do this, so I've removed it. Perhaps we should check for the existence of ./boot before running it (in the source dist it seems to be called boot.source; I don't know why it's even bundled).

Yeah, maybe checking for existence makes sense.

hasufell · 2023-07-06T16:42:23Z

Am pushing some commits to your branch.

hasufell · 2023-07-07T08:51:26Z

I managed to compile and install:

Compile:

emconfigure ghcup -v compile ghc -j 10 --git-ref ghc-9.6 -b 9.6.2 --hadrian -x javascript-unknown-ghcjs -o 9.6.2 --bignum=native

Installing from bindist:

emconfigure ghcup install ghc -u file:///home/hasufell/.ghcup/cache/ghc-javascript-unknown-ghcjs-9.6.2-x86_64-linux-fedora-37-2023-07-06T18:08:23.090934984Z-16882bc4.tar.xz javascript-unknown-ghcjs-9.6.2.20230523

Setting as default:

ghcup set ghc javascript-unknown-ghcjs-9.6.2.20230523

Then created the hello world according to https://gitlab.haskell.org/ghc/ghc/-/wikis/javascript-backend/building#compiling-hello-world and:

 ~/tmp 
$ javascript-unknown-ghcjs-ghc -fforce-recomp HelloJS.hs                                                                          276ms  Fri 10:48
[1 of 2] Compiling Main             ( HelloJS.hs, HelloJS.o )
[2 of 2] Linking HelloJS.jsexe [Objects changed]
 ~/tmp 
$ ./HelloJS                                                                                                                      2482ms  Fri 10:50
Hello, JavaScript!

Boom 💥

The problems/remarks at the moment are:

need to set --bignum=native, so I had to introduce another cli flag for that... which I want to avoid rather, but it's hard to get this interface right, see Improve 'ghcup compile ghc' interface #846 ...we could also force native bignum for cross, which is what @bgamari suggested
emconfigure is unwieldy...
could switch hadrian as default

hasufell · 2023-07-07T08:53:04Z

lib/GHCup/Types/JSON.hs

+    Right x -> pure x
+    Left  e -> fail $ "Failure in GHCTargetVersion (FromJSON)" <> show e
+
+instance ToJSONKey GHCTargetVersion where


need to test whether that actually works in ghcup-metadata file

hsyl20 · 2023-07-07T09:44:07Z

Good work!

need to set --bignum=native, so I had to introduce another cli flag for that...

You don't need this. There is already a flag to specify the flavour and we can use it like this: --flavour=default+native_bignum

native_bignum has the same effect as --bignum=native. We should probably deprecate the latter now that we have flavour transformers.

hsyl20 · 2023-07-10T09:04:39Z

Last CI failure is a "No space left on device"

hasufell · 2023-07-10T09:32:50Z

Yes, I've been poking @angerman about it

hasufell · 2023-07-11T15:30:15Z

I think we can make a pre-release and build some more bindists manually

hasufell reviewed Jul 6, 2023

View reviewed changes

hasufell and others added 3 commits July 7, 2023 00:38

Fix cross bindist installation

186a37c

Allow cross-compilation with Hadrian

3218aaa

Fix cross target being ignored

4361ef7

hasufell force-pushed the hsyl20/cross-build branch from 9bfa85b to 4361ef7 Compare July 6, 2023 16:41

More cross fixes to install bindist

a43fa7d

hasufell reviewed Jul 7, 2023

View reviewed changes

Fix brick min size for version column

9798e0f

Avoid duplicates in cross compilers showing up

055df58

hasufell force-pushed the hsyl20/cross-build branch 2 times, most recently from 48483d0 to 6483ec5 Compare July 7, 2023 15:05

hasufell added 2 commits July 7, 2023 23:20

Make cabal-cache non-fatal

422b99a

Don't need --bignum option

0da5572

hasufell force-pushed the hsyl20/cross-build branch from 6483ec5 to 0da5572 Compare July 7, 2023 15:20

hasufell added 2 commits July 7, 2023 23:41

Disable fking cabal-cache

6ff07d3

Fix tests

07dfb1e

hasufell merged commit a05f272 into haskell:master Jul 11, 2023

georgefst mentioned this pull request Jul 15, 2023

Prefer Hadrian #259

Closed

dfordivam mentioned this pull request Jun 19, 2024

Improve 'ghcup compile ghc' interface #846

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow cross-compilation with Hadrian #844

Allow cross-compilation with Hadrian #844

hsyl20 commented Jul 4, 2023

hasufell commented Jul 4, 2023

hasufell commented Jul 4, 2023

hsyl20 commented Jul 5, 2023

hasufell commented Jul 5, 2023

hsyl20 commented Jul 5, 2023

hasufell commented Jul 5, 2023

hsyl20 commented Jul 5, 2023

hasufell Jul 6, 2023

hsyl20 Jul 6, 2023

hasufell Jul 6, 2023

hasufell commented Jul 6, 2023

hasufell commented Jul 7, 2023

hasufell Jul 7, 2023

hsyl20 commented Jul 7, 2023

hsyl20 commented Jul 10, 2023

hasufell commented Jul 10, 2023

hasufell commented Jul 11, 2023

Allow cross-compilation with Hadrian #844

Allow cross-compilation with Hadrian #844

Conversation

hsyl20 commented Jul 4, 2023

hasufell commented Jul 4, 2023

hasufell commented Jul 4, 2023

hsyl20 commented Jul 5, 2023

hasufell commented Jul 5, 2023

hsyl20 commented Jul 5, 2023

hasufell commented Jul 5, 2023

hsyl20 commented Jul 5, 2023

hasufell Jul 6, 2023

Choose a reason for hiding this comment

hsyl20 Jul 6, 2023

Choose a reason for hiding this comment

hasufell Jul 6, 2023

Choose a reason for hiding this comment

hasufell commented Jul 6, 2023

hasufell commented Jul 7, 2023

hasufell Jul 7, 2023

Choose a reason for hiding this comment

hsyl20 commented Jul 7, 2023

hsyl20 commented Jul 10, 2023

hasufell commented Jul 10, 2023

hasufell commented Jul 11, 2023