Alternatives to a self-hosted runner to build statically-linked Stack for Linux/AArch64?

[mpilgrem]

Stack currently uses a GitHub workflow and a self-hosted runner to build a statically-linked Stack for Linux/AArch64.

Historically, the runner has been provided by FP Complete. It provides Ubuntu 20.04.2 LTS on AArch64. The statically-linked Stack is then built in an Docker container providing Alpine Linux, with:

/usr/local/bin/stack etc/scripts/release.hs build --alpine --build-args --docker-stack-exe=image

The Docker image being specified (currently) by:

docker:
  enable: false
  repo: quay.io/benz0li/ghc-musl:9.6.4

@benz0li having managed to compile versions of GHC that work on Alpine Linux/AArch64.

The Haskell Foundation tried to provide a replacement runner but its machine provides NixOS on AArch64 and, as @chreekat has explained elsewhere, exposure to that environment is not good for building binaries.

I am thinking about possible alternatives. One thing I am going to explore - it may fail - is that GitHub runner macos-14 provides an AArch64 machine architecture and Docker is available for macOS/AArch64.

[hasufell]

I have runners for aarch64 linux that are not based on NixOS. They are currently hooked up to:

• haskell org
• stable-haskell org

I'm already building stack binaries there and it works.

The problem is you can't share runners across repos/orgs. So you have to start a separate agent on the same machine and they don't know about each other, potentially leading to resource exhaustion if they happen to run at the same time.

[mpilgrem]

@hasufell, thanks - your observation tallies with your issue #6252, which discussion was primarily about macOS/AArch64 but is now pertinent for Linux/AArch64.

[mpilgrem]

My experiment is not succeeding, so far, with:

Raw command: /opt/homebrew/bin/docker inspect quay.io/benz0li/ghc-musl:9.6.4
Standard output:
[]
Standard error:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

To investigate: (1) what is the source of that (2) can it be overcome?

EDIT: Perhaps it is that I need to start the Docker daemon manually: https://docker-docs.uclv.cu/config/daemon/#start-the-daemon-manually

EDIT2: That part of the Docker documentation appears to be misleading: moby/moby#27102

EDIT3: It seems possible that it is simply not possible to connect to the Docker daemon on macOS from the command line:

• https://stackoverflow.com/questions/44084846/cannot-connect-to-the-docker-daemon-on-macos
• create dev environment failed: Cannot connect to the Docker daemon at unix:///var/run/docker.sock docker/for-mac#6531

EDIT4: These blog posts may be of assistance:

• https://dhwaneetbhatt.com/blog/run-docker-without-docker-desktop-on-macos/
• https://dev.to/elliotalexander/how-to-use-docker-without-docker-desktop-on-macos-217m

[mpilgrem]

Using https://github.com/abiosoft/colima seemed promising, but colima fails to start, with (edited):

level=info msg="[hostagent] hostagent socket created at /Users/runner/.colima/_lima/colima/ha.sock"
level=info msg="[hostagent] Using system firmware (\"/opt/homebrew/share/qemu/edk2-aarch64-code.fd\")"
level=info msg="[hostagent] Starting QEMU (hint: to watch the boot progress, see \"/Users/runner/.colima/_lima/colima/serial*.log\")"
level=info msg="SSH Local Port: 49212"
level=info msg="[hostagent] Waiting for the essential requirement 1 of 4: \"ssh\""
level=info msg="[hostagent] Driver stopped due to error: \"signal: abort trap\""
level=info msg="[hostagent] Shutting down the host agent"
level=warning msg="[hostagent] failed to exit SSH master"
level=info msg="[hostagent] Shutting down QEMU with ACPI"
level=warning msg="[hostagent] Failed to remove SSH binding for port 49212"
level=warning msg="[hostagent] failed to open the QMP socket \"/Users/runner/.colima/_lima/colima/qmp.sock\", forcibly killing QEMU"
level=info msg="[hostagent] QEMU has already exited"
level=fatal msg="exiting, status={Running:false Degraded:false Exiting:true Errors:[] SSHLocalPort:0} (hint: see \"/Users/runner/.colima/_lima/colima/ha.stderr.log\")"
level=fatal msg="error starting vm: error at 'creating and starting': exit status 1"
##[error]Process completed with exit code 1.

This is a known problem: the macos-14 runner uses M1 machine architecture, M1 does not support nested virtualisation, and colima needs the latter: abiosoft/colima#970.

[mpilgrem]

However, colima may offer a local solution because the virtualisation is not nested. On my Mac mini/M1:

brew install docker
brew install colima
stack upgrade --force-download # Ensure Stack version same as in Docker image
colima start
stack etc/scripts/release.hs build --alpine --build-args --docker-stack-exe=image

built executable files but, understandably, failed when the Haskell script came to test them in macOS:

_release/bin/stack-2.15.4.1-osx-aarch64/stack --version
_release/bin/stack-2.15.4.1-osx-aarch64/stack: _release/bin/stack-2.15.4.1-osx-aarch64/stack: cannot execute binary file

EDIT: If I create an Alpine Linux VM on my Mac mini with UTM and share the relevant directory with the VM, I can, however, test the executable:

/media/share/GitHub/commercialhaskell/stack/_release/bin/stack-2.15.4.1-osx-aarch64/stack --version
Version 2.15.4.1, Git revision c34001760ea06ca44537e3a0ead349b98006f324 (10548 commits) RELEASE-CANDIDATE aarch64 hack-0.436.0

[benz0li]

https://github.blog/changelog/2023-10-30-accelerate-your-ci-cd-with-arm-based-hosted-runners-in-github-actions/

[hasufell]

The private beta is enabled in the haskell org. But those runners are not free.

[mpilgrem]

@benz0li, may I ask - the (statically-linked?) GHC for Alpine Linux/AArch64 that you build - are the binary distributions separately available for use on the OS, or are they only published as part of a Docker image?

[mpilgrem]

I implemented my 'local' macOS/AArch64 solution as a separate experimental Haskell script etc/scripts/release-linux-aarch64.hs.

[benz0li]

@benz0li, may I ask - the (statically-linked?) GHC for Alpine Linux/AArch64 that you build

They are dynamically-linked

are the binary distributions separately available for use on the OS, or are they only published as part of a Docker image?

and only available as part of the docker images.

ℹ️ I have no plans to upload the bindists to a permanent public storage.

---

That is the reason for /etc/stack.yaml

# Use only the GHC available on the PATH
system-ghc: true
# Do not automatically install GHC when necessary
install-ghc: false

in the Dev Containers. Use --no-install-ghc --system-ghc with the docker images.

Cross Reference: #6141 (comment) ff

[mpilgrem]

@benz0li, noted. My reason for asking is that I tried to use the GHC project's own GHC 9.8.2 for Alpine Linux 3.18/AArch64 in an Alpine Linux 3.19.1 VM on macOS/AArch64 to build a statically-linked Stack, but it fell over. (It worked fine when building a dynamically-linked Stack.) However, I can now build a statically-linked Stack for Linux/AArch64 if I use your Docker images on a Mac mini/M1 locally.

[benz0li]

My reason for asking is that I tried to use the GHC project's own GHC 9.8.2 for Alpine Linux 3.18/AArch64 in an Alpine Linux 3.19.1 VM on macOS/AArch64 to build a statically-linked Stack, but it fell over.

I was hoping that the official Alpine Linux[/AArch64] releases would work correctly¹ – and make my project obsolete.

As long as this is not the case I will continue to maintain https://github.com/benz0li/ghc-musl.

Footnotes

1. https://gitlab.haskell.org/ghc/ghc/-/issues/25093 ↩

[benz0li]

An alternative might be building the image under emulation with QEMU.

Cross references:

• GitHub Action: Docker Setup QEMU
• Docker image: tonistiigi/binfmt
  • Use tonistiigi/binfmt:master (current, QEMU v8.1.5)
    instead of
    tonistiigi/binfmt:latest (2022-08-02, QEMU v7.0.0)

This allows building linux/arm64 images on Linux/x86_64 by passing --platform linux/arm64 to the docker build command.

---

Emulation with QEMU can be much slower than native builds, especially for compute-heavy tasks like compilation and compression or decompression.

– Multi-platform | Docker Docs > Strategies > QEMU

[benz0li]

@mpilgrem This is how I build Stack v3.1.1¹ on AArch64 for the glcr.b-data.ch/ghc/ghc-musl:9.10.1-linux-riscv64 image.

Cross reference:

• Add RISC-V (64-bit) support benz0li/ghc-musl#9 (comment)

Footnotes

1. using glcr.b-data.ch/ghc/ghc-musl:9.8.2-linux-riscv64 ↩

[benz0li]

ℹ️ I may start building and releasing unofficial and untested RISC-V (64-bit) release assets at https://gitlab.b-data.ch/commercialhaskell/stack/-/releases until official RISC-V release assets are available.

[benz0li]

My reason for asking is that I tried to use the GHC project's own GHC 9.8.2 for Alpine Linux 3.18/AArch64 in an Alpine Linux 3.19.1 VM on macOS/AArch64 to build a statically-linked Stack, but it fell over.

@mpilgrem Due to https://gitlab.haskell.org/ghc/ghc/-/issues/25093?

[benz0li]

Unofficial and untested RISC-V release assets are now available at https://gitlab.b-data.ch/commercialhaskell/stack/-/releases.
ℹ️ These are built automatically as soon as a new version is tagged – and published if the build was successful.