-
Notifications
You must be signed in to change notification settings - Fork 494
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP: Support arm64 architecture to enable usage on the current M1 MacBooks #434
Conversation
This will take some time to review as I have to check if I need to do |
As far as i understand it it is doable on just one architecture. There is a Maybe i can help by setting up an example... |
Yes an example would be great. |
I found this example online (.gitlab-ci.yml) ... but not tested it yet.
|
Add arm64 support for php-nginx-dev/8.0-alpine
Thanx to @allansun for PHP 8.0 example |
Thanks to @bweinzierl I've managed to get webdevops/php-nginx-dev:8.0-alpine compiled under arm64 on my M1 MacBookPro. I DID NOT use the 'docker-container' driver ('docker buildx create --use'), instead I used the default 'docker' driver ('docker buildx use default'), because this is the only way I can get buildx to use local built images. ( toolbox -> php -> php-nginx -> php-nginx-dev) To get @hhoechtl running this in CI environment, you DO need to use the 'docker-container' driver ('docker buildx create --use'). What it does is to bring up a docker-in-docker builder, which supports multi arch build including arm64. Even though I couldn't try it myself (because I cannot push to webdevops/* to get later images using correct inheritance), the method @bweinzierl mentioned seems to be good enough. We just need to alter the python code to use 'docker buildx build' instead of 'docker build'. Can't wait to see this happening! |
Today i spent quite some time setting up a build task for building with buildx on our gitlab runner. As easy as it looked on first glance, when using buildx on a amd64 machine with qemu there were a few problems. Getting the runner workingAs always it is not as simple as promoted in all the howtos. But i have it working now for our gitlab runner:
The webdevops golang binaries (gosu and go-replace) throw an error when executed#8 5.164 runtime: failed to create new OS thread (have 2 already; errno=22) Looks like it is some problem with qemu and golang? @hhoechtl Do you happen to know why the golang scripts cannot be called in a qemu environment but native it works? Update: I found this: |
@bweinzierl from what I can tell it's nothing to do with qemu. The main problem is that as long as you use 'buildx' to build and in Dockerfile you are 'FROM webdevops/*', this may create problem, because no matter how you tweak the settings, if the 'parent' image was not built for your target arch, you will definitely get some problems later on. I saw exactly the same link https://issueexplorer.com/issue/tianon/gosu/93 maybe we can give it a try. |
That is not the problem. I am building the whole chain. Starting with toolbox:latest which depends on alpine:latest. So this should work (and does in deed work on my M1). But on amd with buildx the call of |
When you build locally with buildx, the webdevops/toolbox:latest is built successfully and it's available in you 'docker images', you would think buildx would use such local image when you start building webdevops/php:7.4-alpine. However according to my test (and some google results, which I can't remember where...), buildx will ignore your local image and still pull remote images. Unless buildx is able to use local image (not possible at the moment), the only way we can get it working is to push the built image to docker hub... |
@allansun The toolbox build is already failing so the chain is not the problem. It is build on alpine:latest and this is a multiarch image. |
Check out this: https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images and this https://docs.docker.com/engine/reference/builder/#from
The reason your build fails should because the alpine:latest wasn't pulled with correct arch in the first place... |
@allansun Ah... thanks! This might really be the problem. I will check it out later! |
@allansun @hhoechtl I now have a running setup to build the webdevops images as multiarch images and push to our internal gitlab docker registry. So to provide a running setup that is consistent and logical we need to solve the go - qemu problem (by updating). If we cannot do that then we need to build the chain with amd executables and replace the executables for the correct architecture ones at the end of the build (and don't execute if afterwards). This would result in the same image but is a hacky / ugly solution. @hhoechtl Could you give an estimation if/when you would be able to provide these new executables or if not would be willing to implement the hacky build pipeline into your current setup. I guess for this i would remove the changes for the architecture specific go-binaries again in this PR and revert to what is there already and only add a last line in the last image in the chain where they are replaced by the correct one. |
Maybe the weekend 5./6. of February. That's the next spare time I have. |
…plete for 7.4 and 8.0 in apache or nginx
Kudos, SonarCloud Quality Gate passed! |
We are now using this modified branch with the qemu workaround to build the php-apache-74 and php-apache-80 images as multiarch: Here i found additional info about the qemu go problem: Especially the linked comment is interesting, there seems to be some configuration variable to prevent this issue. @hhoechtl If you find time on this weekend i would be open for a hangout session. Could also help with stuff on that weekend. |
@bweinzierl builds and runs on M1, thank you very much for your efforts! For reference, here is the script I used to test it locally, without Gitlab CI: #!/bin/bash
set -e
echo "Building Multiarch Webdevops images ..."
if ! [[ -d ./webdevops-fork ]]; then
git clone --branch temp-qemu-workaround https://github.com/Format-D/Dockerfile.git webdevops-fork
else
cd webdevops-fork && git pull && cd ..
fi
docker buildx version
docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-* # this was needed / without it somehow the arm support is missing
docker run --privileged --rm tonistiigi/binfmt --install all
if [[ -z `docker context ls | grep multiarch-builder-context` ]]; then
docker context create multiarch-builder-context
docker buildx create --name multiarch-builder --use multiarch-builder-context --platform linux/amd64,linux/arm64
fi
docker context ls
docker buildx inspect --bootstrap
docker buildx ls
# to push to a gitlab registry, use e.g.
#docker buildx build --push --platform linux/amd64,linux/arm64 --tag my-webdevops-registry.com/toolbox:latest webdevops-fork/docker/toolbox/latest
#docker buildx build --push --platform linux/amd64,linux/arm64 --tag my-webdevops-registry.com/php:7.4 webdevops-fork/docker/php-official/7.4
#docker buildx build --push --platform linux/amd64,linux/arm64 --tag my-webdevops-registry.com/php:8.0 webdevops-fork/docker/php-official/8.0
# you can only --load one target platform image locally, see https://github.com/docker/buildx/issues/59
docker buildx build --load --platform linux/arm64 --tag webdevops/toolbox:latest webdevops-fork/docker/toolbox/latest
docker buildx build --load --platform linux/arm64 --tag webdevops/php:7.4 webdevops-fork/docker/php-official/7.4
# the following images will use the official webdevops.io images, not our local base image - those are still AMD only :(
# see https://github.com/docker/buildx/issues/301
docker buildx build --load --platform linux/arm64 --tag webdevops/php-apache:7.4 webdevops-fork/docker/php-apache/7.4
docker buildx build --load --platform linux/arm64 --tag webdevops/php-apache-dev:7.4 webdevops-fork/docker/php-apache-dev/7.4
# further php version ...
#docker buildx build --load --platform linux/arm64 --tag webdevops/php-apache-dev:8.0 webdevops-fork/docker/php-apache-dev/8.0
echo "Done" |
We are building the complete image chain and pushing them to our internal gitlab registry. So this was no problem for me. |
That would be awesome! Did you need to setup anything so that buildx uses your own base images from your custom registry and not the official webdevops (AMD) images from Github? |
@smxsm apache php 7.4 is now available here (and php 8.0 is currently being built): Please be aware that this is only intended to be a temporary location until webdevops implements the buildx in their pipeleine. This is our CI Pipeline. We are using the fork and replacing the FROM ... with our Images (see line 8):
|
@bweinzierl thanks a lot! I did a quick test run with "formatdgmbh/webdevops-php-apache-dev:7.4" ... the weird thing is, I still have "qemu" running inside the container ... can you check if you have that, too? If you're building the whole chain that shouldn't be the case ... strange. root@2f92565e40ed:/app# ps -aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.5 0.1 175636 42864 ? Ssl 20:55 0:01 /usr/bin/qemu-x86_64 /usr/bin/python2 /usr/bin/supervisord -c /opt/docker/etc/supervisor.conf --logfile /dev/null --pidf
root 232 0.0 0.1 315800 27712 ? Sl 20:55 0:00 /usr/bin/qemu-x86_64 /usr/sbin/syslog-ng -F --no-caps -p /var/run/syslog-ng.pid
root 234 0.1 0.3 929376 83684 ? Sl 20:55 0:00 /usr/bin/qemu-x86_64 /usr/local/bin/php-fpm --nodaemonize
root 235 0.1 0.1 160960 26484 ? Sl 20:55 0:00 /usr/bin/qemu-x86_64 /usr/sbin/apache2 -DFOREGROUND -DAPACHE_LOCK_DIR
root 238 0.0 0.0 151712 10272 ? Sl 20:55 0:00 /usr/bin/qemu-x86_64 /usr/sbin/cron -f
root 239 0.0 0.0 150492 12556 ? Sl 20:55 0:00 /usr/bin/qemu-x86_64 /bin/bash /opt/docker/bin/service.d/postfix.sh
root 240 0.0 0.0 161704 19192 ? Sl 20:55 0:00 /usr/bin/qemu-x86_64 /usr/sbin/sshd -D
www-data 345 0.0 0.2 2503336 54008 ? Sl 20:55 0:00 /usr/bin/qemu-x86_64 /usr/sbin/apache2 -DFOREGROUND -DAPACHE_LOCK_DIR
www-data 347 0.0 0.2 2503452 54292 ? Sl 20:55 0:00 /usr/bin/qemu-x86_64 /usr/sbin/apache2 -DFOREGROUND -DAPACHE_LOCK_DIR
www-data 477 2.6 0.5 962060 123204 ? Sl 20:55 0:08 /usr/bin/qemu-x86_64 /usr/local/bin/php-fpm --nodaemonize
www-data 479 2.4 0.5 1037472 130256 ? Sl 20:55 0:07 /usr/bin/qemu-x86_64 /usr/local/bin/php-fpm --nodaemonize
root 763 0.0 0.0 188792 14208 ? Ssl 20:55 0:00 /usr/bin/qemu-x86_64 /usr/lib/postfix/sbin/master -w
postfix 765 0.0 0.0 188444 15396 ? Sl 20:55 0:00 /usr/bin/qemu-x86_64 /usr/lib/postfix/sbin/pickup -l -t unix -u -c
postfix 767 0.0 0.0 188532 15524 ? Sl 20:55 0:00 /usr/bin/qemu-x86_64 /usr/lib/postfix/sbin/qmgr -l -t unix -u
root 1272 2.0 0.0 150744 14116 pts/0 Ssl 21:00 0:00 /usr/bin/qemu-x86_64 /bin/bash
root 1285 2.0 0.0 148168 6748 ? Sl 21:00 0:00 /usr/bin/qemu-x86_64 /bin/sleep 5
root 1288 0.0 0.0 155868 9636 ? Rl+ 19:12 0:00 /bin/ps -aux If I run my custom-built single PHP 7.4. image (not using webdevops, but "FROM arm64v8/php:7.4-apache", I get this: root@485d3b7a84b8:/app# ps -aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.1 0.1 87932 27536 ? Ss 21:03 0:00 apache2 -DFOREGROUND
www-data 19 0.0 0.0 87964 9520 ? S 21:03 0:00 apache2 -DFOREGROUND
www-data 20 0.0 0.0 87964 9520 ? S 21:03 0:00 apache2 -DFOREGROUND
www-data 21 0.0 0.0 87964 9520 ? S 21:03 0:00 apache2 -DFOREGROUND
www-data 22 0.0 0.0 87964 9520 ? S 21:03 0:00 apache2 -DFOREGROUND
www-data 23 0.0 0.0 87964 9520 ? S 21:03 0:00 apache2 -DFOREGROUND
root 24 0.0 0.0 3960 3004 pts/0 Ss 21:04 0:00 /bin/bash
root 31 0.0 0.0 6448 2400 pts/0 R+ 21:04 0:00 ps -aux |
Hey @smxsm, ich checked and i don't have any qemu processes running in my container. Are you running on M1 MacBookPro? What does the docker desktop UI show? It should display a label if the containers are run with emulation. |
@smxsm I just tried it on my Macbook Pro 13" M1 on macOS 12.1 (21C52) with Docker:
And it worked for me as well: ➜ ~ docker run -it --entrypoint /bin/sh formatdgmbh/webdevops-php-apache-dev:7.4
Unable to find image 'formatdgmbh/webdevops-php-apache-dev:7.4' locally
7.4: Pulling from formatdgmbh/webdevops-php-apache-dev
4c7c9f6f1115: Pull complete
24d17ac9533d: Pull complete
7aa05af31bdf: Pull complete
4adb58801e0c: Pull complete
c4d341a770db: Pull complete
0c2c8a0c5f03: Pull complete
d0c31a1b672b: Pull complete
393076d9a7d8: Pull complete
afc1b0072ad1: Pull complete
caf07fd559ce: Pull complete
fc65bddc85fa: Pull complete
6f09b1807fe2: Pull complete
ba712be57a9c: Pull complete
96f6a737c262: Pull complete
3dc3afc2438d: Pull complete
8a7b683df78f: Pull complete
4f25cf576ef6: Pull complete
a2e767a3d4d3: Pull complete
ac99d4df781c: Pull complete
bea2eb794e40: Pull complete
c18ded887584: Pull complete
f31074a7878c: Pull complete
6673339b19bb: Pull complete
Digest: sha256:396632f30a7b99e9e1021d020b7ab4d036340ba2eabd037ba8f902afeaef700b
Status: Downloaded newer image for formatdgmbh/webdevops-php-apache-dev:7.4
# ps -aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.2 0.0 2040 460 pts/0 Ss 09:38 0:00 /bin/sh
root 8 0.0 0.0 7132 2620 pts/0 R+ 09:38 0:00 ps -aux
# arch
aarch64 |
@fdsis thx for testing! Ok, so it must be something on my end ... 😞 Docker 4.3.0 (71111) (Experimental Build with VirtioFS) uname -a says: "21.2.0 Darwin Kernel Version 21.2.0: Sun Nov 28 20:28:41 PST 2021; root:xnu-8019.61.5~1/RELEASE_ARM64_T6000 arm64" arch says: "arm64" I have asked a colleague to run a test container, too, but he didn't respond yet ... |
Ok, it's showing up as arm64 image for my colleague, too, so it really has to be a problem on my Macbook only ... maybe it's related to my "experimental" Docker build which I am using currently to test VirtioFS ... sorry for the confusion! |
@smxsm Great! No problem! But if we can make the go binaries work with qemu (probably by updating go?) we would have pretty straightforward dockerfiles (see this PR). |
Thx. I'm trying to review, adapt and merge in february. |
@hhoechtl what is the status on this PR? |
@daniel-fahl Until this PR is resolved you can use these temporary images: It is only Apache 8.0 and 7.4 for now though.... |
@hhoechtl do you have an update on this? It would be great to have the original images working again! @bweinzierl as the provided image didn't work for me, I found the following image which was much closer to a drop in replacement. |
@daniel-fahl Why did the images not work? We are using them (to be precise the same build pushed to a internal gitlab instead) since months on 10+ different macs (arm and amd) without problems. |
@hhoechtl now that February is over, what is the status of this? It's june now and we are still waiting for working images for M1 ... can we help with anything? |
Kudos, SonarCloud Quality Gate passed! |
Any news on this one? Love the images but unfortunately they're pretty much unusable for me since upgrading to the M1 Would love to help but I have like zero experience |
Is there anyone who can merge this? As it looks there is a solution available for all of us with a M1 Mac and it just needs to be implemented. |
Would be nice if anyone with rights reviews and merges if all good |
Somone it'll take a look at this? |
Any updates to this? |
@hhoechtl If you are up for it we could make a little "Hackothon" to integrate M1/M2 support. Maybe a evening is enought... |
M1/M2 Support is now granted by a new version of go-replace which works with both architectures. |
@hhoechtl That means you are building all the images as multiarch now? |
Doesn't sound/look like it .. I guess they are working now, but are not optimized / built for M-architectures ... right? Time to finish our switch to DDEV 😄 |
I'm not sure if optimizing some binary utilities has that much of an effect. PHP would need to be optimized for ARM64 architecture. |
@hhoechtl Thanks, the qemu workaround is now no longer neccessary. It looks like more that it is. Basically only 3 changes need to be done:
With these 3 changes your images are multiarch buildable with the following script:
|
This PR shows an example how to generate the dockerfiles to support arm architecture. Also it tries to give a proposal how to modify the jinja files to get the desired results. This part is actually not tested.
Modification of dockerfile files
I modified the Dockerfile
docker/php-official/7.4/Dockerfile
directly to show the desired result when they are automatically generated for this example. This dockerfile can be bild on ARM (tested on the latest MacBook Pro) and also on a MacBook with Intel Chip (Tested on MacBook Pro (16-inch, 2019)).Modification of jinja2 files
Without knowing if the syntax is correct i also modified the jinja templates. With the modifications the dockerfiles should be generated in the shown way.
I removed the argument
arch
in the macro 'gosu(path="/sbin", arch="amd64", version="1.10")' as it is determined dynamicly during build. I did not find any usage of the argument. Hope that is ok...