How to compile on MacOS for x86, M1 and M2 (apple silicon)

[ikus060]

Hello,

I have a build pipeline created to compile a python project with c++ dependenciesand freeze it with Pyinstaller on multiple platform including Windows, Linux and MacOS. For MacOS, I'm compiling on x86 AMD64 architecture and I wish to include support for the new architecture M1 and M2.

1. Is it possible to cross-compile for M1, M2 and x86 into the same dmg file ?
2. Is it possible to cross-compile M1 and M2 on a x86 or do I need to buy hardware for each of them !?
   3 I also read something about cross-compiling from a Linux for MacOS ? https://pyinstaller.org/en/stable/bootloader-building.html#cross-building-for-macos Is it working on M1 and M2 ?

[rokm]

Hello,

I have a build pipeline created to compile a python project with c++ dependenciesand freeze it with Pyinstaller on multiple platform including Windows, Linux and MacOS. For MacOS, I'm compiling on x86 AMD64 architecture and I wish to include support for the new architecture M1 and M2.

1. Is it possible to cross-compile for M1, M2 and x86 into the same dmg file ?

M1 and M2 are the same architecture (arm64), and by x86 you probably mean x86_64 (the last version of macOS that supported 32-bit Intel Macs was 10.14 I think). Your application can be either single-arch and consists of thin-binaries (e.g., x64_64-only or arm64 only) or it can be universal2 - consisting of fat binaries that contain both x86_64 and arm64 slices.

So if you want the same application build to support both architectures, you need to go universal2 way. (I'm not sure how this translates to DMG, though - can you put in only one application build? Or could you put in two separate thin-arch builds as well?)

2. Is it possible to cross-compile M1 and M2 on a x86 or do I need to buy hardware for each of them !?

Technically, PyInstaller is not compiling any machine-level code (i.e., binaries) - it is collecting binaries (shared libs and binary extension modules) from your python environment. So whether you can create a universal2 application or not depends on your python environment and all binary dependencies (including your C++ stuff). All of these need to be universal2 if you want to create universal2 application using PyInstaller. See the docs.

You can cross-compile your C++ code for arm64 or universal2 on x86_64 (and you can cross-compile for x86_64 on arm64). But you also need to ensure that if you use binary wheels for non-pure-python packages, that those are universal2 as well (and of course, python must be universal2 build as well - so you cannot use Homebrew python or Anaconda python, which are both single-arch).

If you cannot create universal2 application, you'll have to create two single-arch applications. And for that you'll need an arm64 Mac, because you need to create the build using python environment that matches the target architecture. So you can create native arm64 build by running native arm64 python environment, and create x86_64 by running x86_64 python environment under Rosetta.

3. I also read something about cross-compiling from a Linux for MacOS ? https://pyinstaller.org/en/stable/bootloader-building.html#cross-building-for-macos Is it working on M1 and M2 ?

That's only for compiling bootloader (you are likely using pre-compiled bootloader), not for creating frozen applications.

[qlrd]

This can be applied for M3 as well?

[rokm]

Yes, what's written above for M1 and M2 also applies to M3.

[ikus060]

@rokm

Thanks ALOT for the clarification.

4. I'm currently using homebrew installation for python. So I will need to move away from that if I want to target universal2. How do you install python to be compatible with universal2 ?

5. Is it possible to cross-compile universal2 from a Linux x86_64 like you are doing for the bootloader ?

If possible, I'm more incline to spent time on cross compiling from Linux, I could make the build more reproducible by using Docker container. Or maybe you have another suggestion to make a reproducible build ?

by x86 you probably mean x86_64

Yes, I mean x86_64

[rokm]

4. I'm currently using homebrew installation for python. So I will need to move away from that if I want to target universal2. How do you install python to be compatible with universal2 ?

python.org provides universal2 python builds for python >= 3.10: https://www.python.org/downloads/macos/

5. Is it possible to cross-compile universal2 from a Linux x86_64 like you are doing for the bootloader ?

No. We collect binaries from the active python environment; so by the very definition, you can only create frozen application whose architecture (both the OS and CPU architecture) matches that of the active python environment. Which on Linux x86_64 would be Linux x86_64 - and even that might be limited to (the particular release version of) your distribution, depending on what shared libraries need to be collected from the system. Or x86 if you set up and ran 32-bit python environment.

The only way of creating macOS builds on a non-Mac is if you ran a copy of macOS in a virtual machine and set up python environment in it (which would probably be x86_64, so it would limit you to building x86_64 or universal2 PyInstaller-frozen applications).

[ikus060]

Ok, Thanks again.

I will explore how to re-create my build environment using Python from python.org to replace my homebrew installation to have a universal2 build at the end.

Thanks !

[ikus060]

@rokm

I've started creating a new VM with MacOS x86_64. I've install Python from python.org with universal2. Using file I can confirm python was installed with universal 2:

Now I'm at the point, where I need to make sure all my dependencies are also universal2 and they aren't.
In a virtualenv, when using pip install yaml, but only the x86_64 is installed. How could I force installation of compilation of universal2 ?

[rokm]

How could I force installation of compilation of universal2 ?

If the package does not provide universal2 wheels, you will probably have to prevent installation of the pre-built x86_64-only wheel and install from sdist instead. Or build a custom wheel from sdist, and install that. In either case, you'll have to ensure that universal2 version is built.

I have no experience with building python universal2 wheels, so you'll have to look up the instructions for that on your own.