MacOS ARM wheels broken?

[jeromekelleher]

It looks like the binary wheels for Mac ARM builds is broken somehow. Basically we get symbol missing problems on import:

  from .cyvcf2 import (VCF, Variant, Writer, r_ as r_unphased, par_relatedness,
39
E   ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/cyvcf2/cyvcf2.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_bcf_float_missing'

I'm afraid I don't have a mac, so I can't help any further.

Some downstream reports:

• ImportError: dlopen[...]: symbol not found in flat namespace '_bcf_float_missing' - sgkit-vcf requirements are not installed sgkit-dev/sgkit#1220
• Run CI on macos-14 (ARM) sgkit-dev/bio2zarr#216

[brentp]

@graphenn do you have an idea about this?

[graphenn]

I don't have a Mac either. Additionally, the GitHub action can't test with macOS ARM. We need someone with this environment to assist.

[jeromekelleher]

@hyanwong or @tomwhite - would either of you be able to replicate? I think this is going to hit us downstream so it's would be great if we could track down the issue and hopefully fix the wheels.

[hyanwong]

I can replicate this on my M2 when installing cyvcf2 via pip (python 3.10, 3.11, and 3.12).

I have cyvcf2 working via another route. I'll see if I can figure out how I did that, and post here.

[hyanwong]

I can get a working cyvcf2 by compiling via pip --no-binary=cyvcf2. This requires both autoconf and automake. Make sure you python -m pip uninstall cyvcf2 first if you have previously installed a non-working version:

mamba install automake autoconf  # or could equivalently use conda
python -m pip install cyvcf2 --no-binary=cyvcf2 

it might be that you also require htslib for compilation: I didn't try it on a completely clean OS install. Let me know if there are still compilation failures and I can see what else I previously installed to get it to work.

[jeromekelleher]

Looking at the wheel building, it seems like we're cross-compiling for ARM on a macos-12 host. Would it be simpler to use the newer macos-14 GitHub actions runner and compile natively?

[jeromekelleher]

Or maybe use cibuildwheel directly? I don't know what the history of the wheel building process is here, sorry if we're going over old ground!

[graphenn]

Looking at the wheel building, it seems like we're cross-compiling for ARM on a macos-12 host. Would it be simpler to use the newer macos-14 GitHub actions runner and compile natively?

Yes, the new macos-14 action runner makes it much easier. I create a PR for changing macos ARM to this runner.
#306

NO cp38 wheel for macos arm

@hyanwong Could you test with the building wheel? https://github.com/graphenn/cyvcf2/actions/runs/9229532344
artifact-cp3xx-macosx_arm64

[brentp]

I pushed a new tag for this with the fix from @graphenn . v0.31.0

[jeromekelleher]

Confirmed packages are working on macos-14 GitHub Action runner! Thanks very much!

[brentp]

🥳 thanks @graphenn

[jmarshall]

Sounds a lot like you were running into a similar problem to pysam-developers/pysam#1231, which we too worked around by ensuring we were compiling on e.g. ARM macOS rather than cross-compiling from a different architecture.