WIP, BLD: meson, spin support

[tylerjereddy]

This is a very early stage/WIP PR that adds support for building MDAnalysis with the meson build system and the spin developer tool, following much of the broader scientific Python library ecosystem. Whether we actually move forward with this has not been decided, and may rest on the reception of my upcoming UGM talk related to this thrust, higher bandwidth discussions about the matter after I present the pros/cons there, and then likely an online team decision after that (as usual).

At the time of writing, it is possible to build this feature branch on ARM MacOS/x86-64 Linux with meson, meson-python, ninja installed locally with pip (+ our usual deps), and steps that look like below (to be simplified once I add spin support, then it would just be i.e., spin build -j 12). In my local testing, the full testsuite passes (on ARM MacOS/x86-64 Linux) for a meson-built MDAnalysis except for test_package_authors which depends on some weird author list magic happening in setup.py (with Python 3.11/Linux I also see an issue with test_libmdaxdr.py::TestOffsets::test_seek_tell_largefile, but that's about it).

• git checkout treddy_meson_expt
• cd package
• meson setup build --prefix=$PWD/build-install
• ninja -C build
• meson install -C build
• export PYTHONPATH=$PWD/build-install/lib/python3.11/site-packages/ (again, this is NOT for a long-term workflow, just temporary for testing this out)
• cd ../testsuite/
• python -m pytest MDAnalysisTests -n 12

TODO (this will probably grow...):

• add basic spin support, since spin build and spin test are just way nicer to develop/iterate with
• add test files/infrastructure to the meson install (for now, I'm just cheating and running them from the source tree directly)
• deal with the one test failure--test_package_authors/author list handling
• test on more architectures, Windows, etc.
• produce some build timing comparisons between the multi-core ninja build support and what we get with setuptools/current build system; perhaps "from scratch" and also comparing rebuild time for i.e, the same diff on a Cython/C file in both build systems
• produce some build process profiles to see where we spend most of our compile time (the tooling around this is one of the advantages of adopting meson/ninja).
• it might be nice if I could either address or suppress some of the compilation warnings so that if I do a "live demo" at the UGM folks can clearly see ninja zipping through our build targets without the terminal noise
• prepare some UGM presentation slides with the above timings/profiles, motivations for doing this

---

📚 Documentation preview 📚: https://mdanalysis--5122.org.readthedocs.build/en/5122/

[IAlibay]

@tylerjereddy - see discord, let me know if completing this might be worth getting an SDG for (edit: or if you're likely to get it done without by the UGM).

[tylerjereddy]

The latest commit is a large one--adding support for spin build and spin test, and migrating our full testsuite over to meson as well. It is now much easier to develop on/play with this branch. You can get the available options by simply doing cd package and then spin, which will show:

On a fresh checkout of this branch, you could build from source and run the full testsuite with a single command--locally on ARM Mac I tend to do spin test -j 12, which will do a 12-process build and a 12-process testsuite run right after it. As the commit message explains, the number of tests executed appears to match our "old school" way of building/running tests, so that's a good sign.

You can also be more selective, for exampling first building with 12 cores: spin build -j 12, and then running only the topology tests on 8 cores: spin test -t MDAnalysisTests/topology -j 8.

The build seems very fast, 9.8 seconds on 2 cores on M3 Mac locally, 7 seconds on 12 cores on M3 Mac locally, 2.9 seconds on 32 cores on i9-13900K Linux box. I'm not exactly sure if I can compare fairly against the "old" way of building, but I'd like to for the UGM talk.

The experience is very similar to developing with NumPy, SciPy; if we want to pass extra arguments to pytest we can just use --, i.e., to stop at the first test failure do: spin test -t MDAnalysisTests/utils -- -x.

The fact that MDAnalysisTests is not in the package source tree was annoying to deal with, and the exact solution we might want to use for this long-term may be different from what I did here (I used subprojects with a symlink to ../../testsuite).

[tylerjereddy]

One thing that would be useful to know is what the core developers think the "fastest" possible from-source build incantation is on current MDAnalysis develop branch--what would you type locally if all you cared about was build speed? For example: time python -m pip install -v . --no-build-isolation --no-deps? What about reducing the optimization level? Once I get a response or two about what folks think is the fastest way to build from source, I could use that as my reference point, since that would be what core developers practically think is best, regardless of whether or not it is a "fair" comparison.

[tylerjereddy]

In anticipation of the UGM, I pushed in draft pixi support--the cargo-like task/workflow support is likely of interest for scientists developing their own reproducible scientific workflows, while also potentially of interest to us as developers (especially because of lockfiles to keep things maximally reproducible...).

With pixi installed, after pulling in this feature branch it should suffice to do pixi install and then pixi run test to build/test MDAnalysis from source, with the locked down deps in the lockfile, which is pretty convenient for 0->running tests.