[Python] Build/upload platform wheels

[turicas]

Main idea

Many users face the issue of dlib Python package taking too long to install. Python provides a way to build and publish pre-compiled platform wheels, which would significantly speed up dlib's installation for everyone and eliminate the need for extra system packages like cmake. For instance, I recently tried installing it, and it took approximately 4 minutes on my local machine and around 9 minutes on a low-performance staging server.

Related to #2610.

Anything else?

A good example of this practice is the psycopg2-binary library. If you look at its list of files on PyPI, you’ll find one source package (the first entry) along with 66 pre-compiled wheels for each library version. Below is the list of files for version 2.9.10:

List of filenames

psycopg2-binary-2.9.10.tar.gz
psycopg2_binary-2.9.10-cp310-cp310-macosx_12_0_x86_64.whl
psycopg2_binary-2.9.10-cp310-cp310-macosx_14_0_arm64.whl
psycopg2_binary-2.9.10-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
psycopg2_binary-2.9.10-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl
psycopg2_binary-2.9.10-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
psycopg2_binary-2.9.10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
psycopg2_binary-2.9.10-cp310-cp310-musllinux_1_2_aarch64.whl
psycopg2_binary-2.9.10-cp310-cp310-musllinux_1_2_i686.whl
psycopg2_binary-2.9.10-cp310-cp310-musllinux_1_2_ppc64le.whl
psycopg2_binary-2.9.10-cp310-cp310-musllinux_1_2_x86_64.whl
psycopg2_binary-2.9.10-cp310-cp310-win32.whl
psycopg2_binary-2.9.10-cp310-cp310-win_amd64.whl
psycopg2_binary-2.9.10-cp311-cp311-macosx_12_0_x86_64.whl
psycopg2_binary-2.9.10-cp311-cp311-macosx_14_0_arm64.whl
psycopg2_binary-2.9.10-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
psycopg2_binary-2.9.10-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl
psycopg2_binary-2.9.10-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
psycopg2_binary-2.9.10-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
psycopg2_binary-2.9.10-cp311-cp311-musllinux_1_2_aarch64.whl
psycopg2_binary-2.9.10-cp311-cp311-musllinux_1_2_i686.whl
psycopg2_binary-2.9.10-cp311-cp311-musllinux_1_2_ppc64le.whl
psycopg2_binary-2.9.10-cp311-cp311-musllinux_1_2_x86_64.whl
psycopg2_binary-2.9.10-cp311-cp311-win32.whl
psycopg2_binary-2.9.10-cp311-cp311-win_amd64.whl
psycopg2_binary-2.9.10-cp312-cp312-macosx_12_0_x86_64.whl
psycopg2_binary-2.9.10-cp312-cp312-macosx_14_0_arm64.whl
psycopg2_binary-2.9.10-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
psycopg2_binary-2.9.10-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl
psycopg2_binary-2.9.10-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
psycopg2_binary-2.9.10-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
psycopg2_binary-2.9.10-cp312-cp312-musllinux_1_2_aarch64.whl
psycopg2_binary-2.9.10-cp312-cp312-musllinux_1_2_i686.whl
psycopg2_binary-2.9.10-cp312-cp312-musllinux_1_2_ppc64le.whl
psycopg2_binary-2.9.10-cp312-cp312-musllinux_1_2_x86_64.whl
psycopg2_binary-2.9.10-cp312-cp312-win32.whl
psycopg2_binary-2.9.10-cp312-cp312-win_amd64.whl
psycopg2_binary-2.9.10-cp313-cp313-macosx_12_0_x86_64.whl
psycopg2_binary-2.9.10-cp313-cp313-macosx_14_0_arm64.whl
psycopg2_binary-2.9.10-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
psycopg2_binary-2.9.10-cp313-cp313-manylinux_2_17_i686.manylinux2014_i686.whl
psycopg2_binary-2.9.10-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
psycopg2_binary-2.9.10-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
psycopg2_binary-2.9.10-cp313-cp313-musllinux_1_2_aarch64.whl
psycopg2_binary-2.9.10-cp313-cp313-musllinux_1_2_i686.whl
psycopg2_binary-2.9.10-cp313-cp313-musllinux_1_2_ppc64le.whl
psycopg2_binary-2.9.10-cp313-cp313-musllinux_1_2_x86_64.whl
psycopg2_binary-2.9.10-cp38-cp38-macosx_12_0_x86_64.whl
psycopg2_binary-2.9.10-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
psycopg2_binary-2.9.10-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl
psycopg2_binary-2.9.10-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
psycopg2_binary-2.9.10-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
psycopg2_binary-2.9.10-cp38-cp38-musllinux_1_2_aarch64.whl
psycopg2_binary-2.9.10-cp38-cp38-musllinux_1_2_i686.whl
psycopg2_binary-2.9.10-cp38-cp38-musllinux_1_2_ppc64le.whl
psycopg2_binary-2.9.10-cp38-cp38-musllinux_1_2_x86_64.whl
psycopg2_binary-2.9.10-cp39-cp39-macosx_12_0_x86_64.whl
psycopg2_binary-2.9.10-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
psycopg2_binary-2.9.10-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl
psycopg2_binary-2.9.10-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
psycopg2_binary-2.9.10-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
psycopg2_binary-2.9.10-cp39-cp39-musllinux_1_2_aarch64.whl
psycopg2_binary-2.9.10-cp39-cp39-musllinux_1_2_i686.whl
psycopg2_binary-2.9.10-cp39-cp39-musllinux_1_2_ppc64le.whl
psycopg2_binary-2.9.10-cp39-cp39-musllinux_1_2_x86_64.whl
psycopg2_binary-2.9.10-cp39-cp39-win32.whl
psycopg2_binary-2.9.10-cp39-cp39-win_amd64.whl

[turicas]

I was thinking if it's possible to skip the building phase during pip install if dlib is already installed on the system. This would solve the issue for the majority of the users, I think (on Debian I could just run apt install -y libdlib19.1 before pip install dlib).

[turicas]

There's a tool from Python Packaging Authority called cibuildwheel which makes it easier to build wheels for different operating systems and Python versions using a CI like Github Actions. Here is an example from psycopg2 for building wheels for different Python versions, architectures and C standard libraries in Linux.
This diagram from its docs summarises the steps it takes on each platform:

Meanwhile, I'm running pip install --no-cache-dir dlib in a prior step on my Dockerfile so it'll be cached by Docker and avoid building the library on every deployment/container build.

[davisking]

The dlib package in apt is for C++ users only. It doesn't get you python bindings so doesn't help.

I know about precompiled python packages. The problem is most (or a lot anyway, it's hard to really know) people using dlib in python want it to be compiled with the various hardware accelerators they have on their system. E.g. AVX instructions, neon instructions, use the GPU, whatever. And that requires a binary that is ticking the right boxes for whatever their hardware is. The precompiled packages are generally lowest common denominator binaries that include none of that.

It's possible to write python packages that do everything (pytorch does that for instance). But making and maintaining such binaries is a huge pain and very time consuming. I do not want to do that :)

If someone else wants to do that then that would be neat though. But they will be constantly asked by people about why id doesn't work on their particular platform. Which is why I don't want to do it :)