feat: Python wheels workflow and build backend

[zachlewis]

Description

Summary

This PR is the spiritual successor to #4011. It implements a scikit-build-core-based python build-backend, making it possible to use pip install . to build from source; and it adds a Github workflow for building with cibuildwheel and publishing to pypi.org binary distributions (bdists) of the Python module / extensions / CLI tools for cpython 3.8-3.13, across major operating systems and architectures.

When you pip install OpenImageIO, pip attempts to retrieve an OpenImageIO bdist from pypi.org for the host's platform / architecture / Python interpreter. If it can't find something appropriate, pip will attempt to build locally from the OpenImageIO source distribution (sdist), downloading and temporarily installing cmake and ninja if necessary.

PEP-Compliant Packaging: pyproject.toml

The pyproject.toml file is organized in three parts:

1. Package metadata: standard attributes identifying and describing the Python package, its run-time and build-time requirements, entry-points to executable scripts, and so forth.
2. scikit-build-core options: governs how pip install ... interacts with cmake.
3. cibuildwheel options: additional steps and considerations for building, packaging, and testing relocatable wheel build artifacts in isolated environments.

Additions to __ init __.py

Previously, we were using a custom OpenImageIO/__ init__.py file to help Python-3.8+ on Windows load the shared libraries linked by the Python module (i.e., the .dll files that live alongside oiiotool.exe under $PATH).

This PR adds an additional method for loading the DLL path, necessitated by differences between pip-based and traditional CMake-based installs.

It also adds a mechanism for invoking binary executables found in the .../site-packages/OpenImageIO/bin directory. This provides a means for exposing Python script "shims" for each CLI tool, installed to platform-specific locations under $PATH, while keeping the actual binaries in a static location relative to the dynamic libraries. Upshot is, in pyproject.toml,
each item under [project.scripts] is turned into a Python script upon installation that behaves indistinguishably to the end user to the CLI binary executable of the same name.

Relocatable Binary Distributions with cibuildwheel + repairwheel

cibuildwheel is a widely-used tool for drastically streamlining the process of building, repairing, and testing Python wheels across platforms, architectures, interpreters, and interpreter versions.

Additionally, the cibuildwheel-based builds set CMAKE_BUILD_TYPE to "MinSizeRel" to optimize for size (instead of speed) -- this seems to shave ~1.5MB off each .whl's size, compared to "Release"

"Wheels" Github workflow

I straight-up copied .github/workflows/wheel.yml from OpenColorIO and made a few OIIO-specific modifications. When pushing a commit tagged v3*, the workflow will invoke a platform-agnostic "build sdist" (source distribution) task, followed by a series of tasks for building OIIO wheels for cpython-3.8-3.13 on Windows, Linux (x86_64 + aarch64, new libstdc++), and MacOS (x86_64 + arm64) and persisting build artifacts; followed finally by a task for publishing the build artifacts to pypi.org

Note: For the sake of simplicity and troubleshooting, I've made as few changes to OpenColorIO's wheel.yml as I could get away with; but in the future, we can also build wheels for the PyPy interpreter, and possibly pyodide.

Note: A "trusted publisher" must be set up on pypi.org. See https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/

Other Changes

I made some minor adjustments to pythonutils.cmake and fancy_add_executable.cmake that only affect scikit-build-core-based installs:

• -- namely, on Linux and macOS, I'm setting the INSTALL_RPATH property to point to the relative path to the dynamic libraries, for the Python module and CLI tools, respectively. This helps ensure that pip-based builds and installs from source (as opposed to installs from repaired, pre-built wheels) yield relocatable, importable packages, without needing to mess with $LD_LIBRARY_PATH etc.

Tests

cibuildwheel tests if oiiotool --buildinfo runs. If that command elicits code zero, it means the "oiiotool" Python script installed by the wheel is able to import OpenImageIO; that the actual binary executable oiiotool is properly packaged and exists in the expected location (e.g., at .../site-packages/OpenImageIO/bin); and that all runtime dependencies are found.

Inspiration, Credit, Prior Art

• @aclark4life's and @JeanChristopheMorinPerso's efforts + direction + discussion + advice. See #3249, and #4011, as well as JCM's python_wheels and python_wheels_windows branches. This PR is an attempt to leverage OIIO-2.6+ self-building-dependency features with # 4011's minimalist and modern approach to packaging.
• OpenColorIO -- I tried to copy as much as I could from @remia et al's fantastic work with all things wheels-related. The __init __.py modifications, the way we're wrapping the CLI tools, and the github Wheels workflow are lifted almost-verbatim from OCIO. Insert pun about reinventing the wheel here.
• @joaovbs96's help and patience with testing stuff on Windows.

[aclark4life]

Well played 😄

[zachlewis]

ha -- we'll see... what I've been trying has apparently broken everything. I'm not sure if I have enough time to debug this myself. We may have to split off the attempt to get rid of tasks.py into its own PR. Right now, it's solving a few problems at once.

[JeanChristopheMorinPerso]

Great work with the latest changes @zachlewis! I like how the extract step to fix the wheels are not needed anymore.

I started to review the generated artifacts. I see that the versioned libraries are not yet fully removed:

$ unzip -l openimageio-3.1.0.0-cp313-cp313-linux_x86_64.whl | grep libOpenImageIO
 11355768  2024-10-17 14:19   OpenImageIO/lib/libOpenImageIO.so
 11355768  2024-10-17 14:19   OpenImageIO/lib/libOpenImageIO.so.3.1.0
  1171672  2024-10-17 14:19   OpenImageIO/lib/libOpenImageIO_Util.so
  1171672  2024-10-17 14:19   OpenImageIO/lib/libOpenImageIO_Util.so.3.1.0

I also see

    33700  2024-10-17 14:11   OpenImageIO/share/doc/OpenImageIO/CHANGES-0.x.md
   223954  2024-10-17 14:11   OpenImageIO/share/doc/OpenImageIO/CHANGES-1.x.md
    60831  2024-10-17 14:11   OpenImageIO/share/doc/OpenImageIO/CHANGES.md
    11357  2024-10-17 14:11   OpenImageIO/share/doc/OpenImageIO/LICENSE.md
    13418  2024-10-17 14:11   OpenImageIO/share/doc/OpenImageIO/THIRD-PARTY.md
    42480  2024-10-17 14:11   OpenImageIO/share/fonts/OpenImageIO/DroidSans-Bold.ttf
    41028  2024-10-17 14:11   OpenImageIO/share/fonts/OpenImageIO/DroidSans.ttf
   117072  2024-10-17 14:11   OpenImageIO/share/fonts/OpenImageIO/DroidSansMono.ttf
    48880  2024-10-17 14:11   OpenImageIO/share/fonts/OpenImageIO/DroidSerif-Bold.ttf
    45652  2024-10-17 14:11   OpenImageIO/share/fonts/OpenImageIO/DroidSerif-BoldItalic.ttf
    40416  2024-10-17 14:11   OpenImageIO/share/fonts/OpenImageIO/DroidSerif-Italic.ttf
    43648  2024-10-17 14:11   OpenImageIO/share/fonts/OpenImageIO/DroidSerif.ttf

Are these really needed?

The OIIO libs have an absolute RPATH:

[jcmorin@arch01 asd]$ readelf -d OpenImageIO/lib/libOpenImageIO.so.3.1.0

Dynamic section at offset 0x9cb738 contains 35 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libOpenImageIO_Util.so.3.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libOpenColorIO_v_2_3_2_OIIO.so]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
 0x000000000000000e (SONAME)             Library soname: [libOpenImageIO.so.3.1.0]
 0x000000000000000f (RPATH)              Library rpath: [/tmp/tmpmkq88nni/wheel/platlib/OpenImageIO/lib]

Right now I think the dynamic linker will do its job only because OpenImageIO.cpython-313-x86_64-linux-gnu.so (the python extension) has the correct RPATH and RPATH is recursive (as opposed to RUNPATH, because yes there is RPATH and RUNPATH and they bahave slithly differently). See https://man7.org/linux/man-pages/man8/ld.so.8.html:

(3) Using the directories specified in the DT_RUNPATH dynamic
section attribute of the binary if present. Such
directories are searched only to find those objects required
by DT_NEEDED (direct dependencies) entries and do not apply
to those objects' children, which must themselves have their
own DT_RUNPATH entries. This is unlike DT_RPATH, which is
applied to searches for all children in the dependency tree.

(emphasis mine)

Though, the tests in CI seem to say that there is a problem with RPATHs since it can't find OCIO.

Lastly, the wheel filenames are wrong on Linux.

[JeanChristopheMorinPerso]

About the rpath, if I had to take a guess, I would guess that the culprit is

OpenImageIO/src/cmake/compiler.cmake

Lines 657 to 658 in c3c6b1e

	if (NOT CMAKE_INSTALL_RPATH)
	set (CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_FULL_LIBDIR}")

[zachlewis]

Hey @JeanChristopheMorinPerso -- once again, I can't tell you how much I appreciate you having a look!

I apologize, I left things in an inconsistent state over the weekend. Good news is, I do believe I've fixed remaining issues...!

Major changes:

1. Added a NAMELINK_SKIP option to the custom install_targets macro in compiler.cmake, and I'm using that option during scikit-build-core builds for installing libOpenImageIO and libOpenImageIO_Util libraries
2. Changed the default CMAKE_INSTALL_RPATH to $ORIGIN:$ORIGIN/lib
3. Explicitly set INSTALL_RPATH to $ORIGIN/lib or $ORIGIN/../lib for certain targets
4. Re-enabled the default CIBW step. (Fixes bad filenames)

About the rpath, if I had to take a guess, I would guess that the culprit is

OpenImageIO/src/cmake/compiler.cmake

Lines 657 to 658 in c3c6b1e

	if (NOT CMAKE_INSTALL_RPATH)
	set (CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_FULL_LIBDIR}")

You were totally right about this -- now I'm doing this:

if (NOT CMAKE_INSTALL_RPATH)
        if(APPLE)
            set(BASEPOINT @loader_path)
        else()
            set(BASEPOINT $ORIGIN)
        endif()
        set (CMAKE_INSTALL_RPATH ${BASEPOINT} ${BASEPOINT}/${CMAKE_INSTALL_LIBDIR})
endif ()

And now the OIIO libs have a relative RPATH:

[zach@alma OpenImageIO]$ readelf -d lib/libOpenImageIO.so.3.1.0 

Dynamic section at offset 0xc5df10 contains 34 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libOpenImageIO_Util.so.3.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libOpenColorIO_v_2_3_2_OIIO.so]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [libOpenImageIO.so.3.1.0]
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN:$ORIGIN/lib]

🤘

I've gotten rid of the docs. For the fonts, actually, what I'd like to do adjust __init __.py and append the relative dir to the env var "OPENIMAGEIO_FONTS". I like the idea of a cross-platform default font shipping with the module...

[JeanChristopheMorinPerso]

This is kind of reverting what was made before. You could pass -DCMAKE_INSTALL_RPATH on a per platform basis instead.

[zachlewis]

Well, before, I think CMAKE_INSTALL_RPATH was defaulting to an absolute path; whereas now, it uses loader-path-relative defaults. We could still pass the option on a per-platform basis, but I'm not sure how that would be advantageous...?

[JeanChristopheMorinPerso]

Thanks @zachlewis. I'm always happy to help!

[MrLixm]

Hey there,
First I want to say thanks to everyone involved in those wheels, the outcome will really be invaluable for some of us.

By curiosity I wanted to test early some those wheels so I downloaded one from the last action's artefacts. However, once installed I do notice that the whell doesn't include support for png, I only see:

openexr,tiff,jpeg,bmp,cineon,dds,dpx,fits,hdr,iff,null,pnm,psd,rla,sgi,softimage,targa,term,zfile

Looking at the previous comments I see that there was some trouble with png so I assume its removal is intentional for now. However, I see those comments seems to be for MacOS and I'm currently testing it on Windows. Did I just misunderstand and png is not yet available for all platform ?

Job I used: https://github.com/AcademySoftwareFoundation/OpenImageIO/actions/runs/11464854677/job/31902028113
(wheel openimageio-3.1.0.0-cp310-cp310-win_amd64.whl)

Feel free to ignore as I fully understand that this PR is still wip and I'm not supposed to use those wheels yet.
Best,
Liam.

[aclark4life]

Feel free to ignore as I fully understand that this PR is still wip and I'm not supposed to use those wheels yet.

Feedback welcome! @zachlewis or @JeanChristopheMorinPerso can probably provide more detail but yeah I think PNG is a WIP and FWIW here's Pillow's PNG build: https://github.com/python-pillow/Pillow/blob/main/.github/workflows/wheels-dependencies.sh

[zachlewis]

Heya --

This PR doesn't intend to do anything specific with PNG (or any other dependency) per se -- it assumes that all dependencies are provided by OIIO's own dependency self-building mechanisms. The reason why PNG isn't included yet is because we're still working on getting the PNG self-builds to work (see #4423). Once a working solution is merged, the Python wheels will automatically inherit PNG support.

(It's true, not long ago, we were disabling PNG for the MacOS ARM wheels earlier here, but that was just a temporary workaround for handling conflicting homebrew-managed libraries. Now we have a more robust CMake option IGNORE_HOMEBREWED_DEPS which forces the build system (and dependency sub-builds) to ignore typical homebrew prefixes.)

As for the status of this PR, I don't really intend to change anything, unless there's something specific you guys would like me to change.