feat: enable type-safe interoperability between different independent Python/C++ bindings systems.

[rwgk]

Description

This PR enables type-safe interoperability between different independent Python/C++ bindings systems, including pybind11 versions with different PYBIND11_INTERNALS_VERSIONs. That means, innovations that require pybind11 internals version bumps become far less disruptive, almost to a negligible degree for most practical purposes (see NOTE below for the limitation).

More concretely, this PR enables extracting raw C++ pointers from Python instances of py::class_-wrapped types.

The implementation involves a

def _pybind11_conduit_v1_(
    self,
    pybind11_platform_abi_id: bytes,
    cpp_type_info_capsule: capsule,
    pointer_kind: bytes) -> capsule

method that is added unconditionally to all py::class_-wrapped types (to avoid "oh, too late!" situations). — The implementation of this method is identical for all types, i.e. it is not templated, therefore it compiles very quickly and adds very little binary size overhead. I.e. the cost is very small, while the benefit of maximizing interoperability is expected to be very large.

The design of the _pybind11_conduit_v1_ feature provides two layers of protection against C++ ABI mismatches:

• The first and most important layer is that the pybind11_platform_abi_ids must match between extensions. — This will never be perfect, but is the same pragmatic approach used in pybind11 since 2017 (PYBIND11_INTERNALS_ID).

• The second layer is that the typeid(std::type_info).name()s must match between extensions.

This PR includes calls of the _pybind11_conduit_v1_() method from:

• pybind11::detail::type_caster_generic (in pybind11/detail/type_caster_base.h, with helper functions in pybind11/detail/cpp_conduit.h)

• tests/exo_planet_c_api.cpp

The latter is a great way to quickly learn how the _pybind11_conduit_v1_ feature works. exo_planet_c_api.cpp only uses the plain CPython API, not any pybind11 features, except one macro resolving to a C++ string literal: PYBIND11_PLATFORM_ABI_ID.

The integration into type_caster_generic means that any object, from any Python/C++ bindings system (e.g. in theory at the moment SWIG, Cython, PyCLIF, nanobind) providing a compatible self._pybind11_conduit_v1_() method can be passed from Python to pybind11-wrapped C++ functions.

For entertainment: This PR makes pirate-like approaches to getting to the wrapped C++ pointers obsolete. Here is a particularly creative piece of code enabling SWIG-pybind11 interoperability, exactly as found in the wild, in an undisclosed location.

NOTE: Matching PYBIND11_INTERNALS_IDs are still critically important for this situation:

• py::class_<base> in one extension,
• py::class_<derived, base> in another.

For a demonstration, please see test_home_planet_wrap_very_lonely_traveler() and test_exo_planet_pybind11_wrap_very_lonely_traveler() in tests/test_cpp_conduit.py.

---

This PR passed Google-global testing:

• merged into the smart_holder branch: internal ID OCL:669841401:BASE:669907294:1725198946622:db3091a3
• merged into pybind11clif, then running with PyCLIF-pybind11: internal ID OCL:669954247:BASE:670062753:1725250171762:cb476a31

---

Suggestions for reviewing this PR (or just understanding what it does in detail):

• Start with test/exo_planet_c_api.cpp

• Review cpp_conduit_method() in pybind11/detail/type_caster_base.h, which is the only implementation in this PR building an object to be returned by self._pybind11_conduit_v1_(). — Other bindings systems would have their own implementations.

• The most interesting part of this PR is this small code fragment in cpp_conduit_method():

    const auto *cpp_type_info = cap_cpp_type_info.get_pointer<const std::type_info>();
    type_caster_generic caster(*cpp_type_info);
    if (!caster.load(self, false)) {
        return none();
    }

Note that this takes care of all inheritance-related pointer conversions (up- and down-casts).

• Review the rest of the changes in include/pybind11/detail/type_caster_base.h, which calls helper functions in:

• include/pybind11/detail/cpp_conduit.h

• Review the very simple change in include/pybind11/detail/internals.h, which just factors out PYBIND11_PLATFORM_ABI_ID from PYBIND11_INTERNALS_ID.

• Review the very simple changes in include/pybind11/pybind11.h. — Note that the auto-magically added _pybind11_conduit_v1_ method is not included in docstring signatures, to not clutter the docstrings.

• Review the added tests.

---

Beyond this PR:

---

Please take a quick look at the top of tests/exo_planet_c_api.cpp.

It is unfortunate that bindings systems other than pybind11 need to #include <pybind11/pybind11.h> just to get to the PYBIND11_PLATFORM_ABI_ID macro (which resolves to a simple C++ string literal). This leads to significant problems in environments that have C++ exception handling disabled (e.g. via -fno-exceptions).

An attempt was made to establish an alternative that maximizes reusability, but the implementation requires refactoring of pybind11/detail/common.h and pybind11/detail/internals.h. One particular difficulty is that PYBIND11_PLATFORM_ABI_ID can only be built after #include <Python.h>, because that transitively #includes system headers that define some of the system macros used in building PYBIND11_PLATFORM_ABI_ID. To not make this fairly simple PR significantly more complicated, the changes for the attempt were backed out. The pure refactoring work is best handled in a follow-on PR.

FWIW: In an ideal world, PYBIND11_PLATFORM_ABI_ID would be spelled as PY_CPP_COMMUNITY_PLATFORM_ABI_ID or similar, and come in a tiny additional header file included with CPython releases, so that all Python/C++ bindings systems could standardize on it.

---

Currently b"raw_pointer_ephemeral" (see above) is the only supported pointer_kind. Ideas for the future go in the direction of b"std::shared_ptr", b"std::unique_ptr", which will probably be relatively simple to add after the smart_holder branch is merged. — It is not clear (to rwgk@) if those additions will be worth the effort and added complexity, but having pointer_kind from the start is expressive and leaves the door open for future developments. — Additionally, the pointer_kind could be leveraged for implementing a b"__diagnose__" (or similar) feature, to report why retrieving a desired C++ pointer is failing.

---

mypy stubgenc will generate this additional signature:

    def _pybind11_conduit_v1_(self, *args, **kwargs) -> Any: ...

If this is undesired by the community (?), the signature could be suppressed similar to how dunder methods are suppressed here.

---

For completeness, the starting point for this PR was PR #3633 & PR #3788, although this PR is very different in many details.

Suggested changelog entry:

A new ``self._pybind11_conduit_v1_()`` method is automatically added to all ``py::class_``-wrapped types, to enable type-safe interoperability between different independent Python/C++ bindings systems, including pybind11 versions with different ``PYBIND11_INTERNALS_VERSION``s.

[ax3l]

This looks like a nice feature to add, e.g., for faster innovation where currently breaking changes are needed. The implementation looks low overhead and sensible to me on first few.

For the concern of long term compatibility:
If we added a spec version to the matched criteria for compatibility, and then tried it out in a few implementations (say nanobind, pybind11), we could experiment it while proposing it for a PEP.
This is the way array interface and cuda array interface were introduced and allows us to propose something we tried out in practice. We can also clearly mark it experimental for the first version.

[henryiii]

Would it be possible to make this internal (and maybe even experimental) for now while we work on a spec? Specifically, if this gets merged, then __cpp_conduit__ becomes specified, and it's hard to change that specification. Also, only Python is officially allowed to use dunder methods; I know some libraries (like NumPy) also add some, but Python's stance is that all dunder names are reserved for Python's use. I think it would be okay to write something a little less official for one, but I don't think we should introduce one quite yet, especially without any specification .

I think if we made this _pybind11_conduit_v1_, for example, then we could have time to work on a proper written spec with buy-in from nanobind and hopefully others, with a more general name, while still getting the benefits for our own use immediately.

[rwgk]

Sorry for the silence. I'll change this to _pybind11_conduit_v1_ as soon as I have access to a Linux workstation again. (Unfortunately that may be next week only.)

Thanks for the suggestions!

[henryiii]

I know it shows up a few other places, but I think this is frowned upon by lawyers from discussing this in context of other not-legally defined entities. What does the "The pybind Community" mean?

[rwgk]

Thanks a lot @henryiii for jumping in!

I just updated the PR description, to systematically replace __cpp_conduit__ with _pybind_conduit_v1_, and a minor edit related to mypy (since the new method name isn't a dunder name anymore it'll need something more special in mypy).

I'll now play a bit with py::class_<base> in one extension, py::class_<derived, base> in another, with non-matching PYBIND11_INTERNALS_ID, to validate my understanding, before maybe adding an explanation to the PR description. — I don't expect that this will lead to any production code changes.

[rwgk]

I'll now play a bit with py::class_<base> in one extension, py::class_<derived, base> in another, with non-matching PYBIND11_INTERNALS_ID, to validate my understanding, before maybe adding an explanation to the PR description. — I don't expect that this will lead to any production code changes.

Done: 22dbfa6

@henryiii if this looks good to you, please merge.

I volunteer to backport this to 2.11 and 2.12.