Skip to content

Commit

Permalink
README_smart_holder.rst update (line count reduced from 356 to 123).
Browse files Browse the repository at this point in the history
  • Loading branch information
Ralf W. Grosse-Kunstleve committed Jul 27, 2024
1 parent 0ca3ca7 commit 4a7f895
Showing 1 changed file with 54 additions and 287 deletions.
341 changes: 54 additions & 287 deletions README_smart_holder.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,58 +6,71 @@ pybind11 — smart_holder branch
Overview
========

- The smart_holder git branch is a strict superset of the master
branch. Everything that works on master is expected to work exactly the same
with the smart_holder branch.

- **Smart-pointer interoperability** (``std::unique_ptr``, ``std::shared_ptr``)
is implemented as an **add-on**.

- The add-on also supports
* passing a Python object back to C++ via ``std::unique_ptr``, safely
**disowning** the Python object.
* safely passing `"trampoline"
<https://pybind11.readthedocs.io/en/stable/advanced/classes.html#overriding-virtual-functions-in-python>`_
objects (objects with C++ virtual function overrides implemented in
Python) via ``std::unique_ptr`` or ``std::shared_ptr`` back to C++:
associated Python objects are automatically kept alive for the lifetime
of the smart-pointer.

- The smart_holder branch can be used in two modes:
* **Conservative mode**: ``py::class_`` works exactly as on master.
``py::classh`` uses ``py::smart_holder``.
* **Progressive mode**: ``py::class_`` uses ``py::smart_holder``
(i.e. ``py::smart_holder`` is the default holder).
- The smart_holder branch is a strict superset of the pybind11 master branch.
Everything that works with the master branch is expected to work exactly the
same with the smart_holder branch.

- Activating the smart_holder functionality for a given C++ type ``T`` is as
easy as changing ``py::class_<T>`` to ``py::classh<T>`` in client code.

- The ``py::classh<T>`` functionality includes

* support for **two-way** Python/C++ conversions for both
``std::unique_ptr<T>`` and ``std::shared_ptr<T>`` **simultaneously**.
— In contrast, ``py::class_<T>`` only supports one-way C++-to-Python
conversions for ``std::unique_ptr<T>``, or alternatively two-way
Python/C++ conversions for ``std::shared_ptr<T>``, which then excludes
the one-way C++-to-Python ``std::unique_ptr<T>`` conversions (this
manifests itself through undefined runtime behavior).

* passing a Python object back to C++ via ``std::unique_ptr<T>``, safely
**disowning** the Python object.

* safely passing `"trampoline"
<https://pybind11.readthedocs.io/en/stable/advanced/classes.html#overriding-virtual-functions-in-python>`_
objects (objects with C++ virtual function overrides implemented in
Python) via ``std::unique_ptr<T>`` or ``std::shared_ptr<T>`` back to C++:
associated Python objects are automatically kept alive for the lifetime
of the smart-pointer.

Note: As of `PR #5257 <https://github.com/pybind/pybind11/pull/5257>`_
the smart_holder functionality is fully baked into pybind11.
Prior to PR #5257 the smart_holder implementation was an "add-on", which made
it necessary to use a ``PYBIND11_SMART_HOLDER_TYPE_CASTERS`` macro. This macro
still exists for backward compatibility, but is now a no-op. The trade-off
for this convenience is that the ``PYBIND11_INTERNALS_VERSION`` needed to be
changed. Consequently, Python extension modules built with the smart_holder
branch no longer interoperate with extension modules built with the pybind11
master branch. If cross-extension-module interoperability is required, all
extension modules involved must be built with the smart_holder branch.
— Probably, most extension modules do not require cross-extension-module
interoperability, but exceptions to this are quite common.


What is fundamentally different?
--------------------------------

- Classic pybind11 has the concept of "smart-pointer is holder".
Interoperability between smart-pointers is completely missing. For
example, when using ``std::shared_ptr`` as holder, ``return``-ing
a ``std::unique_ptr`` leads to undefined runtime behavior
(`#1138 <https://github.com/pybind/pybind11/issues/1138>`_). A
`systematic analysis is here <https://github.com/pybind/pybind11/pull/2672#issuecomment-748392993>`_.
Interoperability between smart-pointers is completely missing. For example,
with ``py::class_<T, std::shared_ptr<T>>``, ``return``-ing a
``std::unique_ptr<T>`` leads to undefined runtime behavior
(`#1138 <https://github.com/pybind/pybind11/issues/1138>`_).
A `systematic analysis can be found here
<https://github.com/pybind/pybind11/pull/2672#issuecomment-748392993>`_.

- ``py::smart_holder`` has a richer concept in comparison, with well-defined
runtime behavior. The holder "knows" about both ``std::unique_ptr`` and
``std::shared_ptr`` and how they interoperate.

- Caveat (#HelpAppreciated): currently the ``smart_holder`` branch does
not have a well-lit path for including interoperability with custom
smart-pointers. It is expected to be a fairly obvious extension of the
``smart_holder`` implementation, but will depend on the exact specifications
of each custom smart-pointer type (generalizations are very likely possible).
runtime behavior in all situations. ``py::smart_holder`` "knows" about both
``std::unique_ptr<T>`` and ``std::shared_ptr<T>``, and how they interoperate.


What motivated the development of the smart_holder code?
--------------------------------------------------------

- Necessity is the mother. The bigger context is the ongoing retooling of
`PyCLIF <https://github.com/google/clif/>`_, to use pybind11 underneath
instead of directly targeting the Python C API. Essentially, the smart_holder
branch is porting established PyCLIF functionality into pybind11.
- The original context was retooling of `PyCLIF
<https://github.com/google/clif/>`_, to use pybind11 underneath,
instead of directly targeting the Python C API. Essentially the smart_holder
branch is porting established PyCLIF functionality into pybind11. (However,
this work also led to bug fixes in PyCLIF.)


Installation
Expand All @@ -72,225 +85,6 @@ Currently ``git clone`` is the only option. We do not have released packages.
Everything else is exactly identical to using the default (master) branch.


Conservative or Progressive mode?
=================================

It depends. To a first approximation, for a stand-alone, new project, the
Progressive mode will be easiest to use. For larger projects or projects
that integrate with third-party pybind11-based projects, the Conservative
mode may be more practical, at least initially, although it comes with the
disadvantage of having to use the ``PYBIND11_SMART_HOLDER_TYPE_CASTERS`` macro.


Conservative mode
-----------------

Here is a minimal example for wrapping a C++ type with ``py::smart_holder`` as
holder:

.. code-block:: cpp
#include <pybind11/smart_holder.h>
struct Foo {};
PYBIND11_SMART_HOLDER_TYPE_CASTERS(Foo)
PYBIND11_MODULE(example_bindings, m) {
namespace py = pybind11;
py::classh<Foo>(m, "Foo");
}
There are three small differences compared to Classic pybind11:

- ``#include <pybind11/smart_holder.h>`` is used instead of
``#include <pybind11/pybind11.h>``.

- The ``PYBIND11_SMART_HOLDER_TYPE_CASTERS(Foo)`` macro is needed.
— NOTE: This macro needs to be in the global namespace.

- ``py::classh`` is used instead of ``py::class_``.

To the 2nd bullet point, the ``PYBIND11_SMART_HOLDER_TYPE_CASTERS`` macro
needs to appear in all translation units with pybind11 bindings that involve
Python⇄C++ conversions for ``Foo``. This is the biggest inconvenience of the
Conservative mode. Practically, at a larger scale it is best to work with a
pair of ``.h`` and ``.cpp`` files for the bindings code, with the macros in
the ``.h`` files.

To the 3rd bullet point, ``py::classh<Foo>`` is simply a shortcut for
``py::class_<Foo, py::smart_holder>``. The shortcut makes it possible to
switch to using ``py::smart_holder`` without disturbing the indentation of
existing code.

When migrating code that uses ``py::class_<Bar, std::shared_ptr<Bar>>``
there are two alternatives. The first one is to use ``py::classh<Bar>``:

.. code-block:: diff
- py::class_<Bar, std::shared_ptr<Bar>>(m, "Bar");
+ py::classh<Bar>(m, "Bar");
This is clean and simple, but makes it difficult to fall back to Classic
mode if needed. The second alternative is to replace ``std::shared_ptr<Bar>``
with ``PYBIND11_SH_AVL(Bar)``:

.. code-block:: diff
- py::class_<Bar, std::shared_ptr<Bar>>(m, "Bar");
+ py::class_<Bar, PYBIND11_SH_AVL(Bar)>(m, "Bar");
The ``PYBIND11_SH_AVL`` macro substitutes ``py::smart_holder``
in Conservative mode, or ``std::shared_ptr<Bar>`` in Classic mode.
See tests/test_classh_mock.cpp for an example. Note that the macro is also
designed to not disturb the indentation of existing code.


Progressive mode
----------------

To work in Progressive mode:

- Add ``-DPYBIND11_USE_SMART_HOLDER_AS_DEFAULT`` to the compilation commands.

- Remove or replace (see below) ``std::shared_ptr<...>`` holders.

- Only if custom smart-pointers are used: the
``PYBIND11_TYPE_CASTER_BASE_HOLDER`` macro is needed (see
tests/test_smart_ptr.cpp for examples).

Overall this is probably easier to work with than the Conservative mode, but

- the macro inconvenience is shifted from ``py::smart_holder`` to custom
smart-pointer holders (which are probably much more rare).

- it will not interoperate with other extensions built against master or
stable, or extensions built in Conservative mode (see the cross-module
compatibility section below).

When migrating code that uses ``py::class_<Bar, std::shared_ptr<Bar>>`` there
are the same alternatives as for the Conservative mode (see previous section).
An additional alternative is to use the ``PYBIND11_SH_DEF(...)`` macro:

.. code-block:: diff
- py::class_<Bar, std::shared_ptr<Bar>>(m, "Bar");
+ py::class_<Bar, PYBIND11_SH_DEF(Bar)>(m, "Bar");
The ``PYBIND11_SH_DEF`` macro substitutes ``py::smart_holder`` only in
Progressive mode, or ``std::shared_ptr<Bar>`` in Classic or Conservative
mode. See tests/test_classh_mock.cpp for an example. Note that the
``PYBIND11_SMART_HOLDER_TYPE_CASTERS`` macro is never needed in combination
with the ``PYBIND11_SH_DEF`` macro, which is an advantage compared to the
``PYBIND11_SH_AVL`` macro. Please review tests/test_classh_mock.cpp for a
concise overview of all available options.


Transition from Classic to Progressive mode
-------------------------------------------

This still has to be tried out more in practice, but in small-scale situations
it may be feasible to switch directly to Progressive mode in a break-fix
fashion. In large-scale situations it seems more likely that an incremental
approach is needed, which could mean incrementally converting ``py::class_``
to ``py::classh`` and using the family of related macros, then flip the switch
to Progressive mode, and convert ``py::classh`` back to ``py:class_`` combined
with removal of the macros if desired (at that point it will work equivalently
either way). It may be smart to delay the final cleanup step until all
third-party projects of interest have made the switch, because then the code
will continue to work in all modes.


Using py::smart_holder but with fallback to Classic pybind11
------------------------------------------------------------

For situations in which compatibility with Classic pybind11
(without smart_holder) is needed for some period of time, fallback
to Classic mode can be enabled by copying the ``BOILERPLATE`` code
block from tests/test_classh_mock.cpp. This code block provides mock
implementations of ``py::classh`` and the family of related macros
(e.g. ``PYBIND11_SMART_HOLDER_TYPE_CASTERS``).


Classic / Conservative / Progressive cross-module compatibility
---------------------------------------------------------------

Currently there are essentially three modes for building a pybind11 extension
module:

- Classic: pybind11 stable (e.g. v2.6.2) or current master branch.

- Conservative: pybind11 smart_holder branch.

- Progressive: pybind11 smart_holder branch with
``-DPYBIND11_USE_SMART_HOLDER_AS_DEFAULT``.

In environments that mix extension modules built with different modes,
this is the compatibility matrix for ``py::class_``-wrapped types:

.. list-table:: Compatibility matrix
:widths: auto
:header-rows: 2

* -
-
-
- Module 2
-
* -
-
- Classic
- Conservative
- Progressive
* -
- **Classic**
- full
- one-and-a-half-way
- isolated
* - **Module 1**
- **Conservative**
- one-and-a-half-way
- full
- isolated
* -
- **Progressive**
- isolated
- isolated
- full

Mixing Classic+Progressive or Conservative+Progressive is very easy to
understand: the extension modules are essentially completely isolated from
each other. This is in fact just the same as using pybind11 versions with
differing `"internals version"
<https://github.com/pybind/pybind11/blob/114be7f4ade0ad798cd4c7f5d65ebe4ba8bd892d/include/pybind11/detail/internals.h#L95>`_
in the past. While this is easy to understand, there is also no incremental
transition path between Classic and Progressive.

The Conservative mode enables incremental transitions, but at the cost of
more complexity. Types wrapped in a Classic module are fully compatible with
a Conservative module. However, a type wrapped in a Conservative module is
compatible with a Classic module only if ``py::smart_holder`` is **not** used
(for that type). A type wrapped with ``py::smart_holder`` is incompatible with
a Classic module. This is an important pitfall to keep in mind: attempts to use
``py::smart_holder``-wrapped types in a Classic module will lead to undefined
runtime behavior, such as a SEGFAULT. This is a more general flavor of the
long-standing issue `#1138 <https://github.com/pybind/pybind11/issues/1138>`_,
often referred to as "holder mismatch". It is important to note that the
pybind11 smart_holder branch solves the smart-pointer interoperability issue,
but not the more general holder mismatch issue. — Unfortunately the existing
pybind11 internals do not track holder runtime type information, therefore
the holder mismatch issue cannot be solved in a fashion that would allow
an incremental transition, which is the whole point of the Conservative
mode. Please proceed with caution. (See `PR #2644
<https://github.com/pybind/pybind11/pull/2644>`_ for background, which is
labeled with "abi break".)

Another pitfall worth pointing out specifically, although it follows
from the previous: mixing base and derived classes between Classic and
Conservative modules means that neither the base nor the derived class can
use ``py::smart_holder``.


Trampolines and std::unique_ptr
-------------------------------

Expand All @@ -307,37 +101,10 @@ inherit from ``py::trampoline_self_life_support``, for example:
...
};
This is the only difference compared to Classic pybind11. A fairly
This is the only difference compared to classic pybind11. A fairly
minimal but complete example is tests/test_class_sh_trampoline_unique_ptr.cpp.


Ideas for the long-term
-----------------------

The macros are clearly an inconvenience in many situations. Highly
speculative: to avoid the need for the macros, a potential approach would
be to combine the Classic implementation (``type_caster_base``) with
the ``smart_holder_type_caster``, but this will probably be very messy and
not great as a long-term solution. The ``type_caster_base`` code is very
complex already. A more maintainable approach long-term could be to work
out and document a smart_holder-based solution for custom smart-pointers
in pybind11 version ``N``, then purge ``type_caster_base`` in version
``N+1``. #HelpAppreciated.


Testing of PRs against the smart_holder branch
----------------------------------------------

In the pybind11 GitHub Actions, PRs against the smart_holder branch are
automatically tested in both modes (Conservative, Progressive), with the
only difference that ``PYBIND11_USE_SMART_HOLDER_AS_DEFAULT`` is defined
for Progressive mode testing.

For interactive testing, the ``PYBIND11_USE_SMART_HOLDER_AS_DEFAULT``
define needs to be manually added to the cmake command. See
.github/workflows/ci_sh.yml for examples.


Related links
=============

Expand All @@ -353,4 +120,4 @@ Related links
* Small `slide deck
<https://docs.google.com/presentation/d/1r7auDN0x-b6uf-XCvUnZz6z09raasRcCHBMVDh7PsnQ/>`_
presented in meeting with pybind11 maintainers on Feb 22, 2021. Slides 5
and 6 show performance comparisons.
and 6 show performance comparisons. (These are outdated but probably not far off.)

0 comments on commit 4a7f895

Please sign in to comment.