From 4a7f895f4e87985d27286da9a650cc16914e234a Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Sat, 27 Jul 2024 01:15:35 -0700 Subject: [PATCH] README_smart_holder.rst update (line count reduced from 356 to 123). --- README_smart_holder.rst | 341 +++++++--------------------------------- 1 file changed, 54 insertions(+), 287 deletions(-) diff --git a/README_smart_holder.rst b/README_smart_holder.rst index 7943d5c6ca..05b1ab56fa 100644 --- a/README_smart_holder.rst +++ b/README_smart_holder.rst @@ -6,58 +6,71 @@ pybind11 — smart_holder branch Overview ======== -- The smart_holder git branch is a strict superset of the master - branch. Everything that works on master is expected to work exactly the same - with the smart_holder branch. - -- **Smart-pointer interoperability** (``std::unique_ptr``, ``std::shared_ptr``) - is implemented as an **add-on**. - -- The add-on also supports - * passing a Python object back to C++ via ``std::unique_ptr``, safely - **disowning** the Python object. - * safely passing `"trampoline" - `_ - objects (objects with C++ virtual function overrides implemented in - Python) via ``std::unique_ptr`` or ``std::shared_ptr`` back to C++: - associated Python objects are automatically kept alive for the lifetime - of the smart-pointer. - -- The smart_holder branch can be used in two modes: - * **Conservative mode**: ``py::class_`` works exactly as on master. - ``py::classh`` uses ``py::smart_holder``. - * **Progressive mode**: ``py::class_`` uses ``py::smart_holder`` - (i.e. ``py::smart_holder`` is the default holder). +- The smart_holder branch is a strict superset of the pybind11 master branch. + Everything that works with the master branch is expected to work exactly the + same with the smart_holder branch. + +- Activating the smart_holder functionality for a given C++ type ``T`` is as + easy as changing ``py::class_`` to ``py::classh`` in client code. + +- The ``py::classh`` functionality includes + + * support for **two-way** Python/C++ conversions for both + ``std::unique_ptr`` and ``std::shared_ptr`` **simultaneously**. + — In contrast, ``py::class_`` only supports one-way C++-to-Python + conversions for ``std::unique_ptr``, or alternatively two-way + Python/C++ conversions for ``std::shared_ptr``, which then excludes + the one-way C++-to-Python ``std::unique_ptr`` conversions (this + manifests itself through undefined runtime behavior). + + * passing a Python object back to C++ via ``std::unique_ptr``, safely + **disowning** the Python object. + + * safely passing `"trampoline" + `_ + objects (objects with C++ virtual function overrides implemented in + Python) via ``std::unique_ptr`` or ``std::shared_ptr`` back to C++: + associated Python objects are automatically kept alive for the lifetime + of the smart-pointer. + +Note: As of `PR #5257 `_ +the smart_holder functionality is fully baked into pybind11. +Prior to PR #5257 the smart_holder implementation was an "add-on", which made +it necessary to use a ``PYBIND11_SMART_HOLDER_TYPE_CASTERS`` macro. This macro +still exists for backward compatibility, but is now a no-op. The trade-off +for this convenience is that the ``PYBIND11_INTERNALS_VERSION`` needed to be +changed. Consequently, Python extension modules built with the smart_holder +branch no longer interoperate with extension modules built with the pybind11 +master branch. If cross-extension-module interoperability is required, all +extension modules involved must be built with the smart_holder branch. +— Probably, most extension modules do not require cross-extension-module +interoperability, but exceptions to this are quite common. What is fundamentally different? -------------------------------- - Classic pybind11 has the concept of "smart-pointer is holder". - Interoperability between smart-pointers is completely missing. For - example, when using ``std::shared_ptr`` as holder, ``return``-ing - a ``std::unique_ptr`` leads to undefined runtime behavior - (`#1138 `_). A - `systematic analysis is here `_. + Interoperability between smart-pointers is completely missing. For example, + with ``py::class_>``, ``return``-ing a + ``std::unique_ptr`` leads to undefined runtime behavior + (`#1138 `_). + A `systematic analysis can be found here + `_. - ``py::smart_holder`` has a richer concept in comparison, with well-defined - runtime behavior. The holder "knows" about both ``std::unique_ptr`` and - ``std::shared_ptr`` and how they interoperate. - -- Caveat (#HelpAppreciated): currently the ``smart_holder`` branch does - not have a well-lit path for including interoperability with custom - smart-pointers. It is expected to be a fairly obvious extension of the - ``smart_holder`` implementation, but will depend on the exact specifications - of each custom smart-pointer type (generalizations are very likely possible). + runtime behavior in all situations. ``py::smart_holder`` "knows" about both + ``std::unique_ptr`` and ``std::shared_ptr``, and how they interoperate. What motivated the development of the smart_holder code? -------------------------------------------------------- -- Necessity is the mother. The bigger context is the ongoing retooling of - `PyCLIF `_, to use pybind11 underneath - instead of directly targeting the Python C API. Essentially, the smart_holder - branch is porting established PyCLIF functionality into pybind11. +- The original context was retooling of `PyCLIF + `_, to use pybind11 underneath, + instead of directly targeting the Python C API. Essentially the smart_holder + branch is porting established PyCLIF functionality into pybind11. (However, + this work also led to bug fixes in PyCLIF.) Installation @@ -72,225 +85,6 @@ Currently ``git clone`` is the only option. We do not have released packages. Everything else is exactly identical to using the default (master) branch. -Conservative or Progressive mode? -================================= - -It depends. To a first approximation, for a stand-alone, new project, the -Progressive mode will be easiest to use. For larger projects or projects -that integrate with third-party pybind11-based projects, the Conservative -mode may be more practical, at least initially, although it comes with the -disadvantage of having to use the ``PYBIND11_SMART_HOLDER_TYPE_CASTERS`` macro. - - -Conservative mode ------------------ - -Here is a minimal example for wrapping a C++ type with ``py::smart_holder`` as -holder: - -.. code-block:: cpp - - #include - - struct Foo {}; - - PYBIND11_SMART_HOLDER_TYPE_CASTERS(Foo) - - PYBIND11_MODULE(example_bindings, m) { - namespace py = pybind11; - py::classh(m, "Foo"); - } - -There are three small differences compared to Classic pybind11: - -- ``#include `` is used instead of - ``#include ``. - -- The ``PYBIND11_SMART_HOLDER_TYPE_CASTERS(Foo)`` macro is needed. - — NOTE: This macro needs to be in the global namespace. - -- ``py::classh`` is used instead of ``py::class_``. - -To the 2nd bullet point, the ``PYBIND11_SMART_HOLDER_TYPE_CASTERS`` macro -needs to appear in all translation units with pybind11 bindings that involve -Python⇄C++ conversions for ``Foo``. This is the biggest inconvenience of the -Conservative mode. Practically, at a larger scale it is best to work with a -pair of ``.h`` and ``.cpp`` files for the bindings code, with the macros in -the ``.h`` files. - -To the 3rd bullet point, ``py::classh`` is simply a shortcut for -``py::class_``. The shortcut makes it possible to -switch to using ``py::smart_holder`` without disturbing the indentation of -existing code. - -When migrating code that uses ``py::class_>`` -there are two alternatives. The first one is to use ``py::classh``: - -.. code-block:: diff - - - py::class_>(m, "Bar"); - + py::classh(m, "Bar"); - -This is clean and simple, but makes it difficult to fall back to Classic -mode if needed. The second alternative is to replace ``std::shared_ptr`` -with ``PYBIND11_SH_AVL(Bar)``: - -.. code-block:: diff - - - py::class_>(m, "Bar"); - + py::class_(m, "Bar"); - -The ``PYBIND11_SH_AVL`` macro substitutes ``py::smart_holder`` -in Conservative mode, or ``std::shared_ptr`` in Classic mode. -See tests/test_classh_mock.cpp for an example. Note that the macro is also -designed to not disturb the indentation of existing code. - - -Progressive mode ----------------- - -To work in Progressive mode: - -- Add ``-DPYBIND11_USE_SMART_HOLDER_AS_DEFAULT`` to the compilation commands. - -- Remove or replace (see below) ``std::shared_ptr<...>`` holders. - -- Only if custom smart-pointers are used: the - ``PYBIND11_TYPE_CASTER_BASE_HOLDER`` macro is needed (see - tests/test_smart_ptr.cpp for examples). - -Overall this is probably easier to work with than the Conservative mode, but - -- the macro inconvenience is shifted from ``py::smart_holder`` to custom - smart-pointer holders (which are probably much more rare). - -- it will not interoperate with other extensions built against master or - stable, or extensions built in Conservative mode (see the cross-module - compatibility section below). - -When migrating code that uses ``py::class_>`` there -are the same alternatives as for the Conservative mode (see previous section). -An additional alternative is to use the ``PYBIND11_SH_DEF(...)`` macro: - -.. code-block:: diff - - - py::class_>(m, "Bar"); - + py::class_(m, "Bar"); - -The ``PYBIND11_SH_DEF`` macro substitutes ``py::smart_holder`` only in -Progressive mode, or ``std::shared_ptr`` in Classic or Conservative -mode. See tests/test_classh_mock.cpp for an example. Note that the -``PYBIND11_SMART_HOLDER_TYPE_CASTERS`` macro is never needed in combination -with the ``PYBIND11_SH_DEF`` macro, which is an advantage compared to the -``PYBIND11_SH_AVL`` macro. Please review tests/test_classh_mock.cpp for a -concise overview of all available options. - - -Transition from Classic to Progressive mode -------------------------------------------- - -This still has to be tried out more in practice, but in small-scale situations -it may be feasible to switch directly to Progressive mode in a break-fix -fashion. In large-scale situations it seems more likely that an incremental -approach is needed, which could mean incrementally converting ``py::class_`` -to ``py::classh`` and using the family of related macros, then flip the switch -to Progressive mode, and convert ``py::classh`` back to ``py:class_`` combined -with removal of the macros if desired (at that point it will work equivalently -either way). It may be smart to delay the final cleanup step until all -third-party projects of interest have made the switch, because then the code -will continue to work in all modes. - - -Using py::smart_holder but with fallback to Classic pybind11 ------------------------------------------------------------- - -For situations in which compatibility with Classic pybind11 -(without smart_holder) is needed for some period of time, fallback -to Classic mode can be enabled by copying the ``BOILERPLATE`` code -block from tests/test_classh_mock.cpp. This code block provides mock -implementations of ``py::classh`` and the family of related macros -(e.g. ``PYBIND11_SMART_HOLDER_TYPE_CASTERS``). - - -Classic / Conservative / Progressive cross-module compatibility ---------------------------------------------------------------- - -Currently there are essentially three modes for building a pybind11 extension -module: - -- Classic: pybind11 stable (e.g. v2.6.2) or current master branch. - -- Conservative: pybind11 smart_holder branch. - -- Progressive: pybind11 smart_holder branch with - ``-DPYBIND11_USE_SMART_HOLDER_AS_DEFAULT``. - -In environments that mix extension modules built with different modes, -this is the compatibility matrix for ``py::class_``-wrapped types: - -.. list-table:: Compatibility matrix - :widths: auto - :header-rows: 2 - - * - - - - - - - Module 2 - - - * - - - - - Classic - - Conservative - - Progressive - * - - - **Classic** - - full - - one-and-a-half-way - - isolated - * - **Module 1** - - **Conservative** - - one-and-a-half-way - - full - - isolated - * - - - **Progressive** - - isolated - - isolated - - full - -Mixing Classic+Progressive or Conservative+Progressive is very easy to -understand: the extension modules are essentially completely isolated from -each other. This is in fact just the same as using pybind11 versions with -differing `"internals version" -`_ -in the past. While this is easy to understand, there is also no incremental -transition path between Classic and Progressive. - -The Conservative mode enables incremental transitions, but at the cost of -more complexity. Types wrapped in a Classic module are fully compatible with -a Conservative module. However, a type wrapped in a Conservative module is -compatible with a Classic module only if ``py::smart_holder`` is **not** used -(for that type). A type wrapped with ``py::smart_holder`` is incompatible with -a Classic module. This is an important pitfall to keep in mind: attempts to use -``py::smart_holder``-wrapped types in a Classic module will lead to undefined -runtime behavior, such as a SEGFAULT. This is a more general flavor of the -long-standing issue `#1138 `_, -often referred to as "holder mismatch". It is important to note that the -pybind11 smart_holder branch solves the smart-pointer interoperability issue, -but not the more general holder mismatch issue. — Unfortunately the existing -pybind11 internals do not track holder runtime type information, therefore -the holder mismatch issue cannot be solved in a fashion that would allow -an incremental transition, which is the whole point of the Conservative -mode. Please proceed with caution. (See `PR #2644 -`_ for background, which is -labeled with "abi break".) - -Another pitfall worth pointing out specifically, although it follows -from the previous: mixing base and derived classes between Classic and -Conservative modules means that neither the base nor the derived class can -use ``py::smart_holder``. - - Trampolines and std::unique_ptr ------------------------------- @@ -307,37 +101,10 @@ inherit from ``py::trampoline_self_life_support``, for example: ... }; -This is the only difference compared to Classic pybind11. A fairly +This is the only difference compared to classic pybind11. A fairly minimal but complete example is tests/test_class_sh_trampoline_unique_ptr.cpp. -Ideas for the long-term ------------------------ - -The macros are clearly an inconvenience in many situations. Highly -speculative: to avoid the need for the macros, a potential approach would -be to combine the Classic implementation (``type_caster_base``) with -the ``smart_holder_type_caster``, but this will probably be very messy and -not great as a long-term solution. The ``type_caster_base`` code is very -complex already. A more maintainable approach long-term could be to work -out and document a smart_holder-based solution for custom smart-pointers -in pybind11 version ``N``, then purge ``type_caster_base`` in version -``N+1``. #HelpAppreciated. - - -Testing of PRs against the smart_holder branch ----------------------------------------------- - -In the pybind11 GitHub Actions, PRs against the smart_holder branch are -automatically tested in both modes (Conservative, Progressive), with the -only difference that ``PYBIND11_USE_SMART_HOLDER_AS_DEFAULT`` is defined -for Progressive mode testing. - -For interactive testing, the ``PYBIND11_USE_SMART_HOLDER_AS_DEFAULT`` -define needs to be manually added to the cmake command. See -.github/workflows/ci_sh.yml for examples. - - Related links ============= @@ -353,4 +120,4 @@ Related links * Small `slide deck `_ presented in meeting with pybind11 maintainers on Feb 22, 2021. Slides 5 - and 6 show performance comparisons. + and 6 show performance comparisons. (These are outdated but probably not far off.)