-
Notifications
You must be signed in to change notification settings - Fork 6
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Make wrapped C++ functions pickleable (#30099)
* Add `test_*_repr()` to test behavior with different Python versions.
* Adjust expected repr for PyPy
* Adjust another expected repr for PyPy
* Try again: undo mistaken adjustment for PyPy
* Give up on test_pytypes test_capsule_with_name_repr (not sufficiently important); PyPy still generates 2 different kinds of errors: test_print failure on macOS with Python 3.8; Python 3.9, 3.10 have no leading `<`
* `_wrapped_simple_callable` proof of concept
* Add `module_::def_as_native()`
* Resolve PyPy `TypeError: cannot create weak reference to builtin_function_or_method object`
* Replace `PyCapsule` with `function_record_PyObject`.
* function_record_PyTypeObject: Replace C++20 designated initializers with full list of values.
* Introduce `PYBIND11_DETAIL_FUNCTION_RECORD_ABI_ID` and use along with `PYBIND11_PLATFORM_ABI_ID_V4` to version `function_record_PyTypeObject` `tp_name`
* Move `std::once_flag` out of `inline` function (in hopes that that fixes flaky behavior of test_gil_scoped.py). IncludeCleaner fixes.
* `tp_vectorcall` was introduced only with Python 3.8
* clang-tidy auto-fixes
* Disable `-Wmissing-field-initializers`. Guard `PyType_Ready(&function_record_PyTypeObject)` also with a simple `static bool first_call`
* Give up on the `std::call_once` idea, for Python 3.6 compatibility (it works with all other Python versions). Instead call `function_record_PyTypeObject_PyType_Ready()` from `get_internals()`.
* Add `__reduce_ex__` to `function_record_PyTypeObject`. Add `_pybind11_detail_function_record_import_helper` (proof of concept).
* Move `function_record_PyTypeObject_PyType_Ready()` call in `get_internals()` so that it is always called when `get_internals()` is called the first time.
* gcc 4.8.5 and 7.5.0 reject `PYBIND11_WARNING_DISABLE_GCC("-Wmissing-field-initializers")`
* `function_record_PyTypeObject_PyType_Ready()`, `get_pybind11_detail_function_record_pickle_helper()` call-once initializations triggered from `cpp_function::initialize_generic()`
* gcc 4.8.5 and 7.5.0 reject `PYBIND11_WARNING_DISABLE_GCC("-Wcast-function-type")`
* Python 3.6, 3.7: Skip `get_pybind11_detail_function_record_pickle_helper()` call-once initialization triggered from `cpp_function::initialize_generic()`
* New version of `_function_record_pickle_helper`, using `collections.namedtuple`
* Explicit `str(tup_obj[1])` to fix 🐍 3 • centos:7 • x64 segfault
* Factor out detail/function_record_pyobject.h
* Use PYBIND11_NAMESPACE_BEGIN/END for function_record_PyTypeObject_methods
* Factor out function_record_PyTypeObject_methods::tp_name_impl, mainly to stop clang-format from breaking up a string literal.
* Simplify implementation of UNEXPECTED CALL functions.
* Factor out `detail::get_scope_module()`
* IncludeCleaner fixes (Google toolchain).
* Comment out unreachable code (to resolve MSVC Werrors).
* Use built-in `eval()` instead of `function_record_pickle_helper()`
Much simpler!
(Note that the `function_record_pickle_helper()` code is NOT removed in this commit.)
This approach was discovered in an attempt to solve the problem that
stubgen picks up `_function_record_pickle_helper_v1`.
For example (tensorflow_text/core/pybinds/tflite_registrar.pyi):
```diff
+from typing import Any
+
+def _function_record_pickle_helper_v1(*args, **kwargs) -> Any: ...
```
* Remove `function_record_pickle_helper()`
* Mark `internals::function_record_capsule_name` as OBSOLETE.
* Add comment pointing to #30099
* Archive experimental code from video meet with @rainwoodman 2024-02-15
* Add a pickle roundtrip test starting with `m.simple_callable.__self__` and a long comment to explain the unusual behavior.
* PyPy does not have `m.simple_callable.__self__`
* Change "UNUSUAL" comment as suggested by @rainwoodman (only very slightly differently as suggested).- Loading branch information
Ralf W. Grosse-Kunstleve
committed
Feb 20, 2024
1 parent
f468b2c
commit de89591
Showing
10 changed files
with
290 additions
and
64 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,205 @@ | ||
| // Copyright (c) 2024 The Pybind Development Team. | ||
| // All rights reserved. Use of this source code is governed by a | ||
| // BSD-style license that can be found in the LICENSE file. | ||
|
|
||
| // For background see the description of PR google/pywrapcc#30099. | ||
|
|
||
| #pragma once | ||
|
|
||
| #include "../attr.h" | ||
| #include "../pytypes.h" | ||
| #include "common.h" | ||
|
|
||
| #include <cstring> | ||
|
|
||
| PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) | ||
| PYBIND11_NAMESPACE_BEGIN(detail) | ||
|
|
||
| struct function_record_PyObject { | ||
| PyObject_HEAD | ||
| function_record *cpp_func_rec; | ||
| }; | ||
|
|
||
| PYBIND11_NAMESPACE_BEGIN(function_record_PyTypeObject_methods) | ||
|
|
||
| PyObject *tp_new_impl(PyTypeObject *type, PyObject *args, PyObject *kwds); | ||
| PyObject *tp_alloc_impl(PyTypeObject *type, Py_ssize_t nitems); | ||
| int tp_init_impl(PyObject *self, PyObject *args, PyObject *kwds); | ||
| void tp_dealloc_impl(PyObject *self); | ||
| void tp_free_impl(void *self); | ||
|
|
||
| static PyObject *reduce_ex_impl(PyObject *self, PyObject *, PyObject *); | ||
|
|
||
| PYBIND11_WARNING_PUSH | ||
| #if defined(__GNUC__) && __GNUC__ >= 8 | ||
| PYBIND11_WARNING_DISABLE_GCC("-Wcast-function-type") | ||
| #endif | ||
| static PyMethodDef tp_methods_impl[] | ||
| = {{"__reduce_ex__", (PyCFunction) reduce_ex_impl, METH_VARARGS | METH_KEYWORDS, nullptr}, | ||
| {nullptr, nullptr, 0, nullptr}}; | ||
| PYBIND11_WARNING_POP | ||
|
|
||
| // Note that this name is versioned. | ||
| constexpr char tp_name_impl[] | ||
| = "pybind11_detail_function_record_" PYBIND11_DETAIL_FUNCTION_RECORD_ABI_ID | ||
| "_" PYBIND11_PLATFORM_ABI_ID_V4; | ||
|
|
||
| PYBIND11_NAMESPACE_END(function_record_PyTypeObject_methods) | ||
|
|
||
| // Designated initializers are a C++20 feature: | ||
| // https://en.cppreference.com/w/cpp/language/aggregate_initialization#Designated_initializers | ||
| // MSVC rejects them unless /std:c++20 is used (error code C7555). | ||
| PYBIND11_WARNING_PUSH | ||
| PYBIND11_WARNING_DISABLE_CLANG("-Wmissing-field-initializers") | ||
| #if defined(__GNUC__) && __GNUC__ >= 8 | ||
| PYBIND11_WARNING_DISABLE_GCC("-Wmissing-field-initializers") | ||
| #endif | ||
| static PyTypeObject function_record_PyTypeObject = { | ||
| PyVarObject_HEAD_INIT(nullptr, 0) | ||
| /* const char *tp_name */ function_record_PyTypeObject_methods::tp_name_impl, | ||
| /* Py_ssize_t tp_basicsize */ sizeof(function_record_PyObject), | ||
| /* Py_ssize_t tp_itemsize */ 0, | ||
| /* destructor tp_dealloc */ function_record_PyTypeObject_methods::tp_dealloc_impl, | ||
| /* Py_ssize_t tp_vectorcall_offset */ 0, | ||
| /* getattrfunc tp_getattr */ nullptr, | ||
| /* setattrfunc tp_setattr */ nullptr, | ||
| /* PyAsyncMethods *tp_as_async */ nullptr, | ||
| /* reprfunc tp_repr */ nullptr, | ||
| /* PyNumberMethods *tp_as_number */ nullptr, | ||
| /* PySequenceMethods *tp_as_sequence */ nullptr, | ||
| /* PyMappingMethods *tp_as_mapping */ nullptr, | ||
| /* hashfunc tp_hash */ nullptr, | ||
| /* ternaryfunc tp_call */ nullptr, | ||
| /* reprfunc tp_str */ nullptr, | ||
| /* getattrofunc tp_getattro */ nullptr, | ||
| /* setattrofunc tp_setattro */ nullptr, | ||
| /* PyBufferProcs *tp_as_buffer */ nullptr, | ||
| /* unsigned long tp_flags */ Py_TPFLAGS_DEFAULT, | ||
| /* const char *tp_doc */ nullptr, | ||
| /* traverseproc tp_traverse */ nullptr, | ||
| /* inquiry tp_clear */ nullptr, | ||
| /* richcmpfunc tp_richcompare */ nullptr, | ||
| /* Py_ssize_t tp_weaklistoffset */ 0, | ||
| /* getiterfunc tp_iter */ nullptr, | ||
| /* iternextfunc tp_iternext */ nullptr, | ||
| /* struct PyMethodDef *tp_methods */ function_record_PyTypeObject_methods::tp_methods_impl, | ||
| /* struct PyMemberDef *tp_members */ nullptr, | ||
| /* struct PyGetSetDef *tp_getset */ nullptr, | ||
| /* struct _typeobject *tp_base */ nullptr, | ||
| /* PyObject *tp_dict */ nullptr, | ||
| /* descrgetfunc tp_descr_get */ nullptr, | ||
| /* descrsetfunc tp_descr_set */ nullptr, | ||
| /* Py_ssize_t tp_dictoffset */ 0, | ||
| /* initproc tp_init */ function_record_PyTypeObject_methods::tp_init_impl, | ||
| /* allocfunc tp_alloc */ function_record_PyTypeObject_methods::tp_alloc_impl, | ||
| /* newfunc tp_new */ function_record_PyTypeObject_methods::tp_new_impl, | ||
| /* freefunc tp_free */ function_record_PyTypeObject_methods::tp_free_impl, | ||
| /* inquiry tp_is_gc */ nullptr, | ||
| /* PyObject *tp_bases */ nullptr, | ||
| /* PyObject *tp_mro */ nullptr, | ||
| /* PyObject *tp_cache */ nullptr, | ||
| /* PyObject *tp_subclasses */ nullptr, | ||
| /* PyObject *tp_weaklist */ nullptr, | ||
| /* destructor tp_del */ nullptr, | ||
| /* unsigned int tp_version_tag */ 0, | ||
| /* destructor tp_finalize */ nullptr, | ||
| #if PY_VERSION_HEX >= 0x03080000 | ||
| /* vectorcallfunc tp_vectorcall */ nullptr, | ||
| #endif | ||
| }; | ||
| PYBIND11_WARNING_POP | ||
|
|
||
| static bool function_record_PyTypeObject_PyType_Ready_first_call = true; | ||
|
|
||
| inline void function_record_PyTypeObject_PyType_Ready() { | ||
| if (function_record_PyTypeObject_PyType_Ready_first_call) { | ||
| if (PyType_Ready(&function_record_PyTypeObject) < 0) { | ||
| throw error_already_set(); | ||
| } | ||
| function_record_PyTypeObject_PyType_Ready_first_call = false; | ||
| } | ||
| } | ||
|
|
||
| inline bool is_function_record_PyObject(PyObject *obj) { | ||
| if (PyType_Check(obj) != 0) { | ||
| return false; | ||
| } | ||
| PyTypeObject *obj_type = Py_TYPE(obj); | ||
| // Fast path (pointer comparison). | ||
| if (obj_type == &function_record_PyTypeObject) { | ||
| return true; | ||
| } | ||
| // This works across extension modules. Note that tp_name is versioned. | ||
| if (strcmp(obj_type->tp_name, function_record_PyTypeObject.tp_name) == 0) { | ||
| return true; | ||
| } | ||
| return false; | ||
| } | ||
|
|
||
| inline function_record *function_record_ptr_from_PyObject(PyObject *obj) { | ||
| if (is_function_record_PyObject(obj)) { | ||
| return ((detail::function_record_PyObject *) obj)->cpp_func_rec; | ||
| } | ||
| return nullptr; | ||
| } | ||
|
|
||
| inline object function_record_PyObject_New() { | ||
| auto *py_func_rec = PyObject_New(function_record_PyObject, &function_record_PyTypeObject); | ||
| if (py_func_rec == nullptr) { | ||
| throw error_already_set(); | ||
| } | ||
| py_func_rec->cpp_func_rec = nullptr; // For clarity/purity. Redundant in practice. | ||
| return reinterpret_steal<object>((PyObject *) py_func_rec); | ||
| } | ||
|
|
||
| PYBIND11_NAMESPACE_BEGIN(function_record_PyTypeObject_methods) | ||
|
|
||
| // Guard against accidents & oversights, in particular when porting to future Python versions. | ||
| inline PyObject *tp_new_impl(PyTypeObject *, PyObject *, PyObject *) { | ||
| pybind11_fail("UNEXPECTED CALL OF function_record_PyTypeObject_methods::tp_new_impl"); | ||
| // return nullptr; // Unreachable. | ||
| } | ||
|
|
||
| inline PyObject *tp_alloc_impl(PyTypeObject *, Py_ssize_t) { | ||
| pybind11_fail("UNEXPECTED CALL OF function_record_PyTypeObject_methods::tp_alloc_impl"); | ||
| // return nullptr; // Unreachable. | ||
| } | ||
|
|
||
| inline int tp_init_impl(PyObject *, PyObject *, PyObject *) { | ||
| pybind11_fail("UNEXPECTED CALL OF function_record_PyTypeObject_methods::tp_init_impl"); | ||
| // return -1; // Unreachable. | ||
| } | ||
|
|
||
| // The implementation needs the definition of `class cpp_function`. | ||
| void tp_dealloc_impl(PyObject *self); | ||
|
|
||
| inline void tp_free_impl(void *) { | ||
| pybind11_fail("UNEXPECTED CALL OF function_record_PyTypeObject_methods::tp_free_impl"); | ||
| } | ||
|
|
||
| inline PyObject *reduce_ex_impl(PyObject *self, PyObject *, PyObject *) { | ||
| // Deliberately ignoring the arguments for simplicity (expected is `protocol: int`). | ||
| const function_record *rec = function_record_ptr_from_PyObject(self); | ||
| if (rec == nullptr) { | ||
| pybind11_fail( | ||
| "FATAL: function_record_PyTypeObject reduce_ex_impl(): cannot obtain cpp_func_rec."); | ||
| } | ||
| if (rec->name != nullptr && rec->name[0] != '\0' && rec->scope | ||
| && PyModule_Check(rec->scope.ptr()) != 0) { | ||
| object scope_module = get_scope_module(rec->scope); | ||
| if (scope_module) { | ||
| return make_tuple(reinterpret_borrow<object>(PyEval_GetBuiltins())["eval"], | ||
| make_tuple(str("__import__('importlib').import_module('") | ||
| + scope_module + str("')"))) | ||
| .release() | ||
| .ptr(); | ||
| } | ||
| } | ||
| set_error(PyExc_RuntimeError, repr(self) + str(" is not pickleable.")); | ||
| return nullptr; | ||
| } | ||
|
|
||
| PYBIND11_NAMESPACE_END(function_record_PyTypeObject_methods) | ||
|
|
||
| PYBIND11_NAMESPACE_END(detail) | ||
| PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.