fix(types): update render for buffer sequence and handle

[sizmailov]

Description

Change docstring render for py::buffer, py::sequence, py::handle and py::object

py::buffer -> Buffer
py::sequence -> Sequence
py::handle ->  Any

Suggested changelog entry:

Change docstring render for ``py::buffer``, ``py::sequence`` and ``py::handle`` (to ``Buffer``, ``Sequence``, ``Any``).

Follow up to #4829

[rwgk]

Thanks, looks great!
I ran this by a typing expert on my team and they confirmed that Sequence and Buffer are good choices.

[wjakob]

AFAIK, the main difference between Any and object is that the former is unconstrained, and the latter is maximally constrained.

For example, when a function returns Any, the caller is allowed to call any method on it, and MyPy will accept this. When it returns object, the caller can only assume that this is an object but no more. In other words, it can invoke methods provided by all object instances (e.g., .__hash__()) but no more specific ones.

So Any basically turns off the type checker. To me it's not a clear-case situation that pybind11 should always use Any instead of Object (for py::handle and py::object), which is what this PR proposes. It seems to me like a developer/project-specific decision that would need to be decided on a case-by-case basis.

The Sequence and Buffer changes seem less controversial.

[rwgk]

AFAIK, the main difference between Any and object is that the former is unconstrained, and the latter is maximally constrained.

Correct.

Did you see my other comment?

#4829 (comment)

The lesson learned was:

object didn't serve any practical purpose; it's just a "you cannot do anything with this" road block.

Any turns out to be much more useful in practice, although yes, it's maximally permissive.

[sizmailov]

The usage and behavior of py::object resemble Python's Any more than object. Thus, Any is a better annotation for py::object.

I find it hard to envision a use case where I'd prefer an object annotation. In my opinion, bare object instances are primarily useful as sentinels.

[rwgk]

controversial.

What we have here is an ambiguity that needs to be resolved by a judgement call.

The mapping from py::object to object is initially intuitive, but is it actually useful?

It seems to me like a developer/project-specific decision that would need to be decided on a case-by-case basis.

Yes, that would be ideal. But we do not have that technology. Until someone figures out how to deliver that ideal solution, we have to choose between mapping to object or Any.

@sizmailov and I are approaching this empirically, from a practical viewpoint:

• From my end, as I wrote under fix: Render py::function as Callable #4829, PyCLIF initially also had the PyObject* to object mapping, but it really wasn't useful in practice (and a scale; ~7k extension modules at that time), so we changed it, with no argument made ever to change it back.

• Sergei: fix(types): update render for buffer sequence and handle  #4831 (comment)

[wjakob]

I agree it's convenient for the person writing bindings because MyPy checks will pass. But this also sounds extremely dangerous. If this was MyPy's decision, I doubt that they would be happy if we use Any as a default here.

Another route would be to make docstring generation more flexible. To contrast, let me show what's implemented in nanobind:

1. Functions can take nb::handle_t as input. That means: do type checking for type T and use that type annotation in the binding docstrings. But when accessed, this is actually a Python handle (i.e. a wrapper type). This mainly makes sense for arguments rather than return values.

2. Functions can take/return nb::typed, which provides complete control over the type annotation.

3. Functions can be bound with nb::raw_doc, which replaces the function docstring with the given text. This includes the overload listing and can, again, be used to take complete control over type annotations.

To me, these seem better solutions than taking pybind11's default annotation and switching it to a dangerous default. The developer should have to actually think about the types and specify something that makes sense. If they aren't happy with the strict results, then Any is still available via options 2 and 3.

[rwgk]

I wasn't aware of the nanobind features, that is nice!

I guess keeping object could act as a forcing/motivating function for someone to port the nanobind features into pybind11 sooner rather than later.

Waiting for @sizmailov to comment.

[sizmailov]

I suggest removing the controversial py::object -> Any renaming from this PR, keeping the other three changes.

Let's create a separate issue for py::object - > Any renaming. There we could summon people from MyPy / Pybind community.

For me, the issue is not crucial since I'm already post-processing docstrings in pybind11-stubgen to generate stubs and documentation.

[sizmailov]

I've reverted 7861dcf to clear the path to the rest of the changes in the PR.

Note, that the opt-in to py::object -> Any rename is straightforward (unlike opt-out) if you want to use it in your own bindings:

// Include this before any pybind code execution in all translation units
// to render `pybind11::object` as `Any` in docstrings
namespace pybind11::detail {
template <>
struct handle_type_name<object> {
    static constexpr auto name = const_name("Any");
};
}

[rwgk]

@wjakob Is this good to merge now?