Should we vendor typing_extensions.NotRequired?

[dimaqq]

The code is really simple:

https://github.com/python/typing_extensions/blob/70cec91bec65155dc339d631ede2a933582558df/src/typing_extensions.py#L2220-L2256

We're using:

• Required in ops/charm.py
• NotRequired in ops/pebble.py
• NotRequired in test/fake_pebble.py

[benhoyt]

What's the motivation for this? Are they being removed from typing_extensions, or is there some other reason?

[dimaqq]

The motivation is to avoide the if TYPE_CHECKING: block.

The Required/NotRequired are used in TypedDict definitions, which means that TypedDict's can't be used without quotes. Granted, those typed dicts are internal.

I think it may partially help with the following comment for public typed dicts:
# In Python 3.11+ 'services' and 'labels' should be NotRequired, and total=True.

The latter could be an improvement for our library users, maybe?

Possibly related: #1336

[benhoyt]

We're uncertain whether this will actually work once they're vendored with Pyright. Investigate on a rainy day next cycle.

[james-garner-canonical]

I suspect that this can't be vendored, given the following quote from typing-extensions:

"typing_extensions is treated specially by static type checkers such as mypy and pyright. Objects defined in typing_extensions are treated the same way as equivalent forms in typing."

I haven't used typing_extensions much, but do we have to use its constructs with if TYPE_CHECKING? I see that we have used it this way, for example in ops/pebble.py:

if TYPE_CHECKING:
    from typing_extensions import NotRequired

As far as I can tell, typing_extensions currently supports Python versions 3.8 and higher, so do we really need if TYPE_CHECKING?

Maybe the outcome (with a little bit of further rainy day investigation) will be closing this and making a new issue: let's use Required/NotRequired from typing_extensions to improve our TypeDicts without if TYPE_CHECKING.

[tonyandrewmeyer]

I haven't used typing_extensions much, but do we have to use its constructs with if TYPE_CHECKING?

The advantage of this is that typing_extensions is not a runtime dependency, only a test time one.

[james-garner-canonical]

That makes sense. With from __future__ import annotations we can put the typing_extensions imports behind if TYPE_CHECKING but use its constructs in annotations freely outside if TYPE_CHECKING. Without it, we can still put just the imports inside if TYPE_CHECKING and have the definitions just use string annotations.

Since we're going to go with from __future__ import annotations in future, I think we can close this in favour of (#1336), but I'll hold off on doing so in case anyone wants to investigate vendoring further.

[dimaqq]

typing_extensions is treated specially by static type checkers such as mypy and pyright. Objects defined in typing_extensions are treated the same way as equivalent forms in typing.

perhaps that means we can't naively vendor some bits from typing_extensions. We could still play with a vendored module that happens to be called "typing_extensions" but I doubt that's worth the hassle, as we'd have to make a funky import to ensure we don't break charms that imports real type_extensions and thus has real module in their virtual env(s).

my take: either real typing_extensions or nothing.

[benhoyt]

The types using NotRequired are _Private in ops/pebble.py. Even if we wanted to make some of them public, we could probably use strings around our type annotations. But we don't think anything is needed here now.