From bc211ee3bd4a926377ae0b6dde84dce35c415222 Mon Sep 17 00:00:00 2001 From: Jelle Zijlstra Date: Tue, 18 Jun 2024 22:01:00 -0700 Subject: [PATCH 1/4] PEP 749: Add section on metaclasses --- peps/pep-0749.rst | 136 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 135 insertions(+), 1 deletion(-) diff --git a/peps/pep-0749.rst b/peps/pep-0749.rst index 7b76fc6d867..43e3547e570 100644 --- a/peps/pep-0749.rst +++ b/peps/pep-0749.rst @@ -390,6 +390,138 @@ More specifically: ``__dict__``. Writing to these attributes will directly update the ``__dict__``, without affecting the wrapped callable. +Annotations and metaclasses +=========================== + +Testing of the initial implementation of this PEP revealed serious problems with +the interaction between metaclasses and class annotations. + +Pre-existing bugs +----------------- + +We found several bugs in the existing behavior of ``__annotations__`` on classes +while investigating the behaviors to be specified in this PEP. Fixing these bugs +on Python 3.13 and earlier is outside the scope of this PEP, but they are noted here +to explain the corner cases that need to be dealt with. + +For context, on Python 3.10 through 3.13 the ``__annotations__`` dictionary is +placed in the class namespace if the class has any annotations. If it does not, +there is no ``__annotations__`` class dictionary key when the class is created, +but accessing ``cls.__annotations__`` invokes a descriptor defined on ``type`` +that returns an empty dictionary and stores it in the class dictionary. +On Python 3.9 and earlier, the behavior was different; see +`gh-88067 `__. + +The following code fails identically on Python 3.10 through 3.13:: + + class Meta(type): pass + + class X(metaclass=Meta): + a: str + + class Y(X): pass + + Meta.__annotations__ # important + assert Y.__annotations__ == {}, Y.__annotations__ # fails: {'a': } + +If the annotations on the metaclass ``M`` are accessed before the annotations +on ``Y``, then the annotations for the base class ``X`` are leaked to ``Y``. +However, if the metaclass's annotations are *not* accessed (i.e., the line ``Meta.__annotations__`` +above is removed), then the annotations for ``Y`` are correctly empty. + +Similarly, annotations from annotated metaclasses leak to unannotated +classes that are instances of the metaclass:: + + class Meta(type): + a: str + + class X(metaclass=Meta): + pass + + assert X.__annotations__ == {}, X.__annotations__ # fails: {'a': } + +The reason for these behaviors is that if the metaclass contains an +``__annotations__`` entry in its class dictionary, this prevents +instances of the metaclass from using the ``__annotations__`` data descriptor +on the base :py:class:`type` class. In the first case, accessing ``Meta.__annotations__`` +sets ``Meta.__dict__["__annotations__"] = {}`` as a side effect. Then, looking +up the ``__annotations__`` attribute on ``Y`` first sees the metaclass attribute, +but skips it because it is a data descriptor. Next, it looks in the class dictionaries +of the classes in its MRO, finds ``X.__annotations__``, and returns it. +In the second example, there are no annotations anywhere in the MRO, so +``type.__getattribute__`` falls back to returning the metaclass attribute. + +Metaclass behavior with PEP 649 +------------------------------- + +With :pep:`649` as originally specified, the behavior of class annotations +in relation to metaclasses becomes even more erratic, because now ``__annotations__`` +is lazily added to the class dictionary even for classes with annotations. +The new ``__annotate__`` attribute is also lazily created on classes without +annotations, which causes further misbehaviors when metaclasses are involved. + +The cause of these problems is that we set the ``__annotate__`` and ``__annotations__`` +class dictionary entries only under some circumstances, and rely on descriptors +defined on :py:class:`type` to fill them in if they are not set. This approach +breaks down in the presence of metaclasses, because entries in the metaclass's own +class dictionary can render the descriptors invisible. Two approaches for fixing +this issue are: + +* Ensure that the entry is *always* present in the class dictionary, even if it + is empty or has not yet been evaluated. This means we do not have to rely on + the descriptors defined on :py:class:`type` to fill in the field, and + therefore the metaclass's attributes will not interfere. +* Ensure that the entry is *never* present in the class dictionary, or at least + never added by logic in the language core. This means that the descriptors + on :py:class:`type` will always be used, without interference from the metaclass. + +The first approach is straightforward to implement for ``__annotate__``: it is +already always set for classes with annotations, and we can set it explicitly +to ``None`` for classes without annotations. + +``__annotations__`` causes more problems, because it must be lazily evaluated. +Therefore, we cannot simply always put an annotations dictionary in the class +dictionary. The alternative approach would be to never set ``__dict__["__annotations__"]`` +and use some other storage to store the cached annotations. This behavior +change would have to apply even to classes defined under +``from __future__ import annotations``, because otherwise there could be buggy +behavior if a class is defined without ``from __future__ import annotations`` +but its metaclass does have the future enabled. As :pep:`649` previously noted, +removing ``__annotations__`` from class dictionaries also has backwards compatibility +implications: ``cls.__dict__.get("__annotations__")`` is a common idiom to +retrieve annotations. + +Alex Waygood has suggested an approach that avoids these problems. On class +creation, ``cls.__dict__["__annotations__"]`` is set to a special descriptor. +On ``__get__``, the descriptor evaluates the annotations by calling ``__annotate__`` +and replaces itself with the result. The descriptor also behaves like a mapping, +so that code that uses ``cls.__dict__["__annotations__"]`` will still usually +work: treating the object as a mapping will evaluate the annotations and behave +as if the descriptor itself was the annotations dictionary. (Code that assumes +that ``cls.__dict__["__annotations__"]`` is specifically an instance of ``dict`` +may break, however.) + +While this approach introduces significant complexity, it fixes all known +problems with metaclasses and it preserves backwards compatibility better +than alternative ideas. + +Specification +------------- + +Accessing ``.__annotations__`` on a class object always returns that class's +annotations dictionary, which may be empty. Accessing ``.__annotate__`` on a +class object returns ``None`` if the class has no annotations, and an annotation +function that returns the class's annotations if it does have annotations. + +Classes always contain ``__annotations__`` and ``__annotate__`` keys in their +class dictionaries. The ``__annotations__`` key may map either to an annotations +dictionary or to a special descriptor that behaves like a mapping containing +the annotations. The ``__annotate__`` key maps either to an annotation function +or to ``None``. + +These guarantees do not apply if a class or metaclass explicitly sets +``__annotations__`` or ``__annotate__``. + Remove code flag for marking ``__annotate__`` functions ======================================================= @@ -695,7 +827,9 @@ Acknowledgments First of all, I thank Larry Hastings for writing :pep:`649`. This PEP modifies some of his initial decisions, but the overall design is still his. -I thank Carl Meyer and Alex Waygood for feedback on early drafts of this PEP. +I thank Carl Meyer and Alex Waygood for feedback on early drafts of this PEP. Alex Waygood, +Alyssa Coghlan, and David Ellis provided insightful feedback and suggestions on the +interaction between metaclasses and ``__annotations__``. Appendix ======== From 629a50b67eb8af559bc98168e96bee9fae980926 Mon Sep 17 00:00:00 2001 From: Jelle Zijlstra Date: Tue, 18 Jun 2024 22:42:47 -0700 Subject: [PATCH 2/4] Update peps/pep-0749.rst Co-authored-by: Carl Meyer --- peps/pep-0749.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0749.rst b/peps/pep-0749.rst index 43e3547e570..88ff76595d5 100644 --- a/peps/pep-0749.rst +++ b/peps/pep-0749.rst @@ -424,7 +424,7 @@ The following code fails identically on Python 3.10 through 3.13:: Meta.__annotations__ # important assert Y.__annotations__ == {}, Y.__annotations__ # fails: {'a': } -If the annotations on the metaclass ``M`` are accessed before the annotations +If the annotations on the metaclass ``Meta`` are accessed before the annotations on ``Y``, then the annotations for the base class ``X`` are leaked to ``Y``. However, if the metaclass's annotations are *not* accessed (i.e., the line ``Meta.__annotations__`` above is removed), then the annotations for ``Y`` are correctly empty. From bdd61982ed26636540e9e80f36e08990597db13e Mon Sep 17 00:00:00 2001 From: Jelle Zijlstra Date: Wed, 19 Jun 2024 06:58:05 -0700 Subject: [PATCH 3/4] code review and expansions --- peps/pep-0749.rst | 40 ++++++++++++++++++++++++++++------------ 1 file changed, 28 insertions(+), 12 deletions(-) diff --git a/peps/pep-0749.rst b/peps/pep-0749.rst index 88ff76595d5..95ed1fd02df 100644 --- a/peps/pep-0749.rst +++ b/peps/pep-0749.rst @@ -409,6 +409,8 @@ placed in the class namespace if the class has any annotations. If it does not, there is no ``__annotations__`` class dictionary key when the class is created, but accessing ``cls.__annotations__`` invokes a descriptor defined on ``type`` that returns an empty dictionary and stores it in the class dictionary. +:py:ref:`Static types ` are an exception: they never have +annotations, and accessing ``.__annotations__`` raises :py:exc:`AttributeError`. On Python 3.9 and earlier, the behavior was different; see `gh-88067 `__. @@ -447,16 +449,17 @@ on the base :py:class:`type` class. In the first case, accessing ``Meta.__annota sets ``Meta.__dict__["__annotations__"] = {}`` as a side effect. Then, looking up the ``__annotations__`` attribute on ``Y`` first sees the metaclass attribute, but skips it because it is a data descriptor. Next, it looks in the class dictionaries -of the classes in its MRO, finds ``X.__annotations__``, and returns it. -In the second example, there are no annotations anywhere in the MRO, so -``type.__getattribute__`` falls back to returning the metaclass attribute. +of the classes in its method resolution order (MRO), finds ``X.__annotations__``, +and returns it. In the second example, there are no annotations +anywhere in the MRO, so ``type.__getattribute__`` falls back to +returning the metaclass attribute. Metaclass behavior with PEP 649 ------------------------------- With :pep:`649` as originally specified, the behavior of class annotations in relation to metaclasses becomes even more erratic, because now ``__annotations__`` -is lazily added to the class dictionary even for classes with annotations. +is only lazily added to the class dictionary even for classes with annotations. The new ``__annotate__`` attribute is also lazily created on classes without annotations, which causes further misbehaviors when metaclasses are involved. @@ -464,8 +467,8 @@ The cause of these problems is that we set the ``__annotate__`` and ``__annotati class dictionary entries only under some circumstances, and rely on descriptors defined on :py:class:`type` to fill them in if they are not set. This approach breaks down in the presence of metaclasses, because entries in the metaclass's own -class dictionary can render the descriptors invisible. Two approaches for fixing -this issue are: +class dictionary can render the descriptors invisible. Approaches for fixing +this issue include: * Ensure that the entry is *always* present in the class dictionary, even if it is empty or has not yet been evaluated. This means we do not have to rely on @@ -474,6 +477,14 @@ this issue are: * Ensure that the entry is *never* present in the class dictionary, or at least never added by logic in the language core. This means that the descriptors on :py:class:`type` will always be used, without interference from the metaclass. +* Bypass normal attribute lookup when accessing these attributes, instead + invoking the base class descriptor (from e.g., ``type.__dict__["__annotations__"]``) + directly. + +The third approach could be used in abstractions like ``annotationlib.get_annotations``, +but it would mean that accessing ``cls.__annotations__`` or ``cls.__annotate__`` +would be unsafe in some cases. This would create an unacceptable inconsistency +in the language. The first approach is straightforward to implement for ``__annotate__``: it is already always set for classes with annotations, and we can set it explicitly @@ -491,10 +502,12 @@ removing ``__annotations__`` from class dictionaries also has backwards compatib implications: ``cls.__dict__.get("__annotations__")`` is a common idiom to retrieve annotations. -Alex Waygood has suggested an approach that avoids these problems. On class -creation, ``cls.__dict__["__annotations__"]`` is set to a special descriptor. +Alex Waygood has suggested an approach that avoids these problems. When a +heap type (such as a class created through the ``class`` statement) is created, +``cls.__dict__["__annotations__"]`` is set to a special descriptor. On ``__get__``, the descriptor evaluates the annotations by calling ``__annotate__`` -and replaces itself with the result. The descriptor also behaves like a mapping, +and returning the result. The annotations dictionary is cached within the +descriptor instance. The descriptor also behaves like a mapping, so that code that uses ``cls.__dict__["__annotations__"]`` will still usually work: treating the object as a mapping will evaluate the annotations and behave as if the descriptor itself was the annotations dictionary. (Code that assumes @@ -508,12 +521,15 @@ than alternative ideas. Specification ------------- -Accessing ``.__annotations__`` on a class object always returns that class's -annotations dictionary, which may be empty. Accessing ``.__annotate__`` on a +Accessing ``.__annotations__`` on a class object that is not a +:py:ref:`static type ` always returns that class's +annotations dictionary, which may be empty. On static types, +the attribute does not exist. Accessing ``.__annotate__`` on a class object returns ``None`` if the class has no annotations, and an annotation function that returns the class's annotations if it does have annotations. -Classes always contain ``__annotations__`` and ``__annotate__`` keys in their +Classes that are not :py:ref:`static types ` always contain +``__annotations__`` and ``__annotate__`` keys in their class dictionaries. The ``__annotations__`` key may map either to an annotations dictionary or to a special descriptor that behaves like a mapping containing the annotations. The ``__annotate__`` key maps either to an annotation function From ef781a5f74f213448f03e6e1d38313d8c3db9e45 Mon Sep 17 00:00:00 2001 From: Jelle Zijlstra Date: Sat, 20 Jul 2024 20:57:06 -0700 Subject: [PATCH 4/4] Third approach --- peps/pep-0749.rst | 125 +++++++++++++++++++++++++--------------------- 1 file changed, 69 insertions(+), 56 deletions(-) diff --git a/peps/pep-0749.rst b/peps/pep-0749.rst index 95ed1fd02df..6503ef832f1 100644 --- a/peps/pep-0749.rst +++ b/peps/pep-0749.rst @@ -187,6 +187,10 @@ The module will contain the following functionality: module, or class. This will replace :py:func:`inspect.get_annotations`. The latter will delegate to the new function. It may eventually be deprecated, but to minimize disruption, we do not propose an immediate deprecation. +* ``get_annotate_function()``: A function that returns the ``__annotate__`` function + of an object, if it has one, or ``None`` if it does not. This is usually equivalent + to accessing the ``.__annotate__`` attribute, except in the presence of metaclasses + (see :ref:`below `). * ``Format``: an enum that contains the possible formats of annotations. This will replace the ``VALUE``, ``FORWARDREF``, and ``SOURCE`` formats in :pep:`649`. PEP 649 proposed to make these values global members of the :py:mod:`inspect` @@ -390,6 +394,8 @@ More specifically: ``__dict__``. Writing to these attributes will directly update the ``__dict__``, without affecting the wrapped callable. +.. _pep749-metaclasses: + Annotations and metaclasses =========================== @@ -457,52 +463,61 @@ returning the metaclass attribute. Metaclass behavior with PEP 649 ------------------------------- -With :pep:`649` as originally specified, the behavior of class annotations -in relation to metaclasses becomes even more erratic, because now ``__annotations__`` -is only lazily added to the class dictionary even for classes with annotations. -The new ``__annotate__`` attribute is also lazily created on classes without -annotations, which causes further misbehaviors when metaclasses are involved. +With :pep:`649`, the behavior of accessing the ``.__annotations__`` attribute +on classes when metaclasses are involved becomes even more erratic, because now +``__annotations__`` is only lazily added to the class dictionary even for classes +with annotations. The new ``__annotate__`` attribute is also lazily created +on classes without annotations, which causes further misbehaviors when +metaclasses are involved. The cause of these problems is that we set the ``__annotate__`` and ``__annotations__`` class dictionary entries only under some circumstances, and rely on descriptors -defined on :py:class:`type` to fill them in if they are not set. This approach -breaks down in the presence of metaclasses, because entries in the metaclass's own -class dictionary can render the descriptors invisible. Approaches for fixing -this issue include: +defined on :py:class:`type` to fill them in if they are not set. When normal +attribute lookup is used, this approach breaks down in the presence of +metaclasses, because entries in the metaclass's own class dictionary can render +the descriptors invisible. + +While we considered several approaches that would allow ``cls.__annotations__`` +and ``cls.__annotate__`` to work reliably when ``cls`` is a type with a custom +metaclass, any such approach would expose significant complexity to advanced users. +Instead, we recommend a simpler approach that confines the complexity to the +``annotationlib`` module: in ``annotationlib.get_annotations``, we bypass normal +attribute lookup by using the ``type.__annotations__`` descriptor directly. + +Specification +------------- + +Users should always use ``annotationlib.get_annotations`` to access the +annotations of a class object, and ``annotationlib.get_annotate_function`` +to access the ``__annotate__`` function. These functions will return only +the class's own annotations, even when metaclasses are involved. + +The behavior of accessing the ``__annotations__`` and ``__annotate__`` +attributes on classes with a metaclass other than ``builtins.type`` is +unspecified. The documentation should warn against direct use of these +attributes and recommend using the ``annotationlib`` module instead. + +Similarly, the presence of ``__annotations__`` and ``__annotate__`` keys +in the class dictionary is an implementation detail and should not be relied +upon. + +Rejected alternatives +--------------------- + +We considered two broad approaches for dealing with the behavior +of the ``__annotations__`` and ``__annotate__`` entries in classes: * Ensure that the entry is *always* present in the class dictionary, even if it is empty or has not yet been evaluated. This means we do not have to rely on the descriptors defined on :py:class:`type` to fill in the field, and - therefore the metaclass's attributes will not interfere. + therefore the metaclass's attributes will not interfere. (Prototype + in `gh-120719 `__.) * Ensure that the entry is *never* present in the class dictionary, or at least never added by logic in the language core. This means that the descriptors on :py:class:`type` will always be used, without interference from the metaclass. -* Bypass normal attribute lookup when accessing these attributes, instead - invoking the base class descriptor (from e.g., ``type.__dict__["__annotations__"]``) - directly. - -The third approach could be used in abstractions like ``annotationlib.get_annotations``, -but it would mean that accessing ``cls.__annotations__`` or ``cls.__annotate__`` -would be unsafe in some cases. This would create an unacceptable inconsistency -in the language. - -The first approach is straightforward to implement for ``__annotate__``: it is -already always set for classes with annotations, and we can set it explicitly -to ``None`` for classes without annotations. - -``__annotations__`` causes more problems, because it must be lazily evaluated. -Therefore, we cannot simply always put an annotations dictionary in the class -dictionary. The alternative approach would be to never set ``__dict__["__annotations__"]`` -and use some other storage to store the cached annotations. This behavior -change would have to apply even to classes defined under -``from __future__ import annotations``, because otherwise there could be buggy -behavior if a class is defined without ``from __future__ import annotations`` -but its metaclass does have the future enabled. As :pep:`649` previously noted, -removing ``__annotations__`` from class dictionaries also has backwards compatibility -implications: ``cls.__dict__.get("__annotations__")`` is a common idiom to -retrieve annotations. + (Prototype in `gh-120816 `__.) -Alex Waygood has suggested an approach that avoids these problems. When a +Alex Waygood suggested an implementation using the first approach. When a heap type (such as a class created through the ``class`` statement) is created, ``cls.__dict__["__annotations__"]`` is set to a special descriptor. On ``__get__``, the descriptor evaluates the annotations by calling ``__annotate__`` @@ -514,29 +529,27 @@ as if the descriptor itself was the annotations dictionary. (Code that assumes that ``cls.__dict__["__annotations__"]`` is specifically an instance of ``dict`` may break, however.) -While this approach introduces significant complexity, it fixes all known -problems with metaclasses and it preserves backwards compatibility better -than alternative ideas. +This approach is also straightforward to implement for ``__annotate__``: this +attribute is already always set for classes with annotations, and we can set +it explicitly to ``None`` for classes without annotations. -Specification -------------- +While this approach would fix the known edge cases with metaclasses, it +introduces significant complexity to all classes, including a new built-in type +(for the annotations descriptor) with unusual behavior. + +The alternative approach would be to never set ``__dict__["__annotations__"]`` +and use some other storage to store the cached annotations. This behavior +change would have to apply even to classes defined under +``from __future__ import annotations``, because otherwise there could be buggy +behavior if a class is defined without ``from __future__ import annotations`` +but its metaclass does have the future enabled. As :pep:`649` previously noted, +removing ``__annotations__`` from class dictionaries also has backwards compatibility +implications: ``cls.__dict__.get("__annotations__")`` is a common idiom to +retrieve annotations. -Accessing ``.__annotations__`` on a class object that is not a -:py:ref:`static type ` always returns that class's -annotations dictionary, which may be empty. On static types, -the attribute does not exist. Accessing ``.__annotate__`` on a -class object returns ``None`` if the class has no annotations, and an annotation -function that returns the class's annotations if it does have annotations. - -Classes that are not :py:ref:`static types ` always contain -``__annotations__`` and ``__annotate__`` keys in their -class dictionaries. The ``__annotations__`` key may map either to an annotations -dictionary or to a special descriptor that behaves like a mapping containing -the annotations. The ``__annotate__`` key maps either to an annotation function -or to ``None``. - -These guarantees do not apply if a class or metaclass explicitly sets -``__annotations__`` or ``__annotate__``. +This approach would also mean that accessing ``.__annotations__`` on an instance +of an annotated class no longer works. While this behavior is not documented, +it is a long-standing feature of Python and is relied upon by some users. Remove code flag for marking ``__annotate__`` functions =======================================================