diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 97c98efb357..bde7d19dd92 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -10,7 +10,6 @@ Python-Version: 3.13 Post-History: 20-Aug-2021 Resolution: https://discuss.python.org/t/pep-667-consistent-views-of-namespaces/46631/25 - Abstract ======== @@ -31,25 +30,47 @@ consistent regardless of threading or coroutines. The ``locals()`` function will act the same as it does now for class and modules scopes. For function scopes it will return an instantaneous -snapshot of the underlying ``frame.f_locals``. +snapshot of the underlying ``frame.f_locals`` rather than implicitly +refreshing a single shared dictionary cached on the frame object. + +Implementation Notes +==================== + +When accepted, the PEP text suggested that ``PyEval_GetLocals`` would start returning a +cached instance of the new write-through proxy, while the implementation sketch indicated +it would continue to return a dictionary snapshot cached on the frame instance. This +discrepancy was identified while implementing the PEP, and +`resolved by the Steering Council `__ +in favour of retaining the Python 3.12 behaviour of returning a dictionary snapshot +cached on the frame instance. +The PEP text has been updated accordingly. + +During the discussions of the C API clarification, it also became apparent that the +rationale behind ``locals()`` being updated to return independent snapshots in +:term:`optimized scopes ` wasn't clear, as it had been inherited +from the original :pep:`558` discussions rather than being independently covered in this +PEP. The PEP text has been updated to better cover this change, with additional updates +to the Specification and Backwards Compatibility sections to cover the impact on code +execution APIs that default to executing code in the ``locals()`` namespace. Additional +motivation and rationale details have also been added to :pep:`558`. + +.. _pep-667-motivation: Motivation ========== -The current implementation of ``locals()`` and ``frame.f_locals`` is slow, -inconsistent and buggy. +The implementation of ``locals()`` and ``frame.f_locals`` in releases up to and +including Python 3.12 is slow, inconsistent and buggy. We want to make it faster, consistent, and most importantly fix the bugs. -For example:: +For example, when attempting to manipulate local variables via frame objects:: class C: x = 1 sys._getframe().f_locals['x'] = 2 print(x) -prints ``2`` - -but:: +prints ``2``, but:: def f(): x = 1 @@ -57,55 +78,118 @@ but:: print(x) f() -prints ``1`` +prints ``1``. -This is inconsistent, and confusing. -With this PEP both examples would print ``2``. +This is inconsistent, and confusing. Worse than that, the Python 3.12 behavior can +result in strange `bugs `__. -Worse than that, the current behavior can result in strange `bugs -`__. +With this PEP both examples would print ``2`` as the function level +change would be written directly to the optimized local variables in +the frame rather than to a cached dictionary snapshot. -There are no compensating advantages for the current behavior; +There are no compensating advantages for the Python 3.12 behavior; it is unreliable and slow. +The ``locals()`` builtin has its own undesirable behaviours. Refer to :pep:`558` +for additional details on those concerns. + + +.. _pep-667-rationale: + Rationale ========= -The current implementation of ``frame.f_locals`` returns a dictionary -that is created on the fly from the array of local variables. +Making the ``frame.f_locals`` attribute a write-through proxy +------------------------------------------------------------- + +The Python 3.12 implementation of ``frame.f_locals`` returns a dictionary +that is created on the fly from the array of local variables. The +``PyFrame_LocalsToFast()`` C API is then called by debuggers and trace +functions that want to write their changes back to the array (until +Python 3.11, this API was called implicitly after every trace function +invocation rather than being called explicitly by the trace functions). + This can result in the array and dictionary getting out of sync with -each other. Writes to the ``f_locals`` may not show up as -modifications to local variables. Writes to local variables can get lost. +each other. Writes to the ``f_locals`` frame attribute may not show up as +modifications to local variables if ``PyFrame_LocalsToFast()`` is never +called. Writes to local variables can get lost if a dictionary snapshot +created before the variables were modified is written back to the frame +(since *every* known variable stored in the snapshot is written back to +the frame, even if the value stored on the frame had changed since the +snapshot was taken). By making ``frame.f_locals`` return a view on the underlying frame, these problems go away. ``frame.f_locals`` is always in sync with the frame because it is a view of it, not a copy of it. +Making the ``locals()`` builtin return independent snapshots +------------------------------------------------------------ + +:pep:`558` considered three potential options for standardising the behavior of the +``locals()`` builtin in optimized scopes: + +* retain the historical behaviour of having each call to ``locals()`` on a given frame + update a single shared snapshot of the local variables +* make ``locals()`` return write-through proxy instances (similar + to ``frame.f_locals``) +* make ``locals()`` return genuinely independent snapshots so that + attempts to change the values of local variables via ``exec()`` + would be *consistently* ignored rather than being accepted in some circumstances + +The last option was chosen as the one which could most easily be explained in the +language reference, and memorised by users: + +* the ``locals()`` builtin gives an instantaneous snapshot of the local variables in + optimized scopes, and read/write access in other scopes; and +* ``frame.f_locals`` gives read/write access to the local variables in all scopes, + including optimized scopes + +This approach allows the intent of a piece of code to be clearer than it would be if both +APIs granted full read/write access in optimized scopes, even when write access wasn't +needed or desired. For additional details on this design decision, refer to :pep:`558`, +especially the :ref:`pep-558-motivation` section and :ref:`pep-558-exec-eval-impact`. + +This approach is not without its drawbacks, which are covered +in the Backwards Compatibility section below. + Specification ============= -Python ------- +Python API changes +------------------ -``frame.f_locals`` will return a view object on the frame that -implements the ``collections.abc.Mapping`` interface. +The ``frame.f_locals`` attribute +'''''''''''''''''''''''''''''''' -For module and class scopes ``frame.f_locals`` will be a dictionary, -for function scopes it will be a custom class. +For module and class scopes (including ``exec()`` and ``eval()`` +invocations), ``frame.f_locals`` will continue to be a direct +reference to the local variable namespace used in code execution. -``locals()`` will be defined as:: +For function scopes (and other optimized scopes) it will be an instance +of a new write-through proxy type that directly modifies the optimized +local variable storage array in the underlying frame, as well as the +contents of any cell references to non-local variables. - def locals(): - frame = sys._getframe(1) - f_locals = frame.f_locals - if frame.is_function(): - f_locals = dict(f_locals) - return f_locals +The view objects fully implement the ``collections.abc.Mapping`` interface, +and also implement the following mutable mapping operations: + +* using assignment to add new key/value pairs +* using assignment to update the value associated with a key +* conditional assignment via the ``setdefault()`` method +* bulk updates via the ``update()`` method + +Removing keys with ``del``, ``pop()`` or ``clear()`` is NOT supported. +Views of different frames compare unequal even if they have the same contents. All writes to the ``f_locals`` mapping will be immediately visible in the underlying variables. All changes to the underlying variables -will be immediately visible in the mapping. The ``f_locals`` object will -be a full mapping, and can have arbitrary key-value pairs added to it. +will be immediately visible in the mapping. + +The ``f_locals`` object will be a full mapping, and can have arbitrary +key-value pairs added to it. New names added via the proxies +will be stored in a dedicated shared dictionary stored on the +underlying frame object (so all proxy instances for a given frame +will be able to access any names added this way). For example:: @@ -124,7 +208,7 @@ For example:: ``test()`` will print ``{'x': 2, 'y': 4, 'z': 5} 2``. -In Python 3.10, the above will fail with an ``UnboundLocalError``, +In Python 3.12, the above will fail with an ``UnboundLocalError``, as the definition of ``y`` by ``l()['y'] = 4`` is lost. If the second-to-last line were changed from ``y`` to ``z``, this would be a @@ -132,11 +216,86 @@ If the second-to-last line were changed from ``y`` to ``z``, this would be a lexically local variables remain visible in ``frame.f_locals``, but do not dynamically become local variables. -C-API ------ +To maintain backwards compatibility, proxy APIs that need to produce a +new mapping (such as ``copy()``) will produce regular builtin ``dict`` +instances, rather than write-through proxy instances. -Extensions to the API -''''''''''''''''''''' +To avoid introducing a circular reference between frame objects and the +write-through proxies, each access to ``frame.f_locals`` returns a *new* +write-through proxy instance. + +The ``locals()`` builtin +'''''''''''''''''''''''' + +``locals()`` will be defined as:: + + def locals(): + frame = sys._getframe(1) + f_locals = frame.f_locals + if frame._is_optimized(): # Not an actual frame method + f_locals = dict(f_locals) + return f_locals + +For module and class scopes (including ``exec()`` and ``eval()`` +invocations), ``locals()`` continues to return a direct +reference to the local variable namespace used in code execution +(which is also the same value reported by ``frame.f_locals``). + +In optimized scopes, each call to ``locals()`` will produce an +*independent* snapshot of the local variables. + +For example:: + + def f(): + exec("x = 1") + print(locals().get("x")) + f() + +will *always* print ``None``, regardless of whether ``x`` is a +defined local variable in the function or not, as the explicit +call to ``locals()`` produces a distinct snapshot from the one +implicitly used in the ``exec()`` call. + +The ``eval()`` and ``exec()`` builtins +'''''''''''''''''''''''''''''''''''''' + +Because this PEP changes the behavior of ``locals()``, the +behavior of ``eval()`` and ``exec()`` also changes. + +Assuming a function ``_eval()`` which performs the job of +``eval()`` with explicit namespace arguments, ``eval()`` +can be defined as follows:: + + FrameProxyType = type((lambda: sys._getframe().f_locals)()) + + def eval(expression, /, globals=None, locals=None): + if globals is None: + # No globals -> use calling frame's globals + _calling_frame = sys._getframe(1) + globals = _calling_frame.f_globals + if locals is None: + # No globals or locals -> use calling frame's locals + locals = _calling_frame.f_locals + if isinstance(locals, FrameProxyType): + # Align with locals() builtin in optimized frame + locals = dict(locals) + elif locals is None: + # Globals but no locals -> use same namespace for both + locals = globals + return _eval(expression, globals, locals) + +The specified argument handling for ``exec()`` is similarly updated. + +(In Python 3.12 and earlier, it was not possible to provide ``locals`` +to ``eval()`` or ``exec()`` without also providing ``globals`` as these +were previously positional-only arguments. Independently of this +PEP, Python 3.13 updated these builtins to accept keyword arguments) + +C API changes +------------- + +Extensions to the C API +''''''''''''''''''''''' Three new C-API functions will be added:: @@ -149,11 +308,14 @@ Three new C-API functions will be added:: All these functions will return a new reference. -Changes to existing APIs -'''''''''''''''''''''''' +Changes to existing C APIs +'''''''''''''''''''''''''' ``PyFrame_GetLocals(f)`` is equivalent to ``f.f_locals``, and hence its return value -will change as described above for accessing ``f.f_locals``. +will change as described above for accessing ``f.f_locals``. Note that this function +can already return arbitrary mappings, as ``exec()`` and ``eval()`` accept arbitrary +mappings as their ``locals`` argument, and metaclasses may return arbitrary mappings +from their ``__prepare__`` methods. The following C-API functions will be deprecated, as they return borrowed references:: @@ -169,8 +331,10 @@ The following functions should be used instead:: which return new references. -The semantics of ``PyEval_GetLocals()`` is changed as it now returns a -proxy for the frame locals in optimized frames, not a dictionary. +The semantics of ``PyEval_GetLocals()`` are technically unchanged, but they do change in +practice as the dictionary cached on optimized frames is no longer shared with other +mechanisms for accessing the frame locals (``locals()`` builtin, ``PyFrame_GetLocals`` +function, frame ``f_locals`` attributes). The following three functions will become no-ops, and will be deprecated:: @@ -178,38 +342,172 @@ The following three functions will become no-ops, and will be deprecated:: PyFrame_FastToLocals() PyFrame_LocalsToFast() -Behavior of f_locals for optimized functions --------------------------------------------- - -Although ``f.f_locals`` behaves as if it were the namespace of the function, -there will be some observable differences. -For example, ``f.f_locals is f.f_locals`` may be ``False``. - -However ``f.f_locals == f.f_locals`` will be ``True``, and -all changes to the underlying variables, by any means, will always be visible. - Backwards Compatibility ======================= -Python ------- +Python API compatibility +------------------------ -The current implementation has many corner cases and oddities. -Code that works around those may need to be changed. +The implementation used in versions up to and including Python 3.12 has many +corner cases and oddities. Code that works around those may need to be changed. Code that uses ``locals()`` for simple templating, or print debugging, will continue to work correctly. Debuggers and other tools that use ``f_locals`` to modify local variables, will now work correctly, even in the presence of threaded code, coroutines and generators. -C-API ------ +``frame.f_locals`` compatibility +-------------------------------- + +Although ``f.f_locals`` behaves as if it were the namespace of the function, +there will be some observable differences. +For example, ``f.f_locals is f.f_locals`` will be ``False`` for optimized +frames, as each access to the attribute produces a new write-through proxy +instance. + +However ``f.f_locals == f.f_locals`` will be ``True``, and +all changes to the underlying variables, by any means, including the +addition of new variable names as mapping keys, will always be visible. + +``locals()`` compatibility +'''''''''''''''''''''''''' + +``locals() is locals()`` will be ``False`` for optimized frames, so +code like the following will raise ``KeyError`` instead of returning +``1``:: + + def f(): + locals()["x"] = 1 + return locals()["x"] + +To continue working, such code will need to explicitly store the namespace +to be modified in a local variable, rather than relying on the previous +implicit caching on the frame object:: + + def f(): + ns = {} + ns["x"] = 1 + return ns["x"] + +While this technically isn't a formal backwards compatibility break +(since the behaviour of writing back to ``locals()`` was explicitly +documented as undefined), there is definitely some code that relies +on the existing behaviour. Accordingly, the updated behaviour will +be explicitly noted in the documentation as a change and it will be +covered in the Python 3.13 porting guide. + +To work with a copy of ``locals()`` in optimized scopes on all +versions without making redundant copies on Python 3.13+, users +will need to define a version-dependent helper function that only +makes an explicit copy on Python versions prior to Python 3.13:: + + if sys.version_info >= (3, 13): + def _ensure_func_snapshot(d): + return d # 3.13+ locals() already returns a snapshot + else: + def _ensure_func_snapshot(d): + return dict(d) # Create snapshot on older versions + + def f(): + ns = _ensure_func_snapshot(locals()) + ns["x"] = 1 + return ns + +In other scopes, ``locals().copy()`` can continue to be called +unconditionally without introducing any redundant copies. + +Impact on ``exec()`` and ``eval()`` +''''''''''''''''''''''''''''''''''' + +Even though this PEP does not modify ``exec()`` or ``eval()`` directly, +the semantic change to ``locals()`` impacts the behavior of ``exec()`` +and ``eval()`` as they default to running code in the calling namespace. + +This poses a potential compatibility issue for some code, as with the +previous implementation that returns the same dict when ``locals()`` is called +multiple times in function scope, the following code usually worked due to +the implicitly shared local variable namespace:: + + def f(): + exec('a = 0') # equivalent to exec('a = 0', globals(), locals()) + exec('print(a)') # equivalent to exec('print(a)', globals(), locals()) + print(locals()) # {'a': 0} + # However, print(a) will not work here + f() + +With the semantic changes to ``locals()`` in this PEP, the ``exec('print(a)')'`` call +will fail with ``NameError``, and ``print(locals())`` will report an empty dictionary, as +each line will be using its own distinct snapshot of the local variables rather than +implicitly sharing a single cached snapshot stored on the frame object. + +A shared namespace across ``exec()`` calls can still be obtained by using explicit +namespaces rather than relying on the previously implicitly shared frame namespace:: + + def f(): + ns = {} + exec('a = 0', locals=ns) + exec('print(a)', locals=ns) # 0 + f() + +You can even reliably change the variables in the local scope by explicitly using +``frame.f_locals``, which was not possible before (even using ``ctypes`` to +invoke ``PyFrame_LocalsToFast`` was subject to the state inconsistency problems +discussed elsewhere in this PEP):: + + def f(): + a = None + exec('a = 0', locals=sys._getframe().f_locals) + print(a) # 0 + f() + +The behavior of ``exec()`` and ``eval()`` for module and class scopes (including +nested invocations) is not changed, as the behaviour of ``locals()`` in those +scopes is not changing. + +Impact on other code execution APIs in the standard library +''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +``pdb`` and ``bdb`` use the ``frame.f_locals`` API, and hence will be able to +reliably update local variables even in optimized frames. Implementing this +PEP will resolve several longstanding bugs in these modules relating to threads, +generators, coroutines, and other mechanisms that allow concurrent code execution +while the debugger is active. + +Other code execution APIs in the standard library (such as the ``code`` module) +do not implicitly access ``locals()`` *or* ``frame.f_locals``, but the behaviour +of explicitly passing these namespaces will change as described in the rest of +this PEP (passing ``locals()`` in optimized scopes will no longer implicitly +share the code execution namespace across calls, passing ``frame.f_locals`` +in optimized scopes will allow reliable modification of local variables and +nonlocal cell references). + +C API compatibility +------------------- PyEval_GetLocals '''''''''''''''' -Because ``PyEval_GetLocals()`` returns a borrowed reference, it requires -the proxy mapping to be cached on the frame, extending its lifetime and -creating a cycle. ``PyEval_GetFrameLocals()`` should be used instead. +``PyEval_GetLocals()`` has never historically distinguished between whether it was +emulating ``locals()`` or ``sys._getframe().f_locals`` at the Python level, as they all +returned references to the same shared cache of the local variable bindings. + +With this PEP, ``locals()`` changes to return independent snapshots on each call for +optimized frames, and ``frame.f_locals`` (along with ``PyFrame_GetLocals``) changes to +return new write-through proxy instances. + +Because ``PyEval_GetLocals()`` returns a borrowed reference, it isn't possible to update +its semantics to align with either of those alternatives, leaving it as the only remaining +API that requires a shared cache dictionary stored on the frame object. + +While this technically leaves the semantics of the function unchanged, it no longer allows +extra dict entries to be made visible to users of the other APIs, as those APIs are no longer +accessing the same underlying cache dictionary. + +Accordingly, the function will be marked as soft-deprecated in Python 3.13, and subsequently +hard-deprecated in Python 3.14, with a target removal date of Python 3.16 (two releases +after Python 3.14). Alternatives wil be recommended as described below. + +When ``PyEval_GetLocals()`` is being used as an equivalent to the Python ``locals()`` +builtin, ``PyEval_GetFrameLocals()`` should be used instead. This code:: @@ -226,6 +524,22 @@ should be replaced with:: goto error_handler; } +When ``PyEval_GetLocals()`` is being used as an equivalent to calling +``sys._getframe().f_locals`` in Python, it should be replaced by calling +``PyFrame_GetLocals()`` on the result of ``PyEval_GetFrame()``. + +In these cases, the original code should be replaced with:: + + frame = PyEval_GetFrame(); + if (frame == NULL) { + goto error_handler; + } + locals = PyFrame_GetLocals(frame); + frame = NULL; // Minimise visibility of borrowed reference + if (locals == NULL) { + goto error_handler; + } + Implementation ============== @@ -364,6 +678,8 @@ C API PyFrameObject * = ...; // Get the current frame. if (frame->_locals_cache == NULL) { frame->_locals_cache = PyEval_GetFrameLocals(); + } else { + PyDict_Update(frame->_locals_cache, PyFrame_GetLocals(frame)); } return frame->_locals_cache; } @@ -379,28 +695,46 @@ same inside or outside of the comprehension, and this will not change. The behavior of ``locals()`` inside functions will generally change as specified in the rest of this PEP. -For inlined comprehensions at module or class scope, currently calling -``locals()`` within the inlined comprehension returns a new dictionary for each -call. This PEP will make ``locals()`` within a function also always return a new -dictionary for each call, improving consistency; class or module scope inlined -comprehensions will appear to behave as if the inlined comprehension is still a -distinct function. +For inlined comprehensions at module or class scope, calling ``locals()`` within +the inlined comprehension returns a new dictionary for each call. This PEP will +make ``locals()`` within a function also always return a new dictionary for each +call, improving consistency; class or module scope inlined comprehensions will +appear to behave as if the inlined comprehension is still a distinct function. Comparison with PEP 558 ======================= -This PEP and :pep:`558` share a common goal: +This PEP and :pep:`558` shared a common goal: to make the semantics of ``locals()`` and ``frame.f_locals()`` intelligible, and their operation reliable. - -The key difference between this PEP and :pep:`558` is that -:pep:`558` keeps an internal copy of the local variables, -whereas this PEP does not. - -:pep:`558` does not specify exactly when the internal copy is -updated, making the behavior of :pep:`558` impossible to reason about. - +The key difference between this PEP and PEP 558 is that +PEP 558 attempted to store extra variables inside a full +internal dictionary copy of the local variables in an effort +to improve backwards compatibility with the legacy +``PyEval_GetLocals()`` API, whereas this PEP does not (it stores +the extra local variables in a dedicated dictionary accessed +solely via the new frame proxy objects, and copies them to the +``PyEval_GetLocals()`` shared dict only when requested). + +PEP 558 did not specify exactly when that internal copy was +updated, making the behavior of PEP 558 impossible to reason +about in several cases where this PEP remains well specified. + +PEP 558 also proposed the introduction of some additional Python +scope introspection interfaces to the C API that would allow +extension modules to more easily determine whether the currently +active Python scope is optimized or not, and hence whether +the C API's ``locals()`` equivalent returns a direct reference +to the frame's local execution namespace or a shallow copy of +the frame's local variables and nonlocal cell references. +Whether or not to add such introspection APIs is independent +of the proposed changes to ``locals()`` and ``frame.f_locals`` +and hence no such proposals have been included in this PEP. + +PEP 558 was +:pep:`ultimately withdrawn <558#pep-withdrawal>` +in favour of this PEP. Implementation ==============