From ab71e9a4672576961fda3d03973aa61cb02855d9 Mon Sep 17 00:00:00 2001 From: Tian Gao Date: Tue, 18 Jun 2024 18:29:03 -0700 Subject: [PATCH 01/36] Add impact on exec() for PEP 667 --- peps/pep-0667.rst | 60 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index cbe1b5d2640..94e32389ea5 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -371,6 +371,66 @@ C API As with all functions that return a borrowed reference, care must be taken to ensure that the reference is not used beyond the lifetime of the object. +Impact on ``exec()`` +==================== + +Even though this PEP does not modify ``exec()`` directly, the semantic change +to ``locals()`` impacts the behavior of ``exec()``. + +``exec(object)`` has been equivalent to ``exec(object, globals(), locals())`` +for a long time, but it's not officially documented. We plan to keep this +relation, which means existing code using ``exec()`` will see some different +behaviors. + +The core difference is that the current implementation returns the same dict +when ``locals()`` is called multiple times in function scope, so the following +code magically works:: + + def f(): + exec('a = 0') # equivalent to exec('a = 0', globals(), locals()) + exec('print(a)') # equivalent to exec('print(a)', globals(), locals()) + print(locals()) # {'a': 0} + # However, print(a) will not work here + f() + +``locals()`` is implicitly used as a scope shared between multiple ``exec()``s. + +This behavior gets harder to predict when the code in ``exec()`` tries to write +to an existing local variable:: + + def f(): + a = None + exec('a = 0') # equivalent to exec('a = 0', globals(), locals()) + exec('print(a)') # equivalent to exec('print(a)', globals(), locals()) + print(locals()) # {'a': None} + f() + +``print(a)`` will print ``None`` because the implicit ``locals()`` call in +``exec()`` refreshes the cached dict with the actual values on the frame. + +With the semantic changes to ``locals()``, it's easier to explain the behavior +of ``exec()`` - it will never affect local function scopes. Any assignment +to the local variables will be discarded when ``exec()`` returns. + +However, you can still achieve the wanted behavior with explicit scopes:: + + def f(): + scope = {} + exec('a = 0', locals=scope) + exec('print(a)', locals=scope) + f() + +You can even change the variables in the local scope by explicitly using +``frame.f_locals`` which is not possible before:: + + def f(): + a = None + exec('a = 0', locals=sys._getframe().f_locals) + print(a) # 0 + f() + +The behavior of ``exec()`` for module and class scopes is not changed. + Impact on PEP 709 inlined comprehensions ======================================== From 3d8602505d79d9bff5ec2b2a4d6bbaa0ad2b49ad Mon Sep 17 00:00:00 2001 From: Tian Gao Date: Tue, 18 Jun 2024 18:32:59 -0700 Subject: [PATCH 02/36] Fix lint --- peps/pep-0667.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 94e32389ea5..651308ac9d1 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -393,7 +393,7 @@ code magically works:: # However, print(a) will not work here f() -``locals()`` is implicitly used as a scope shared between multiple ``exec()``s. +``locals()`` is implicitly used as a scope shared between multiple ``exec()``. This behavior gets harder to predict when the code in ``exec()`` tries to write to an existing local variable:: From 8729874d1bb393c1a67d322205ff92f074a88d65 Mon Sep 17 00:00:00 2001 From: Tian Gao Date: Thu, 20 Jun 2024 17:25:08 -0700 Subject: [PATCH 03/36] Apply suggestions from code review Co-authored-by: Carl Meyer --- peps/pep-0667.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 651308ac9d1..4858ca41e2c 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -407,8 +407,10 @@ to an existing local variable:: ``print(a)`` will print ``None`` because the implicit ``locals()`` call in ``exec()`` refreshes the cached dict with the actual values on the frame. +So "real" locals, unlike "fake" locals created via ``exec()``, can't easily be +modified by ``exec()``. -With the semantic changes to ``locals()``, it's easier to explain the behavior +With the semantic changes to ``locals()`` in this PEP, it's easier to explain the behavior of ``exec()`` - it will never affect local function scopes. Any assignment to the local variables will be discarded when ``exec()`` returns. From e2f0b943fc90091ab8ad9c8b7be1267b87419181 Mon Sep 17 00:00:00 2001 From: Carol Willing Date: Fri, 21 Jun 2024 10:26:47 -0700 Subject: [PATCH 04/36] Apply @carljm suggestions from review Co-authored-by: Carl Meyer --- peps/pep-0667.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 4858ca41e2c..00c37c880c7 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -393,7 +393,9 @@ code magically works:: # However, print(a) will not work here f() -``locals()`` is implicitly used as a scope shared between multiple ``exec()``. +``locals()`` in a function currently returns the same persisted dict for every call. This allows stashing +extra "fake locals" in that dict, which aren't real locals known by the compiler, but can still be seen in +``locals()`` and shared between multiple ``exec()`` in the same function scope. This behavior gets harder to predict when the code in ``exec()`` tries to write to an existing local variable:: @@ -414,7 +416,7 @@ With the semantic changes to ``locals()`` in this PEP, it's easier to explain th of ``exec()`` - it will never affect local function scopes. Any assignment to the local variables will be discarded when ``exec()`` returns. -However, you can still achieve the wanted behavior with explicit scopes:: +However, you can still achieve a shared namespace across `exec()` calls with explicit namespaces:: def f(): scope = {} @@ -423,7 +425,7 @@ However, you can still achieve the wanted behavior with explicit scopes:: f() You can even change the variables in the local scope by explicitly using -``frame.f_locals`` which is not possible before:: +``frame.f_locals``, which was not possible before:: def f(): a = None From 98edffaab7d6b735c75ff86ef089cef40a910683 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 25 Jun 2024 14:26:04 +1000 Subject: [PATCH 05/36] Incorporate feedback and PyEval_GetLocals PR --- peps/pep-0667.rst | 608 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 482 insertions(+), 126 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 00c37c880c7..47e154c9d5a 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -31,7 +31,31 @@ consistent regardless of threading or coroutines. The ``locals()`` function will act the same as it does now for class and modules scopes. For function scopes it will return an instantaneous -snapshot of the underlying ``frame.f_locals``. +snapshot of the underlying ``frame.f_locals`` rather than implicitly +refreshing a single shared dictionary cached on the frame object. + +Implementation Notes +==================== + +When accepted, the PEP text suggested that ``PyEval_GetLocals`` would start returning a +cached instance of the new write-through proxy, while the implementation sketch indicated +it would continue to return a dictionary snapshot cached on the frame instance. This +discrepancy was identified while implementing the PEP, and +`resolved by the Steering Council `__ +in favour of retaining the Python 3.12 behaviour of returning a dictionary snapshot +cached on the frame instance. + +To avoid confusion when following the reference link from the Python 3.13 What's New +documentation, the PEP text has been updated accordingly. + +During the discussions of the C API clarification, it also became apparent that the +rationale behind ``locals()`` being updated to return independent snapshots in optimized +scopes wasn't clear, as it had been inherited from the original :pep:`558` discussions +rather than being independently covered in this PEP. The Abstract, Motivation, and +Rationale sections have been updated to cover this change, while the Backwards +Compatibility section has been updated to cover the impact on code execution APIs +like ``exec()`` and ``eval()`` that default to executing code in the ``locals()`` +namespace. Motivation ========== @@ -40,16 +64,14 @@ The current implementation of ``locals()`` and ``frame.f_locals`` is slow, inconsistent and buggy. We want to make it faster, consistent, and most importantly fix the bugs. -For example:: +For example, when attempting to manipulate local variables via frame objects:: class C: x = 1 sys._getframe().f_locals['x'] = 2 print(x) -prints ``2`` - -but:: +prints ``2``, but:: def f(): x = 1 @@ -57,55 +79,217 @@ but:: print(x) f() -prints ``1`` +prints ``1``. -This is inconsistent, and confusing. -With this PEP both examples would print ``2``. +This is inconsistent, and confusing. Worse than that, the current behavior can +result in strange `bugs `__. -Worse than that, the current behavior can result in strange `bugs -`__. +With this PEP both examples would print ``2`` as the function level +change would be written directly to the optimized local variables in +the frame rather than to a cached dictionary snapshot. There are no compensating advantages for the current behavior; it is unreliable and slow. +The ``locals()`` builtin has its own undesirable behaviours, where +adding a new local variable can change the behaviour of code +executed with ``exec()`` in function scopes, even if that code +runs *before* the local variable is defined. + +For example:: + + def f(): + exec("x = 1") + print(locals().get("x")) + f() + +prints ``1``, but:: + + def f(): + exec("x = 1") + print(locals().get("x")) + x = 0 + f() + +prints ``None`` (the default value from the ``.get()`` call). + +With this PEP both examples would print ``None``, as the call to +``exec()`` and the subsequent call to ``locals()`` would use +independent dictionary snapshots of the local variables rather +than using the same shared dictionary cached on the frame object. + Rationale ========= +Making the ``frame.f_locals`` attribute a write-through proxy +------------------------------------------------------------- + The current implementation of ``frame.f_locals`` returns a dictionary -that is created on the fly from the array of local variables. +that is created on the fly from the array of local variables. The +``PyEval_LocalsToFast()`` C API is then called by debuggers and trace +functions that want to write their changes back to the array (until +Python 3.11, this API was called implicitly after every trace function +invocation rather than being called explicitly by the trace functions). + This can result in the array and dictionary getting out of sync with -each other. Writes to the ``f_locals`` may not show up as -modifications to local variables. Writes to local variables can get lost. +each other. Writes to the ``f_locals`` frame atribute may not show up as +modifications to local variables if ``PyEval_LocalsToFast()`` is never +called. Writes to local variables can get lost if the dictionary snapshot +is not refreshed before being written back to the frame. By making ``frame.f_locals`` return a view on the underlying frame, these problems go away. ``frame.f_locals`` is always in sync with the frame because it is a view of it, not a copy of it. +To avoid introducing a circular reference between frame objects and the +write-through proxies, each access to ``frame.f_locals`` returns a *new* +write-through proxy instance. + +Making the ``locals()`` builtin return independent snapshots +------------------------------------------------------------ + +When ``exec`` was converted from a statement to a builtin function +in Python 3.0 (part of the core language changes in :pep`3100`), the +associated implicit call to ``PyEval_LocalsToFast()` was removed, so +it typically appears as if attempts to write to local variables with +``exec``in optimized frames are ignored:: + + >>> def f(): + ... x = 0 + ... exec("x = 1") + ... print(x) + ... print(locals()["x"]) + ... + >>> f() + 0 + 0 + +In truth, the writes aren't being ignored, they just aren't +being copied from the dictionary cache back to the optimized local +variable array. The changes to the dictonary are then overwritten +the next time the dictionary cache is refreshed from the array:: + + >>> def f(): + ... x = 0 + ... locals_cache = locals() + ... exec("x = 1") + ... print(x) + ... print(locals_cache["x"]) + ... print(locals()["x"]) + ... + >>> f() + 0 + 1 + 0 + +The behaviour becomes even stranger if a tracing function +or another piece of code invokes ``PyLocals_ToFast()`` before +the cache is next refreshed, as in those cases the change *is* +written back to the optimized local variable array:: + +>>> from sys import _getframe +>>> from ctypes import pythonapi, py_object, c_int +>>> _locals_to_fast = pythonapi.PyFrame_LocalsToFast +>>> _locals_to_fast.argtypes = [py_object, c_int] +>>> def f(): +... _frame = _getframe() +... _f_locals = _frame.f_locals +... x = 0 +... exec("x = 1") +... _locals_to_fast(_frame, 0) +... print(x) +... print(locals()["x"]) +... print(_f_locals["x"]) +... +>>> f() +1 +1 +1 + +This situation was more common in Python 3.10 and earlier +versions, as merely installing a tracing function was enough +to trigger implicit calls to ``PyEval_LocalsToFast()`` after +every line of Python code. However, it can still happen in Python +3.11+ depending on exactly which tracing functions are active +(e.g. interactive debuggers intentionally do this so that changes +made at the debugging prompt are visible when code execution +resumes). + +All of the above comments in relation to ``exec()`` apply to +*any* attempt to mutate the result of ``locals()`` in optimized +scopes, and are the main reason that the ``locals()`` builtin +docs contain this caveat: + + Note: The contents of this dictionary should not be modified; + changes may not affect the values of local and free variables + used by the interpreter. + +Two options were considered to replace this confusing behaviour: + +* make ``locals()`` return write-through proxy instances (similar + to ``frame.f_locals``) +* make ``locals()`` return genuinely independent snapshots so that + attempts to change the values of local variables via ``exec()`` + would be *consistently* ignored without any of the caveats + noted above. + +The PEP chooses the second option for the following reasons: + +* returning independent snapshots in optimized scopes preserves + the Python 3.0 change to ``exec()`` that resulted in attempts + to mutate local variables via ``exec()`` being ignored in most + cases +* the distinction between "``locals()`` gives an instantaneous + snapshot of the local variables in optimized scopes, and + read/write access in other scopes" and "``frame.f_locals`` + gives read/write access to the local variables in all scopes, + including optimized scopes" allows the intent of a piece of + code to be clearer than it would be if both APIs granted + full read/write access in optimized scopes, even when write + access wasn't needed or desired +* that clarity of intent allows optimizing compilers and + interpreters to assume all local variable rebindings are + visible in the code, even when `locals()`` is accessed + (or the default arguments are used in APIs like ``exec()`` + and ``eval()``). Deoptimization will only be needed when + the frame introspection API is accessed for that frame. +* only Python implementations that support the optional frame + introspection APIs will need to provide the new write-through + proxy support for optimized frames + +This approach is not without its drawbacks, which are covered +in the Backwards Compatibility section below. + Specification ============= -Python ------- +Python API changes +------------------ + +The ``frame.f_locals`` attribute +'''''''''''''''''''''''''''''''' ``frame.f_locals`` will return a view object on the frame that implements the ``collections.abc.Mapping`` interface. -For module and class scopes ``frame.f_locals`` will be a dictionary, -for function scopes it will be a custom class. +For module and class scopes (including ``exec()`` and ``eval()`` +invocations), ``frame.f_locals`` will continue to be a direct +reference to the local variable namespace used in code execution. -``locals()`` will be defined as:: - - def locals(): - frame = sys._getframe(1) - f_locals = frame.f_locals - if frame.is_function(): - f_locals = dict(f_locals) - return f_locals +For function scopes it will be an instance of a new write-through +proxy type that directly modifies the optimized local variable storage +array in the underlying frame, as well as the contents of any cell +references to non-local variables. All writes to the ``f_locals`` mapping will be immediately visible in the underlying variables. All changes to the underlying variables -will be immediately visible in the mapping. The ``f_locals`` object will -be a full mapping, and can have arbitrary key-value pairs added to it. +will be immediately visible in the mapping. + +The ``f_locals`` object will be a full mapping, and can have arbitrary +key-value pairs added to it. New names added via the proxies +will be stored in a dedicated shared dictionary stored on the +underlying frame object (so all proxy instances for a given frame +will be able to access any names added this way). For example:: @@ -124,7 +308,7 @@ For example:: ``test()`` will print ``{'x': 2, 'y': 4, 'z': 5} 2``. -In Python 3.10, the above will fail with an ``UnboundLocalError``, +In Python 3.12, the above will fail with an ``UnboundLocalError``, as the definition of ``y`` by ``l()['y'] = 4`` is lost. If the second-to-last line were changed from ``y`` to ``z``, this would be a @@ -132,11 +316,47 @@ If the second-to-last line were changed from ``y`` to ``z``, this would be a lexically local variables remain visible in ``frame.f_locals``, but do not dynamically become local variables. -C-API ------ +To maintain backwards compatibility, proxy APIs that need to produce a +new mapping (such as ``copy()``) will produce regular builtin ``dict`` +instances, rather than write-through proxy instances. + +The ``locals()`` builtin +'''''''''''''''''''''''' + +``locals()`` will be defined as:: + + def locals(): + frame = sys._getframe(1) + f_locals = frame.f_locals + if frame.is_function(): + f_locals = dict(f_locals) + return f_locals + +For module and class scopes (including ``exec()`` and ``eval()`` +invocations), ``locals()`` continues to return a direct +reference to the local variable namespace used in code execution +(which is also the same value reported by ``frame.f_locals``). + +In optimized scopes, each call to ``locals()`` will produce an +*independent* snapshot of the local variables. -Extensions to the API -''''''''''''''''''''' +For example:: + + def f(): + exec("x = 1") + print(locals().get("x")) + f() + +will *always* print ``None``, regardless of whether ``x`` is a +defined local variable in the function or not, as the explicit +call to ``locals()`` produces a distinct snapshot from the one +implicitly used in the ``exec()`` call. + +C API changes +------------- + +Extensions to the C API +''''''''''''''''''''''' Three new C-API functions will be added:: @@ -149,11 +369,14 @@ Three new C-API functions will be added:: All these functions will return a new reference. -Changes to existing APIs -'''''''''''''''''''''''' +Changes to existing C APIs +'''''''''''''''''''''''''' ``PyFrame_GetLocals(f)`` is equivalent to ``f.f_locals``, and hence its return value -will change as described above for accessing ``f.f_locals``. +will change as described above for accessing ``f.f_locals``. Note that this function +can already return arbitrary mappings, as ``exec()`` and ``eval()`` accept arbitrary +mappings as their ``locals`` argument, and metaclasses may return arbitrary mappings +from their ``__prepare__`` methods. The following C-API functions will be deprecated, as they return borrowed references:: @@ -169,8 +392,10 @@ The following functions should be used instead:: which return new references. -The semantics of ``PyEval_GetLocals()`` is changed as it now returns a -proxy for the frame locals in optimized frames, not a dictionary. +The semantics of ``PyEval_GetLocals()`` are technically unchanged, but they do change in +practice as the dictionary cached on optimized frames is no longer shared with other +mechanisms for accessing the frame locals (``locals()`` builtin, ``PyFrame_GetLocals`` +function, frame ``f_locals`` attributes). The following three functions will become no-ops, and will be deprecated:: @@ -178,21 +403,11 @@ The following three functions will become no-ops, and will be deprecated:: PyFrame_FastToLocals() PyFrame_LocalsToFast() -Behavior of f_locals for optimized functions --------------------------------------------- - -Although ``f.f_locals`` behaves as if it were the namespace of the function, -there will be some observable differences. -For example, ``f.f_locals is f.f_locals`` may be ``False``. - -However ``f.f_locals == f.f_locals`` will be ``True``, and -all changes to the underlying variables, by any means, will always be visible. - Backwards Compatibility ======================= -Python ------- +Python API compatibility +------------------------ The current implementation has many corner cases and oddities. Code that works around those may need to be changed. @@ -201,15 +416,199 @@ will continue to work correctly. Debuggers and other tools that use ``f_locals`` to modify local variables, will now work correctly, even in the presence of threaded code, coroutines and generators. -C-API ------ +``frame.f_locals`` compatibility +-------------------------------- + +Although ``f.f_locals`` behaves as if it were the namespace of the function, +there will be some observable differences. +For example, ``f.f_locals is f.f_locals`` will be ``False`` for optimized +frames, as each access to the atribute produces a new write-through proxy +instance. + +However ``f.f_locals == f.f_locals`` will be ``True``, and +all changes to the underlying variables, by any means, including the +addition of new variable names as mapping keys, will always be visible. + +``locals()`` compatibility +'''''''''''''''''''''''''' + +``locals() is locals()`` will be ``False`` for optimized frames, so +code like the following will raise ``KeyError`` instead of returning +``1``:: + + def f(): + locals()["x"] = 1 + return locals()["x"] + +To continue working, such code will need to explicitly cache the namespace +being modified in a local variable, rather than relying on the implicit +caching on the frame object:: + + def f(): + ns = locals() + ns["x"] = 1 + return ns["x"] + +This isn't formally a backwards compatibility break (since the behaviour of +writing back to ``locals()`` was explicitly documented as undefined), +but it will still be explicitly noted in the documentation as a change, +and covered in the Python 3.13 porting guide. + +Impact on ``exec()`` and ``eval()`` +''''''''''''''''''''''''''''''''''' + +Even though this PEP does not modify ``exec()`` or ``eval()`` directly, +the semantic change to ``locals()`` impacts the behavior of ``exec()``, +and ``eval()`` as they default to running code in the calling namespace. + +While the exact wording in the library reference is not entirely explicit, +both ``exec()`` and ``eval()`` have long used the results of calling +``globals()`` and ``locals()`` in the calling Python frame as their default +execution namespace. + +This was historically also equivalent to using the calling frame's +``frame.f_globals`` and ``frame.f_locals`` attributes, but this PEP maps +them to ``globals()`` and ``locals()`` in order to preserve the property +of ignoring attempted writes to the local namespace by default. + +However, as noted above for ``locals()``, this change has an additional +effect: each ``exec()`` call in an optimized scope will now run in a +*different* implicit namespace rather than a shared one. Furthermore, +separately calling ``locals()`` will also return a different namespace. + +This poses a potentially compatibility issue for some code, as with the +current implementation returning the same dict when ``locals()`` is called +multiple times in function scope, the following code usually works due to +the implicitly shared local variable namespace:: + + def f(): + exec('a = 0') # equivalent to exec('a = 0', globals(), locals()) + exec('print(a)') # equivalent to exec('print(a)', globals(), locals()) + print(locals()) # {'a': 0} + # However, print(a) will not work here + f() + +With ``locals()`` in an optimised scope returning the same shared dict for each call, +it is possible to store extra "fake locals" in that dict. While these aren't real +locals known by the compiler (so they can't be printed with code like ``print(a)``), +they can still be accessed via ``locals()`` and shared between multiple ``exec()`` +calls in the same function scope. Furthermore, because they're *not* real locals, +they don't get implicitly updated or removed when the shared cache is refreshed +from the local variable storage array. + +When the code in ``exec()`` tries to write to an existing local variable, the +runtime behaviour gets harder to predict:: + + def f(): + a = None + exec('a = 0') # equivalent to exec('a = 0', globals(), locals()) + exec('print(a)') # equivalent to exec('print(a)', globals(), locals()) + print(locals()) # {'a': None} + f() + +``print(a)`` will print ``None`` because the implicit ``locals()`` call in +``exec()`` refreshes the cached dict with the actual values on the frame. +So "real" locals, unlike "fake" locals created via ``exec()``, can't easily be +modified by ``exec()``. + +As noted in the Motivation section, this confusing side effect happens even if the +local variable is only defined *after* the ``exec`` calls:: + + >>> def f(): + ... exec("a = 0") + ... exec("print('a' in locals())") # Printing 'a' directly won't work + ... print(locals()) + ... a = None + ... print(locals()) + ... + >>> f() + False + {} + {'a': None} + +Because ``a`` is a real local variable, it gets removed from ``locals()`` when +it hasn't been bound yet, rather than being left alone like an entirely unknown +name. + +As noted above in the Rationale section, the above behavioural description may be +invalidated if the CPython ``PyFrame_LocalsToFast()`` API gets invoked while the frame +is still running. In that case, the changes to ``a`` *might* become visible to the +running code, depending on exactly when that API is called. + +With the semantic changes to ``locals()`` in this PEP, it becomes much easier to explain the +behavior of ``exec()`` and ``eval()`` in optimized scopes: they will never implicitly affect +local variables in optimized scopes. Any assignment to the local variables will be discarded +when the code execution API returns, since a fresh copy of the local variables is used on each +invocation. + +A shared namespace across ``exec()`` calls can still be obtained by using explicit namespaces +rather than relying on the implicitly shared frame namespace:: + + def f(): + ns = {} + exec('a = 0', locals=ns) + exec('print(a)', locals=ns) + f() + +You can even reliably change the variables in the local scope by explicitly using +``frame.f_locals``, which was not possible before (even using ``ctypes`` to +invoke ``PyFrame_LocalsToFast`` was subject to the state inconsistency problems +discussed elsewhere in this PEP):: + + def f(): + a = None + exec('a = 0', locals=sys._getframe().f_locals) + print(a) # 0 + f() + +The behavior of ``exec()`` and ``eval()`` for module and class scopes (including +nested invocations) is not changed, as the behaviour of ``locals()`` in those +scopes is not changing. + +Impact on other code execution APIs in the standard library +''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +``pdb`` and ``bdb` use the ``frame.f_locals`` API, and hence will be able to +reliably update local variables even in optimized frames. Implementing this +PEP will resolve several longstanding bugs in these modules relating to threads, +generators, coroutines, and other mechanisms that allow concurrent code execution +while the debugger is active. + +Other code execution APIs in the standard library (such as the ``code`` module) +do not implicitly access ``locals()`` *or* ``frame.f_locals``, but the behaviour +of explicitly passing these namespaces will change as described in the rest of +this PEP (passing ``locals()`` in optimized scopes will no longer implicitly +share the code execution namespace, passing ``frame.f_locals`` in optimized +scopes will allow reliable modification of local variables and nonlocal cell +references) + +C API compatibility +------------------- PyEval_GetLocals '''''''''''''''' -Because ``PyEval_GetLocals()`` returns a borrowed reference, it requires -the proxy mapping to be cached on the frame, extending its lifetime and -creating a cycle. ``PyEval_GetFrameLocals()`` should be used instead. +``PyEval_GetLocals()`` has never historically distinguished between whether it was +emulating ``locals()`` or ``sys._getframe().f_locals`` at the Python level, as they all +returned references to the same shared cache of the local variable bindings. + +With this PEP, ``locals()`` changes to return independent snapshots on each call for +optimized frames, and ``frame.f_locals`` (along with ``PyFrame_GetLocals``) changes to +return new write-through proxy instances. + +Because ``PyEval_GetLocals()`` returns a borrowed reference, it isn't possible to update +its semantics to align with either of those alternatives, leaving it as the only remaining +API that requires a shared cache dictionary stored on the frame object. + +While this technically leaves the semantics of the function unchanged, it no longer allows +extra dict entries to be made visible to users of the other APIs, as those APIs are no longer +accessing the same underlying cache dictionary. + +Accordingly, the function will be marked as deprecated, with a target removal date of +Python 3.15 (two releases after Python 3.13), and alternatives recommended as described below. + +When ``PyEval_GetLocals()`` is being used as an equivalent to the Python ``locals()`` +builtin, ``PyEval_GetFrameLocals()`` should be used instead. This code:: @@ -226,6 +625,22 @@ should be replaced with:: goto error_handler; } +When ``PyEval_GetLocals()`` is being used as an equivalent to calling +``sys._getframe().f_locals`` in Python, it should be replaced by calling +``PyFrame_GetLocals()`` on the result of ``PyEval_GetFrame()``. + +In these cases, the original code should be replaced with:: + + frame = PyEval_GetFrame(); + if (frame == NULL) { + goto error_handler; + } + locals = PyFrame_GetLocals(frame); + frame = NULL; // Minimise visibility of borrowed reference + if (locals == NULL) { + goto error_handler; + } + Implementation ============== @@ -371,70 +786,6 @@ C API As with all functions that return a borrowed reference, care must be taken to ensure that the reference is not used beyond the lifetime of the object. -Impact on ``exec()`` -==================== - -Even though this PEP does not modify ``exec()`` directly, the semantic change -to ``locals()`` impacts the behavior of ``exec()``. - -``exec(object)`` has been equivalent to ``exec(object, globals(), locals())`` -for a long time, but it's not officially documented. We plan to keep this -relation, which means existing code using ``exec()`` will see some different -behaviors. - -The core difference is that the current implementation returns the same dict -when ``locals()`` is called multiple times in function scope, so the following -code magically works:: - - def f(): - exec('a = 0') # equivalent to exec('a = 0', globals(), locals()) - exec('print(a)') # equivalent to exec('print(a)', globals(), locals()) - print(locals()) # {'a': 0} - # However, print(a) will not work here - f() - -``locals()`` in a function currently returns the same persisted dict for every call. This allows stashing -extra "fake locals" in that dict, which aren't real locals known by the compiler, but can still be seen in -``locals()`` and shared between multiple ``exec()`` in the same function scope. - -This behavior gets harder to predict when the code in ``exec()`` tries to write -to an existing local variable:: - - def f(): - a = None - exec('a = 0') # equivalent to exec('a = 0', globals(), locals()) - exec('print(a)') # equivalent to exec('print(a)', globals(), locals()) - print(locals()) # {'a': None} - f() - -``print(a)`` will print ``None`` because the implicit ``locals()`` call in -``exec()`` refreshes the cached dict with the actual values on the frame. -So "real" locals, unlike "fake" locals created via ``exec()``, can't easily be -modified by ``exec()``. - -With the semantic changes to ``locals()`` in this PEP, it's easier to explain the behavior -of ``exec()`` - it will never affect local function scopes. Any assignment -to the local variables will be discarded when ``exec()`` returns. - -However, you can still achieve a shared namespace across `exec()` calls with explicit namespaces:: - - def f(): - scope = {} - exec('a = 0', locals=scope) - exec('print(a)', locals=scope) - f() - -You can even change the variables in the local scope by explicitly using -``frame.f_locals``, which was not possible before:: - - def f(): - a = None - exec('a = 0', locals=sys._getframe().f_locals) - print(a) # 0 - f() - -The behavior of ``exec()`` for module and class scopes is not changed. - Impact on PEP 709 inlined comprehensions ======================================== @@ -453,18 +804,23 @@ distinct function. Comparison with PEP 558 ======================= -This PEP and :pep:`558` share a common goal: +This PEP and :pep:`558` shared a common goal: to make the semantics of ``locals()`` and ``frame.f_locals()`` intelligible, and their operation reliable. - The key difference between this PEP and :pep:`558` is that -:pep:`558` keeps an internal copy of the local variables, -whereas this PEP does not. - -:pep:`558` does not specify exactly when the internal copy is -updated, making the behavior of :pep:`558` impossible to reason about. - +:pep:`558` kept an internal copy of the local variables in an +effort to improve backwards compatibility with the legacy +``PyEval_GetLocals()`` API, whereas this PEP does not. + +:pep:`558` did not specify exactly when the internal copy was +updated, making the behavior of :pep:`558` impossible to +reason about in some cases where this PEP remains well +specified. + +:pep:`558` was +`ultimately withdrawn `__ +in favour of this PEP. Implementation ============== From 9a2a7d3b848b6a2d0919aeafbbaf985aac706984 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 25 Jun 2024 14:41:32 +1000 Subject: [PATCH 06/36] Syntax fixes, PEP 558 comparison tweaks --- peps/pep-0667.rst | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 47e154c9d5a..a01d392f03c 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -150,7 +150,7 @@ Making the ``locals()`` builtin return independent snapshots When ``exec`` was converted from a statement to a builtin function in Python 3.0 (part of the core language changes in :pep`3100`), the -associated implicit call to ``PyEval_LocalsToFast()` was removed, so +associated implicit call to ``PyEval_LocalsToFast()`` was removed, so it typically appears as if attempts to write to local variables with ``exec``in optimized frames are ignored:: @@ -249,7 +249,7 @@ The PEP chooses the second option for the following reasons: access wasn't needed or desired * that clarity of intent allows optimizing compilers and interpreters to assume all local variable rebindings are - visible in the code, even when `locals()`` is accessed + visible in the code, even when ``locals()`` is accessed (or the default arguments are used in APIs like ``exec()`` and ``eval()``). Deoptimization will only be needed when the frame introspection API is accessed for that frame. @@ -809,14 +809,18 @@ to make the semantics of ``locals()`` and ``frame.f_locals()`` intelligible, and their operation reliable. The key difference between this PEP and :pep:`558` is that -:pep:`558` kept an internal copy of the local variables in an -effort to improve backwards compatibility with the legacy -``PyEval_GetLocals()`` API, whereas this PEP does not. - -:pep:`558` did not specify exactly when the internal copy was -updated, making the behavior of :pep:`558` impossible to -reason about in some cases where this PEP remains well -specified. +:pep:`558` attempted to store extra variables inside a full +internal dictionary copy of the local variables in an effort +to improve backwards compatibility with the legacy +``PyEval_GetLocals()`` API, whereas this PEP does not (it stores +the extra local variables in a dedicated dictionary accessed +solely via the new frame proxy objects, and copies them to the +``PyEval_GetLocals()`` shared dict only when requested). + +:pep:`558` did not specify exactly when that internal copy was +updated by code execution and frame proxy operations, making +the behavior of :pep:`558` impossible to reason about in some +cases where this PEP remains well specified. :pep:`558` was `ultimately withdrawn `__ From e67a37db17850f08e14ee7fcf31929c7c2580aee Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 25 Jun 2024 15:15:18 +1000 Subject: [PATCH 07/36] Move f_locals spec note to spec section --- peps/pep-0667.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index a01d392f03c..0a65e4298ab 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -141,10 +141,6 @@ By making ``frame.f_locals`` return a view on the underlying frame, these problems go away. ``frame.f_locals`` is always in sync with the frame because it is a view of it, not a copy of it. -To avoid introducing a circular reference between frame objects and the -write-through proxies, each access to ``frame.f_locals`` returns a *new* -write-through proxy instance. - Making the ``locals()`` builtin return independent snapshots ------------------------------------------------------------ @@ -320,6 +316,10 @@ To maintain backwards compatibility, proxy APIs that need to produce a new mapping (such as ``copy()``) will produce regular builtin ``dict`` instances, rather than write-through proxy instances. +To avoid introducing a circular reference between frame objects and the +write-through proxies, each access to ``frame.f_locals`` returns a *new* +write-through proxy instance. + The ``locals()`` builtin '''''''''''''''''''''''' From a2b6b3395c2647eefa762ec3153d77d759de7668 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 25 Jun 2024 15:16:54 +1000 Subject: [PATCH 08/36] Add missing space --- peps/pep-0667.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 0a65e4298ab..0768e28b6e6 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -148,7 +148,7 @@ When ``exec`` was converted from a statement to a builtin function in Python 3.0 (part of the core language changes in :pep`3100`), the associated implicit call to ``PyEval_LocalsToFast()`` was removed, so it typically appears as if attempts to write to local variables with -``exec``in optimized frames are ignored:: +``exec`` in optimized frames are ignored:: >>> def f(): ... x = 0 From 6be7b286850e40cda5c5c352800696634ba9f777 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 25 Jun 2024 15:18:16 +1000 Subject: [PATCH 09/36] Indent ctypes example --- peps/pep-0667.rst | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 0768e28b6e6..aca18c9f435 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -183,24 +183,24 @@ or another piece of code invokes ``PyLocals_ToFast()`` before the cache is next refreshed, as in those cases the change *is* written back to the optimized local variable array:: ->>> from sys import _getframe ->>> from ctypes import pythonapi, py_object, c_int ->>> _locals_to_fast = pythonapi.PyFrame_LocalsToFast ->>> _locals_to_fast.argtypes = [py_object, c_int] ->>> def f(): -... _frame = _getframe() -... _f_locals = _frame.f_locals -... x = 0 -... exec("x = 1") -... _locals_to_fast(_frame, 0) -... print(x) -... print(locals()["x"]) -... print(_f_locals["x"]) -... ->>> f() -1 -1 -1 + >>> from sys import _getframe + >>> from ctypes import pythonapi, py_object, c_int + >>> _locals_to_fast = pythonapi.PyFrame_LocalsToFast + >>> _locals_to_fast.argtypes = [py_object, c_int] + >>> def f(): + ... _frame = _getframe() + ... _f_locals = _frame.f_locals + ... x = 0 + ... exec("x = 1") + ... _locals_to_fast(_frame, 0) + ... print(x) + ... print(locals()["x"]) + ... print(_f_locals["x"]) + ... + >>> f() + 1 + 1 + 1 This situation was more common in Python 3.10 and earlier versions, as merely installing a tracing function was enough From f2423f070c5ca5bce69618cdd11c36f08f475b0b Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 25 Jun 2024 15:37:04 +1000 Subject: [PATCH 10/36] Minor syntax and wording edits --- peps/pep-0667.rst | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index aca18c9f435..e3a6d74674f 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -422,7 +422,7 @@ even in the presence of threaded code, coroutines and generators. Although ``f.f_locals`` behaves as if it were the namespace of the function, there will be some observable differences. For example, ``f.f_locals is f.f_locals`` will be ``False`` for optimized -frames, as each access to the atribute produces a new write-through proxy +frames, as each access to the attribute produces a new write-through proxy instance. However ``f.f_locals == f.f_locals`` will be ``True``, and @@ -441,8 +441,8 @@ code like the following will raise ``KeyError`` instead of returning return locals()["x"] To continue working, such code will need to explicitly cache the namespace -being modified in a local variable, rather than relying on the implicit -caching on the frame object:: +being modified in a local variable, rather than relying on the previous +implicit caching on the frame object:: def f(): ns = locals() @@ -456,7 +456,6 @@ and covered in the Python 3.13 porting guide. Impact on ``exec()`` and ``eval()`` ''''''''''''''''''''''''''''''''''' - Even though this PEP does not modify ``exec()`` or ``eval()`` directly, the semantic change to ``locals()`` impacts the behavior of ``exec()``, and ``eval()`` as they default to running code in the calling namespace. @@ -542,7 +541,7 @@ when the code execution API returns, since a fresh copy of the local variables i invocation. A shared namespace across ``exec()`` calls can still be obtained by using explicit namespaces -rather than relying on the implicitly shared frame namespace:: +rather than relying on the previously implicitly shared frame namespace:: def f(): ns = {} @@ -568,7 +567,7 @@ scopes is not changing. Impact on other code execution APIs in the standard library ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' -``pdb`` and ``bdb` use the ``frame.f_locals`` API, and hence will be able to +``pdb`` and ``bdb`` use the ``frame.f_locals`` API, and hence will be able to reliably update local variables even in optimized frames. Implementing this PEP will resolve several longstanding bugs in these modules relating to threads, generators, coroutines, and other mechanisms that allow concurrent code execution @@ -578,9 +577,9 @@ Other code execution APIs in the standard library (such as the ``code`` module) do not implicitly access ``locals()`` *or* ``frame.f_locals``, but the behaviour of explicitly passing these namespaces will change as described in the rest of this PEP (passing ``locals()`` in optimized scopes will no longer implicitly -share the code execution namespace, passing ``frame.f_locals`` in optimized -scopes will allow reliable modification of local variables and nonlocal cell -references) +share the code execution namespace across call, passing ``frame.f_locals`` +in optimized scopes will allow reliable modification of local variables and +nonlocal cell references) C API compatibility ------------------- @@ -818,9 +817,8 @@ solely via the new frame proxy objects, and copies them to the ``PyEval_GetLocals()`` shared dict only when requested). :pep:`558` did not specify exactly when that internal copy was -updated by code execution and frame proxy operations, making -the behavior of :pep:`558` impossible to reason about in some -cases where this PEP remains well specified. +updated, making the behavior of :pep:`558` impossible to reason +about in several cases where this PEP remains well specified. :pep:`558` was `ultimately withdrawn `__ From 82f94fbe96860445024691b03536d6953e9a47eb Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 25 Jun 2024 15:52:44 +1000 Subject: [PATCH 11/36] Reword mention of "real" locals. --- peps/pep-0667.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index e3a6d74674f..fcae87c95b2 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -507,8 +507,9 @@ runtime behaviour gets harder to predict:: ``print(a)`` will print ``None`` because the implicit ``locals()`` call in ``exec()`` refreshes the cached dict with the actual values on the frame. -So "real" locals, unlike "fake" locals created via ``exec()``, can't easily be -modified by ``exec()``. +This means that, unlike the "fake" locals created by writing back to ``locals()`` +(including via previous calls to ``exec()``), the real locals known to the +compiler can't easily be modified by ``exec()``. As noted in the Motivation section, this confusing side effect happens even if the local variable is only defined *after* the ``exec`` calls:: From 4a85e39a1e4e1672010de0eb293ad5745a7f81c1 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 25 Jun 2024 16:05:15 +1000 Subject: [PATCH 12/36] PEP link fix, minor text edits --- peps/pep-0667.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index fcae87c95b2..832b5b7720c 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -547,7 +547,7 @@ rather than relying on the previously implicitly shared frame namespace:: def f(): ns = {} exec('a = 0', locals=ns) - exec('print(a)', locals=ns) + exec('print(a)', locals=ns) # 0 f() You can even reliably change the variables in the local scope by explicitly using @@ -578,7 +578,7 @@ Other code execution APIs in the standard library (such as the ``code`` module) do not implicitly access ``locals()`` *or* ``frame.f_locals``, but the behaviour of explicitly passing these namespaces will change as described in the rest of this PEP (passing ``locals()`` in optimized scopes will no longer implicitly -share the code execution namespace across call, passing ``frame.f_locals`` +share the code execution namespace across calls, passing ``frame.f_locals`` in optimized scopes will allow reliable modification of local variables and nonlocal cell references) @@ -822,7 +822,7 @@ updated, making the behavior of :pep:`558` impossible to reason about in several cases where this PEP remains well specified. :pep:`558` was -`ultimately withdrawn `__ +:pep:`ultimately withdrawn <558#pep-withdrawal>` in favour of this PEP. Implementation From 6115fbd848e75adb36dd1ac9399d2ec33ebf8b0d Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 25 Jun 2024 16:18:12 +1000 Subject: [PATCH 13/36] Add note about the extra PEP 558 C APIs being omitted --- peps/pep-0667.rst | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 832b5b7720c..38a2bbdac14 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -808,8 +808,8 @@ This PEP and :pep:`558` shared a common goal: to make the semantics of ``locals()`` and ``frame.f_locals()`` intelligible, and their operation reliable. -The key difference between this PEP and :pep:`558` is that -:pep:`558` attempted to store extra variables inside a full +The key difference between this PEP and PEP 558 is that +PEP 558 attempted to store extra variables inside a full internal dictionary copy of the local variables in an effort to improve backwards compatibility with the legacy ``PyEval_GetLocals()`` API, whereas this PEP does not (it stores @@ -817,11 +817,22 @@ the extra local variables in a dedicated dictionary accessed solely via the new frame proxy objects, and copies them to the ``PyEval_GetLocals()`` shared dict only when requested). -:pep:`558` did not specify exactly when that internal copy was -updated, making the behavior of :pep:`558` impossible to reason +PEP 558 did not specify exactly when that internal copy was +updated, making the behavior of PEP 558 impossible to reason about in several cases where this PEP remains well specified. -:pep:`558` was +PEP 558 also proposed the introduction of some additional Python +scope introspection interfaces to the C API that would allow +extension modules to more easily determine whether the currently +active Python scope is optimized or not, and hence whether +the C API's ``locals()`` equivalent returns a direct reference +to the frame's local execution namespace or a shallow copy of +the frame's local variables and nonlocal cell references. +Whether or not to add such introspection APIs is independent +of the proposed changes to ``locals()`` and ``frame.f_locals`` +and hence no such proposals have been included in this PEP. + +PEP 558 was :pep:`ultimately withdrawn <558#pep-withdrawal>` in favour of this PEP. From 669a0bc6b96092479d0842baf28c097766fe91bb Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 25 Jun 2024 16:30:37 +1000 Subject: [PATCH 14/36] Another link syntax fix --- peps/pep-0667.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 38a2bbdac14..2f9b164d18e 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -145,7 +145,7 @@ Making the ``locals()`` builtin return independent snapshots ------------------------------------------------------------ When ``exec`` was converted from a statement to a builtin function -in Python 3.0 (part of the core language changes in :pep`3100`), the +in Python 3.0 (part of the core language changes in :pep:`3100`), the associated implicit call to ``PyEval_LocalsToFast()`` was removed, so it typically appears as if attempts to write to local variables with ``exec`` in optimized frames are ignored:: From 9a966dd0613c62b981fea7af9da49ec1489a75c7 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 25 Jun 2024 16:42:47 +1000 Subject: [PATCH 15/36] Fix incorrect prefix in function name --- peps/pep-0667.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 2f9b164d18e..37a91001d05 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -126,14 +126,14 @@ Making the ``frame.f_locals`` attribute a write-through proxy The current implementation of ``frame.f_locals`` returns a dictionary that is created on the fly from the array of local variables. The -``PyEval_LocalsToFast()`` C API is then called by debuggers and trace +``PyFrame_LocalsToFast()`` C API is then called by debuggers and trace functions that want to write their changes back to the array (until Python 3.11, this API was called implicitly after every trace function invocation rather than being called explicitly by the trace functions). This can result in the array and dictionary getting out of sync with each other. Writes to the ``f_locals`` frame atribute may not show up as -modifications to local variables if ``PyEval_LocalsToFast()`` is never +modifications to local variables if ``PyFrame_LocalsToFast()`` is never called. Writes to local variables can get lost if the dictionary snapshot is not refreshed before being written back to the frame. @@ -146,7 +146,7 @@ Making the ``locals()`` builtin return independent snapshots When ``exec`` was converted from a statement to a builtin function in Python 3.0 (part of the core language changes in :pep:`3100`), the -associated implicit call to ``PyEval_LocalsToFast()`` was removed, so +associated implicit call to ``PyFrame_LocalsToFast()`` was removed, so it typically appears as if attempts to write to local variables with ``exec`` in optimized frames are ignored:: @@ -204,7 +204,7 @@ written back to the optimized local variable array:: This situation was more common in Python 3.10 and earlier versions, as merely installing a tracing function was enough -to trigger implicit calls to ``PyEval_LocalsToFast()`` after +to trigger implicit calls to ``PyFrame_LocalsToFast()`` after every line of Python code. However, it can still happen in Python 3.11+ depending on exactly which tracing functions are active (e.g. interactive debuggers intentionally do this so that changes From 98eddaf080ecca2a2c8baa31b435c7e4e8a83f48 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 25 Jun 2024 16:44:51 +1000 Subject: [PATCH 16/36] Improve locals() implementation note --- peps/pep-0667.rst | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 37a91001d05..a52423cf7f1 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -51,11 +51,10 @@ documentation, the PEP text has been updated accordingly. During the discussions of the C API clarification, it also became apparent that the rationale behind ``locals()`` being updated to return independent snapshots in optimized scopes wasn't clear, as it had been inherited from the original :pep:`558` discussions -rather than being independently covered in this PEP. The Abstract, Motivation, and -Rationale sections have been updated to cover this change, while the Backwards -Compatibility section has been updated to cover the impact on code execution APIs -like ``exec()`` and ``eval()`` that default to executing code in the ``locals()`` -namespace. +rather than being independently covered in this PEP. The PEP text has been updated to +better cover this change, with additional updates to the Backwards Compatibility section +to cover the impact on code execution APIs like ``exec()`` and ``eval()`` that +default to executing code in the ``locals()`` namespace. Motivation ========== From a8a700d301658fb6b2c76ef925027c9e2c459f7f Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 25 Jun 2024 17:15:09 +1000 Subject: [PATCH 17/36] Clarify the note about deoptimization --- peps/pep-0667.rst | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index a52423cf7f1..90cd59e481a 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -245,9 +245,20 @@ The PEP chooses the second option for the following reasons: * that clarity of intent allows optimizing compilers and interpreters to assume all local variable rebindings are visible in the code, even when ``locals()`` is accessed - (or the default arguments are used in APIs like ``exec()`` - and ``eval()``). Deoptimization will only be needed when - the frame introspection API is accessed for that frame. + or the default arguments are used in APIs like ``exec()`` + and ``eval()`` (optimization in this case refers to code + specialisation and other more advanced techniques, not + the basic optimization of replacing string dict lookups + with array index lookups - the latter feature is always + enabled in optimized scopes). Deoptimization will only + be needed when a write-through proxy instance is used to + modify a local variable. In a sufficiently sophisticated + interpreter, the deoptimization impact could potentially + be limited only to optimizations involving the modified + variable, but even less sophisticated interpreters will + be able to postpone frame deoptimization until at least + one local variable is modified outside of the normal flow + of code execution. * only Python implementations that support the optional frame introspection APIs will need to provide the new write-through proxy support for optimized frames From 78087595151eaddd0505d28fed3b10030692a57c Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 25 Jun 2024 17:20:19 +1000 Subject: [PATCH 18/36] Be clear there is no `is_function` frame method --- peps/pep-0667.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 90cd59e481a..3bf82b8402c 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -338,7 +338,7 @@ The ``locals()`` builtin def locals(): frame = sys._getframe(1) f_locals = frame.f_locals - if frame.is_function(): + if frame._is_optimized(): # Not an actual frame method f_locals = dict(f_locals) return f_locals From 6a89a65c794712e23ac471cea302f1fee648e3e9 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 25 Jun 2024 18:04:45 +1000 Subject: [PATCH 19/36] More syntax and wording tweaks --- peps/pep-0667.rst | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 3bf82b8402c..b815ebf3b14 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -466,6 +466,7 @@ and covered in the Python 3.13 porting guide. Impact on ``exec()`` and ``eval()`` ''''''''''''''''''''''''''''''''''' + Even though this PEP does not modify ``exec()`` or ``eval()`` directly, the semantic change to ``locals()`` impacts the behavior of ``exec()``, and ``eval()`` as they default to running code in the calling namespace. @@ -485,7 +486,7 @@ effect: each ``exec()`` call in an optimized scope will now run in a *different* implicit namespace rather than a shared one. Furthermore, separately calling ``locals()`` will also return a different namespace. -This poses a potentially compatibility issue for some code, as with the +This poses a potential compatibility issue for some code, as with the current implementation returning the same dict when ``locals()`` is called multiple times in function scope, the following code usually works due to the implicitly shared local variable namespace:: @@ -518,7 +519,7 @@ runtime behaviour gets harder to predict:: ``print(a)`` will print ``None`` because the implicit ``locals()`` call in ``exec()`` refreshes the cached dict with the actual values on the frame. This means that, unlike the "fake" locals created by writing back to ``locals()`` -(including via previous calls to ``exec()``), the real locals known to the +(including via previous calls to ``exec()``), the real locals known by the compiler can't easily be modified by ``exec()``. As noted in the Motivation section, this confusing side effect happens even if the @@ -540,15 +541,16 @@ Because ``a`` is a real local variable, it gets removed from ``locals()`` when it hasn't been bound yet, rather than being left alone like an entirely unknown name. -As noted above in the Rationale section, the above behavioural description may be +As noted in the Rationale section, the above behavioural description may be invalidated if the CPython ``PyFrame_LocalsToFast()`` API gets invoked while the frame is still running. In that case, the changes to ``a`` *might* become visible to the running code, depending on exactly when that API is called. With the semantic changes to ``locals()`` in this PEP, it becomes much easier to explain the -behavior of ``exec()`` and ``eval()`` in optimized scopes: they will never implicitly affect -local variables in optimized scopes. Any assignment to the local variables will be discarded -when the code execution API returns, since a fresh copy of the local variables is used on each +behavior of ``exec()`` and ``eval()``: in optimized scopes, they will *never* implicitly affect +local variables; in other scopes, they will *always* implicitly affect local variables. +In optimized scopes, any implicit assignment to the local variables will be discarded when +the code execution API returns, since a fresh copy of the local variables is used on each invocation. A shared namespace across ``exec()`` calls can still be obtained by using explicit namespaces @@ -572,7 +574,7 @@ discussed elsewhere in this PEP):: f() The behavior of ``exec()`` and ``eval()`` for module and class scopes (including -nested invocations) is not changed, as the behaviour of ``locals()`` in those +nested invocations) is not changed, as the behaviour of ``locals()`` in those scopes is not changing. Impact on other code execution APIs in the standard library From 22494e45aa6e0d5e9abb1d4bcf5b550739291d37 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 25 Jun 2024 19:31:26 +1000 Subject: [PATCH 20/36] Further updates * try to highlight C code and interactive shell sessions properly * make compatibility notes about redundant locals() copies * show dict update in PyEval_GetLocals implementation sketch --- peps/pep-0667.rst | 39 ++++++++++++++++++++++++++++++++------- 1 file changed, 32 insertions(+), 7 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index b815ebf3b14..718ab599bdd 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -10,6 +10,8 @@ Python-Version: 3.13 Post-History: 20-Aug-2021 Resolution: https://discuss.python.org/t/pep-667-consistent-views-of-namespaces/46631/25 +.. Syntax blocks are a mix of Python, C, and Python interactive prompts, so let Pygments guess +.. highlight:: guess Abstract ======== @@ -450,19 +452,40 @@ code like the following will raise ``KeyError`` instead of returning locals()["x"] = 1 return locals()["x"] -To continue working, such code will need to explicitly cache the namespace -being modified in a local variable, rather than relying on the previous +To continue working, such code will need to explicitly store the namespace +to be modified in a local variable, rather than relying on the previous implicit caching on the frame object:: def f(): - ns = locals() + ns = {} ns["x"] = 1 return ns["x"] -This isn't formally a backwards compatibility break (since the behaviour of -writing back to ``locals()`` was explicitly documented as undefined), -but it will still be explicitly noted in the documentation as a change, -and covered in the Python 3.13 porting guide. +While this technically isn't a formal backwards compatibility break +(since the behaviour of writing back to ``locals()`` was explicitly +documented as undefined), there is definitely some code that relies +on the existing behaviour. Accordingly, the updated behaviour will +be explicitly noted in the documentation as a change and it will be +covered in the Python 3.13 porting guide. + +To work with a copy of ``locals()`` on all versions without making +redundant copies on Python 3.13+, a helper function will need to +be defined in the affected code:: + + if sys.version_info >= (3, 13): + def _ensure_func_snapshot(d): + return d # 3.13+ locals() already returns a snapshot + else: + def _ensure_func_snapshot(d): + return dict(d) # Create snapshot on older versions + + def f(): + ns = _ensure_func_snapshot(locals()) + ns["x"] = 1 + return ns + +In other scopes, ``locals().copy()`` can continue to be called +unconditionally without introducing any redundant copies. Impact on ``exec()`` and ``eval()`` ''''''''''''''''''''''''''''''''''' @@ -791,6 +814,8 @@ C API PyFrameObject * = ...; // Get the current frame. if (frame->_locals_cache == NULL) { frame->_locals_cache = PyEval_GetFrameLocals(); + } else { + PyDict_Update(frame->_locals_cache, PyFrame_GetLocals(frame)); } return frame->_locals_cache; } From 828daa25be869f3c04b18977ee486c8560af6905 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 25 Jun 2024 19:35:53 +1000 Subject: [PATCH 21/36] No joy on mixed syntax highlighting (and I'm not worried enough about it to add per-block adjustments) --- peps/pep-0667.rst | 3 --- 1 file changed, 3 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 718ab599bdd..84b84e80d6d 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -10,9 +10,6 @@ Python-Version: 3.13 Post-History: 20-Aug-2021 Resolution: https://discuss.python.org/t/pep-667-consistent-views-of-namespaces/46631/25 -.. Syntax blocks are a mix of Python, C, and Python interactive prompts, so let Pygments guess -.. highlight:: guess - Abstract ======== From 1bc2eaa1f4bd6e2e40fb8448e2ee06bc72a6ead5 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Thu, 27 Jun 2024 10:56:37 +1000 Subject: [PATCH 22/36] Remove leftover comma Co-authored-by: Carl Meyer --- peps/pep-0667.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 84b84e80d6d..a5a33a1baf6 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -488,7 +488,7 @@ Impact on ``exec()`` and ``eval()`` ''''''''''''''''''''''''''''''''''' Even though this PEP does not modify ``exec()`` or ``eval()`` directly, -the semantic change to ``locals()`` impacts the behavior of ``exec()``, +the semantic change to ``locals()`` impacts the behavior of ``exec()`` and ``eval()`` as they default to running code in the calling namespace. While the exact wording in the library reference is not entirely explicit, From a924f4bdcbcc8c2df80aea6d90abe1d9a306f28c Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Fri, 28 Jun 2024 16:31:03 +1000 Subject: [PATCH 23/36] Reword note about better optimization support --- peps/pep-0667.rst | 22 +++++----------------- 1 file changed, 5 insertions(+), 17 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index a5a33a1baf6..edaafe59845 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -241,23 +241,11 @@ The PEP chooses the second option for the following reasons: code to be clearer than it would be if both APIs granted full read/write access in optimized scopes, even when write access wasn't needed or desired -* that clarity of intent allows optimizing compilers and - interpreters to assume all local variable rebindings are - visible in the code, even when ``locals()`` is accessed - or the default arguments are used in APIs like ``exec()`` - and ``eval()`` (optimization in this case refers to code - specialisation and other more advanced techniques, not - the basic optimization of replacing string dict lookups - with array index lookups - the latter feature is always - enabled in optimized scopes). Deoptimization will only - be needed when a write-through proxy instance is used to - modify a local variable. In a sufficiently sophisticated - interpreter, the deoptimization impact could potentially - be limited only to optimizations involving the modified - variable, but even less sophisticated interpreters will - be able to postpone frame deoptimization until at least - one local variable is modified outside of the normal flow - of code execution. +* in addition to improving clarity for human readers, ensuring + that name rebinding in optimized scopes remains lexically + visible in the code (as long as the frame introspection APIs + are not accessed) allows compilers and interpreters to apply + related performance optimizations more consistently * only Python implementations that support the optional frame introspection APIs will need to provide the new write-through proxy support for optimized frames From 32fc5fb7c75d43e7c3b04ae50e1595f4323d5e0c Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Thu, 4 Jul 2024 13:09:00 +1000 Subject: [PATCH 24/36] Address review comments from Barry & Tian --- peps/pep-0667.rst | 93 ++++++++++++++++++++++++++++------------------- 1 file changed, 55 insertions(+), 38 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 66caac484d7..1bce1924422 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -48,18 +48,20 @@ To avoid confusion when following the reference link from the Python 3.13 What's documentation, the PEP text has been updated accordingly. During the discussions of the C API clarification, it also became apparent that the -rationale behind ``locals()`` being updated to return independent snapshots in optimized -scopes wasn't clear, as it had been inherited from the original :pep:`558` discussions -rather than being independently covered in this PEP. The PEP text has been updated to -better cover this change, with additional updates to the Backwards Compatibility section -to cover the impact on code execution APIs like ``exec()`` and ``eval()`` that -default to executing code in the ``locals()`` namespace. +rationale behind ``locals()`` being updated to return independent snapshots in +:term:`optimized scopes ` wasn't clear, as it had been inherited +from the original :pep:`558` discussions rather than being independently covered in this +PEP. The PEP text has been updated to better cover this change, with additional updates +to the Backwards Compatibility section to cover the impact on code execution APIs like +``exec()`` and ``eval()`` that default to executing code in the ``locals()`` namespace. + +.. _pep-667-motivation: Motivation ========== -The current implementation of ``locals()`` and ``frame.f_locals`` is slow, -inconsistent and buggy. +The implementation of ``locals()`` and ``frame.f_locals`` in releases up to and +including Python 3.12 is slow, inconsistent and buggy. We want to make it faster, consistent, and most importantly fix the bugs. For example, when attempting to manipulate local variables via frame objects:: @@ -79,14 +81,14 @@ prints ``2``, but:: prints ``1``. -This is inconsistent, and confusing. Worse than that, the current behavior can +This is inconsistent, and confusing. Worse than that, the Python 3.12 behavior can result in strange `bugs `__. With this PEP both examples would print ``2`` as the function level change would be written directly to the optimized local variables in the frame rather than to a cached dictionary snapshot. -There are no compensating advantages for the current behavior; +There are no compensating advantages for the Python 3.12 behavior; it is unreliable and slow. The ``locals()`` builtin has its own undesirable behaviours, where @@ -116,13 +118,15 @@ With this PEP both examples would print ``None``, as the call to independent dictionary snapshots of the local variables rather than using the same shared dictionary cached on the frame object. +.. _pep-667-rationale: + Rationale ========= Making the ``frame.f_locals`` attribute a write-through proxy ------------------------------------------------------------- -The current implementation of ``frame.f_locals`` returns a dictionary +The Python 3.12 implementation of ``frame.f_locals`` returns a dictionary that is created on the fly from the array of local variables. The ``PyFrame_LocalsToFast()`` C API is then called by debuggers and trace functions that want to write their changes back to the array (until @@ -132,8 +136,11 @@ invocation rather than being called explicitly by the trace functions). This can result in the array and dictionary getting out of sync with each other. Writes to the ``f_locals`` frame atribute may not show up as modifications to local variables if ``PyFrame_LocalsToFast()`` is never -called. Writes to local variables can get lost if the dictionary snapshot -is not refreshed before being written back to the frame. +called. Writes to local variables can get lost if a dictionary snapshot +created before the variables were modified is written back to the frame +(since *every* known variable stored in the snapshot is written back to +the frame, even if the value stored on the frame had changed since the +snapshot was taken). By making ``frame.f_locals`` return a view on the underlying frame, these problems go away. ``frame.f_locals`` is always in @@ -160,7 +167,7 @@ it typically appears as if attempts to write to local variables with In truth, the writes aren't being ignored, they just aren't being copied from the dictionary cache back to the optimized local -variable array. The changes to the dictonary are then overwritten +variable array. The changes to the dictionary are then overwritten the next time the dictionary cache is refreshed from the array:: >>> def f(): @@ -176,9 +183,11 @@ the next time the dictionary cache is refreshed from the array:: 1 0 +.. _pep-667-ctypes-example: + The behaviour becomes even stranger if a tracing function -or another piece of code invokes ``PyLocals_ToFast()`` before -the cache is next refreshed, as in those cases the change *is* +or another piece of code invokes ``PyFrame_LocalsToFast()`` before +the cache is next refreshed. In those cases the change *is* written back to the optimized local variable array:: >>> from sys import _getframe @@ -406,8 +415,8 @@ Backwards Compatibility Python API compatibility ------------------------ -The current implementation has many corner cases and oddities. -Code that works around those may need to be changed. +The implementation used in versions up to and including Python 3.12 has many +corner cases and oddities. Code that works around those may need to be changed. Code that uses ``locals()`` for simple templating, or print debugging, will continue to work correctly. Debuggers and other tools that use ``f_locals`` to modify local variables, will now work correctly, @@ -454,8 +463,9 @@ be explicitly noted in the documentation as a change and it will be covered in the Python 3.13 porting guide. To work with a copy of ``locals()`` on all versions without making -redundant copies on Python 3.13+, a helper function will need to -be defined in the affected code:: +redundant copies on Python 3.13+, users will need to define a +version-dependent helper function that only makes an explicit +copy on Python versions prior to Python 3.13:: if sys.version_info >= (3, 13): def _ensure_func_snapshot(d): @@ -486,8 +496,10 @@ execution namespace. This was historically also equivalent to using the calling frame's ``frame.f_globals`` and ``frame.f_locals`` attributes, but this PEP maps -them to ``globals()`` and ``locals()`` in order to preserve the property -of ignoring attempted writes to the local namespace by default. +the default namespace arguments for ``exec()`` and ``eval()`` to +``globals()`` and ``locals()`` in the calling frame in order to preserve +the property of defaulting to ignoring attempted writes to the local +namespace in optimized scopes. However, as noted above for ``locals()``, this change has an additional effect: each ``exec()`` call in an optimized scope will now run in a @@ -495,7 +507,7 @@ effect: each ``exec()`` call in an optimized scope will now run in a separately calling ``locals()`` will also return a different namespace. This poses a potential compatibility issue for some code, as with the -current implementation returning the same dict when ``locals()`` is called +previous implementation that returns the same dict when ``locals()`` is called multiple times in function scope, the following code usually works due to the implicitly shared local variable namespace:: @@ -528,10 +540,13 @@ runtime behaviour gets harder to predict:: ``exec()`` refreshes the cached dict with the actual values on the frame. This means that, unlike the "fake" locals created by writing back to ``locals()`` (including via previous calls to ``exec()``), the real locals known by the -compiler can't easily be modified by ``exec()``. +compiler can't easily be modified by ``exec()`` (it can be done, but it requires +both retrieving the `frame.f_locals` attribute to enables writes back to the frame, +and then invoking ``PyFrame_LocalsToFast()``, as :ref:`shown ` +using ``ctypes`` in the :ref:`pep-667-rationale` section above). -As noted in the Motivation section, this confusing side effect happens even if the -local variable is only defined *after* the ``exec`` calls:: +As noted in the :ref:`pep-667-motivation` section, this confusing side effect +happens even if the local variable is only defined *after* the ``exec`` calls:: >>> def f(): ... exec("a = 0") @@ -545,14 +560,17 @@ local variable is only defined *after* the ``exec`` calls:: {} {'a': None} -Because ``a`` is a real local variable, it gets removed from ``locals()`` when -it hasn't been bound yet, rather than being left alone like an entirely unknown -name. +Because ``a`` is a real local variable that is not currently bound to a value, it +gets explicitly removed from the dictionary returned by ``locals()`` whenever +``locals()`` is called prior to the ``a = None`` line. This removal is intentional, +as it allows the contents of ``locals()`` to be updated correctly in optimized +scopes when ``del`` statements are used to delete previously bound local variables. -As noted in the Rationale section, the above behavioural description may be +As noted in the :ref:`pep-667-rationale` section, the above behavioural description may be invalidated if the CPython ``PyFrame_LocalsToFast()`` API gets invoked while the frame is still running. In that case, the changes to ``a`` *might* become visible to the -running code, depending on exactly when that API is called. +running code, depending on exactly when that API is called (and whether the frame +has been primed for locals modification by accessing the ``frame.f_locals`` attribute). With the semantic changes to ``locals()`` in this PEP, it becomes much easier to explain the behavior of ``exec()`` and ``eval()``: in optimized scopes, they will *never* implicitly affect @@ -600,7 +618,7 @@ of explicitly passing these namespaces will change as described in the rest of this PEP (passing ``locals()`` in optimized scopes will no longer implicitly share the code execution namespace across calls, passing ``frame.f_locals`` in optimized scopes will allow reliable modification of local variables and -nonlocal cell references) +nonlocal cell references). C API compatibility ------------------- @@ -816,12 +834,11 @@ same inside or outside of the comprehension, and this will not change. The behavior of ``locals()`` inside functions will generally change as specified in the rest of this PEP. -For inlined comprehensions at module or class scope, currently calling -``locals()`` within the inlined comprehension returns a new dictionary for each -call. This PEP will make ``locals()`` within a function also always return a new -dictionary for each call, improving consistency; class or module scope inlined -comprehensions will appear to behave as if the inlined comprehension is still a -distinct function. +For inlined comprehensions at module or class scope, calling ``locals()`` within +the inlined comprehension returns a new dictionary for each call. This PEP will +make ``locals()`` within a function also always return a new dictionary for each +call, improving consistency; class or module scope inlined comprehensions will +appear to behave as if the inlined comprehension is still a distinct function. Comparison with PEP 558 ======================= From 6449585d8374f0d9054798c338deabcc93c2af05 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Thu, 4 Jul 2024 13:12:58 +1000 Subject: [PATCH 25/36] Syntax fix --- peps/pep-0667.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 1bce1924422..e7081ff187c 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -541,7 +541,7 @@ runtime behaviour gets harder to predict:: This means that, unlike the "fake" locals created by writing back to ``locals()`` (including via previous calls to ``exec()``), the real locals known by the compiler can't easily be modified by ``exec()`` (it can be done, but it requires -both retrieving the `frame.f_locals` attribute to enables writes back to the frame, +both retrieving the ``frame.f_locals`` attribute to enables writes back to the frame, and then invoking ``PyFrame_LocalsToFast()``, as :ref:`shown ` using ``ctypes`` in the :ref:`pep-667-rationale` section above). From a0881b198394d5435e783117948f967882710987 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Thu, 4 Jul 2024 13:16:52 +1000 Subject: [PATCH 26/36] Fix typo --- peps/pep-0667.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index e7081ff187c..a0b5c202d9a 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -134,7 +134,7 @@ Python 3.11, this API was called implicitly after every trace function invocation rather than being called explicitly by the trace functions). This can result in the array and dictionary getting out of sync with -each other. Writes to the ``f_locals`` frame atribute may not show up as +each other. Writes to the ``f_locals`` frame attribute may not show up as modifications to local variables if ``PyFrame_LocalsToFast()`` is never called. Writes to local variables can get lost if a dictionary snapshot created before the variables were modified is written back to the frame From 8dec69641295c53d5881a301caedee1578eb6f63 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Thu, 4 Jul 2024 13:25:04 +1000 Subject: [PATCH 27/36] f_locals is mutable, not read-only! --- peps/pep-0667.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index a0b5c202d9a..ac61e2c0d59 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -272,7 +272,7 @@ The ``frame.f_locals`` attribute '''''''''''''''''''''''''''''''' ``frame.f_locals`` will return a view object on the frame that -implements the ``collections.abc.Mapping`` interface. +implements the ``collections.abc.MutableMapping`` interface. For module and class scopes (including ``exec()`` and ``eval()`` invocations), ``frame.f_locals`` will continue to be a direct From 4151acb2adad5502c404c019c663190b6991102f Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Thu, 4 Jul 2024 13:43:21 +1000 Subject: [PATCH 28/36] Misc proofreading edits --- peps/pep-0667.rst | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index ac61e2c0d59..9823483d606 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -278,10 +278,10 @@ For module and class scopes (including ``exec()`` and ``eval()`` invocations), ``frame.f_locals`` will continue to be a direct reference to the local variable namespace used in code execution. -For function scopes it will be an instance of a new write-through -proxy type that directly modifies the optimized local variable storage -array in the underlying frame, as well as the contents of any cell -references to non-local variables. +For function scopes (and other optimized scopes) it will be an instance +of a new write-through proxy type that directly modifies the optimized +local variable storage array in the underlying frame, as well as the +contents of any cell references to non-local variables. All writes to the ``f_locals`` mapping will be immediately visible in the underlying variables. All changes to the underlying variables @@ -462,10 +462,10 @@ on the existing behaviour. Accordingly, the updated behaviour will be explicitly noted in the documentation as a change and it will be covered in the Python 3.13 porting guide. -To work with a copy of ``locals()`` on all versions without making -redundant copies on Python 3.13+, users will need to define a -version-dependent helper function that only makes an explicit -copy on Python versions prior to Python 3.13:: +To work with a copy of ``locals()`` in optimized scopes on all +versions without making redundant copies on Python 3.13+, users +will need to define a version-dependent helper function that only +makes an explicit copy on Python versions prior to Python 3.13:: if sys.version_info >= (3, 13): def _ensure_func_snapshot(d): @@ -541,12 +541,12 @@ runtime behaviour gets harder to predict:: This means that, unlike the "fake" locals created by writing back to ``locals()`` (including via previous calls to ``exec()``), the real locals known by the compiler can't easily be modified by ``exec()`` (it can be done, but it requires -both retrieving the ``frame.f_locals`` attribute to enables writes back to the frame, +both retrieving the ``frame.f_locals`` attribute to enable writes back to the frame, and then invoking ``PyFrame_LocalsToFast()``, as :ref:`shown ` using ``ctypes`` in the :ref:`pep-667-rationale` section above). As noted in the :ref:`pep-667-motivation` section, this confusing side effect -happens even if the local variable is only defined *after* the ``exec`` calls:: +happens even if the local variable is only defined *after* the ``exec()`` calls:: >>> def f(): ... exec("a = 0") From 4871a396da3af7f9b9502c24ac77ad6ea85029bf Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 9 Jul 2024 18:26:03 +1000 Subject: [PATCH 29/36] Define exec/eval, fix proxy mutability * exec/eval semantics should be clearly specified * proxies do not fully implement MutableMapping --- peps/pep-0667.rst | 48 ++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 45 insertions(+), 3 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 9823483d606..68709458eb1 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -271,9 +271,6 @@ Python API changes The ``frame.f_locals`` attribute '''''''''''''''''''''''''''''''' -``frame.f_locals`` will return a view object on the frame that -implements the ``collections.abc.MutableMapping`` interface. - For module and class scopes (including ``exec()`` and ``eval()`` invocations), ``frame.f_locals`` will continue to be a direct reference to the local variable namespace used in code execution. @@ -283,6 +280,16 @@ of a new write-through proxy type that directly modifies the optimized local variable storage array in the underlying frame, as well as the contents of any cell references to non-local variables. +The view objects fully implement the ``collections.abc.Mapping`` interface, +and also implement the following mutable mapping operations: + +* using assignment to add new key/value pairs +* using assignment to update the value associated with a key +* conditional assignment via the `setdefault()` method +* bulk updates via the `update()` method + +Removing keys with `del` statements or the `pop()` method is NOT supported. + All writes to the ``f_locals`` mapping will be immediately visible in the underlying variables. All changes to the underlying variables will be immediately visible in the mapping. @@ -358,6 +365,41 @@ defined local variable in the function or not, as the explicit call to ``locals()`` produces a distinct snapshot from the one implicitly used in the ``exec()`` call. +The ``eval()`` and ``exec()`` builtins +'''''''''''''''''''''''''''''''''''''' + +Because this PEP changes the behavior of ``locals()`, the +behavior of ``eval()`` and ``exec()`` also changes. + +Assuming a function ``_eval()`` which performs the job of +``eval()`` with explicit namespace arguments, ``eval()`` +can be defined as follows:: + + FrameProxyType = type((lambda: sys._getframe().f_locals)()) + + def eval(expression, the_globals=None, the_locals=None): + if the_globals is None: + # No globals -> use calling frame's globals + _calling_frame = sys._getframe(1) + the_globals = _calling_frame.f_globals + if the_locals is None: + # No globals or locals -> use calling frame's locals + the_locals = _calling_frame.f_locals + if isinstance(the_locals, FrameProxyType): + # Align with locals() builtin in optimized frame + the_locals = the_locals.copy() + elif the_locals is None: + # Globals but no locals -> use same namespace for both + the_locals = the_globals + return _eval(expression, the_globals, the_locals) + +The argument handling for ``exec()`` is similarly updated. + +(In Python 3.12 and earlier, it was not possible to provide ``locals`` +to ``eval()`` or ``exec()`` without also providing ``globals`` as these +were previously positional-only arguments. Independently of this +PEP, Python 3.13 updated these builtins to accept keyword arguments) + C API changes ------------- From ad391ea4fefe129400dce541a45c1ac30c97e047 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 9 Jul 2024 18:38:38 +1000 Subject: [PATCH 30/36] Fix syntax, update implementation note --- peps/pep-0667.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 68709458eb1..1f0db7f02d3 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -52,8 +52,8 @@ rationale behind ``locals()`` being updated to return independent snapshots in :term:`optimized scopes ` wasn't clear, as it had been inherited from the original :pep:`558` discussions rather than being independently covered in this PEP. The PEP text has been updated to better cover this change, with additional updates -to the Backwards Compatibility section to cover the impact on code execution APIs like -``exec()`` and ``eval()`` that default to executing code in the ``locals()`` namespace. +to the Specification and Backwards Compatibility sections to cover the impact on code +execution APIs that default to executing code in the ``locals()`` namespace. .. _pep-667-motivation: @@ -285,10 +285,10 @@ and also implement the following mutable mapping operations: * using assignment to add new key/value pairs * using assignment to update the value associated with a key -* conditional assignment via the `setdefault()` method -* bulk updates via the `update()` method +* conditional assignment via the ``setdefault()`` method +* bulk updates via the ``update()`` method -Removing keys with `del` statements or the `pop()` method is NOT supported. +Removing keys with ``del`` statements or the ``pop()`` method is NOT supported. All writes to the ``f_locals`` mapping will be immediately visible in the underlying variables. All changes to the underlying variables @@ -393,7 +393,7 @@ can be defined as follows:: the_locals = the_globals return _eval(expression, the_globals, the_locals) -The argument handling for ``exec()`` is similarly updated. +The specified argument handling for ``exec()`` is similarly updated. (In Python 3.12 and earlier, it was not possible to provide ``locals`` to ``eval()`` or ``exec()`` without also providing ``globals`` as these From 2269f659cdb1e418f2d98933ac6ce6d886c91b66 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 9 Jul 2024 19:32:53 +1000 Subject: [PATCH 31/36] Another syntax fix --- peps/pep-0667.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 1f0db7f02d3..93bb0ec3fb4 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -368,7 +368,7 @@ implicitly used in the ``exec()`` call. The ``eval()`` and ``exec()`` builtins '''''''''''''''''''''''''''''''''''''' -Because this PEP changes the behavior of ``locals()`, the +Because this PEP changes the behavior of ``locals()``, the behavior of ``eval()`` and ``exec()`` also changes. Assuming a function ``_eval()`` which performs the job of From 15399411c9a94b09e1a77dcec09cd27d8c36952c Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 9 Jul 2024 19:39:49 +1000 Subject: [PATCH 32/36] Make snapshot code consistent --- peps/pep-0667.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index 93bb0ec3fb4..ac05d4283c6 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -387,7 +387,7 @@ can be defined as follows:: the_locals = _calling_frame.f_locals if isinstance(the_locals, FrameProxyType): # Align with locals() builtin in optimized frame - the_locals = the_locals.copy() + the_locals = dict(the_locals) elif the_locals is None: # Globals but no locals -> use same namespace for both the_locals = the_globals From 0c5aea9b9b1fbc80d73c07348e5dd2e50e688be2 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 16 Jul 2024 13:22:51 +1000 Subject: [PATCH 33/36] Use 3.13 signature for eval spec --- peps/pep-0667.rst | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index ac05d4283c6..d62157c87f3 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -377,21 +377,21 @@ can be defined as follows:: FrameProxyType = type((lambda: sys._getframe().f_locals)()) - def eval(expression, the_globals=None, the_locals=None): - if the_globals is None: + def eval(expression, /, globals=None, locals=None): + if globals is None: # No globals -> use calling frame's globals _calling_frame = sys._getframe(1) - the_globals = _calling_frame.f_globals - if the_locals is None: + globals = _calling_frame.f_globals + if locals is None: # No globals or locals -> use calling frame's locals - the_locals = _calling_frame.f_locals - if isinstance(the_locals, FrameProxyType): + locals = _calling_frame.f_locals + if isinstance(locals, FrameProxyType): # Align with locals() builtin in optimized frame - the_locals = dict(the_locals) - elif the_locals is None: + locals = dict(locals) + elif locals is None: # Globals but no locals -> use same namespace for both - the_locals = the_globals - return _eval(expression, the_globals, the_locals) + locals = globals + return _eval(expression, globals, locals) The specified argument handling for ``exec()`` is similarly updated. From 8abca041f0e326643316bb9ab6c0c4670da62d3c Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Sun, 28 Jul 2024 02:05:58 +1000 Subject: [PATCH 34/36] Delay `PyEval_GetLocals` hard deprecation to 3.14 --- peps/pep-0667.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index d62157c87f3..efcd76000b8 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -684,8 +684,9 @@ While this technically leaves the semantics of the function unchanged, it no lon extra dict entries to be made visible to users of the other APIs, as those APIs are no longer accessing the same underlying cache dictionary. -Accordingly, the function will be marked as deprecated, with a target removal date of -Python 3.15 (two releases after Python 3.13), and alternatives recommended as described below. +Accordingly, the function will be marked as soft-deprecated in Python 3.13, and subsequently +hard-deprecated in Python 3.14, with a target removal date of Python 3.16 (two releases +after Python 3.14). Alternatives wil be recommended as described below. When ``PyEval_GetLocals()`` is being used as an equivalent to the Python ``locals()`` builtin, ``PyEval_GetFrameLocals()`` should be used instead. From d3bff7bd889fac9ad409ba880a4584dbe8dde9e6 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Tue, 30 Jul 2024 22:23:06 +1000 Subject: [PATCH 35/36] Apply suggestions from code review Co-authored-by: Petr Viktorin --- peps/pep-0667.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index efcd76000b8..ae1c320780c 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -43,9 +43,7 @@ discrepancy was identified while implementing the PEP, and `resolved by the Steering Council `__ in favour of retaining the Python 3.12 behaviour of returning a dictionary snapshot cached on the frame instance. - -To avoid confusion when following the reference link from the Python 3.13 What's New -documentation, the PEP text has been updated accordingly. +The PEP text has been updated accordingly. During the discussions of the C API clarification, it also became apparent that the rationale behind ``locals()`` being updated to return independent snapshots in @@ -288,7 +286,8 @@ and also implement the following mutable mapping operations: * conditional assignment via the ``setdefault()`` method * bulk updates via the ``update()`` method -Removing keys with ``del`` statements or the ``pop()`` method is NOT supported. +Removing keys with ``del``, ``pop()`` or ``clear()`` is NOT supported. +Views of different frames compare unequal even if they have the same contents. All writes to the ``f_locals`` mapping will be immediately visible in the underlying variables. All changes to the underlying variables From 1d4447d2a9cc63449aebd5791f27577fd9c0eba6 Mon Sep 17 00:00:00 2001 From: Alyssa Coghlan Date: Mon, 5 Aug 2024 13:38:40 +1000 Subject: [PATCH 36/36] Move locals rationale details to PEP 558 --- peps/pep-0558.rst | 284 ++++++++++++++++++++++++++++++++++++++++++---- peps/pep-0667.rst | 239 +++++--------------------------------- 2 files changed, 294 insertions(+), 229 deletions(-) diff --git a/peps/pep-0558.rst b/peps/pep-0558.rst index 5eb32ef68af..16219f76094 100644 --- a/peps/pep-0558.rst +++ b/peps/pep-0558.rst @@ -34,6 +34,12 @@ are outweighed by the availability of a viable reference implementation. Accordingly, this PEP has been withdrawn in favour of proceeding with :pep:`667`. +Note: while implementing :pep:`667` it became apparent that the rationale for and impact +of ``locals()`` being updated to return independent snapshots in +:term:`optimized scopes ` was not entirely clear in either PEP. +The Motivation and Rationale sections in this PEP have been updated accordingly (since those +aspects are equally applicable to the accepted :pep:`667`). + Abstract ======== @@ -64,6 +70,7 @@ Python C API/ABI: It also proposes the addition of several supporting functions and type definitions to the CPython C API. +.. _pep-558-motivation: Motivation ========== @@ -89,6 +96,32 @@ independent snapshot of the function locals and closure variables on each call, rather than continuing to return the semi-dynamic intermittently updated shared copy that it has historically returned in CPython. +Specifically, the proposal in this PEP eliminates the historical behaviour where +adding a new local variable can change the behaviour of code executed with +``exec()`` in function scopes, even if that code runs *before* the local variable +is defined. + +For example:: + + def f(): + exec("x = 1") + print(locals().get("x")) + f() + +prints ``1``, but:: + + def f(): + exec("x = 1") + print(locals().get("x")) + x = 0 + f() + +prints ``None`` (the default value from the ``.get()`` call). + +With this PEP both examples would print ``None``, as the call to +``exec()`` and the subsequent call to ``locals()`` would use +independent dictionary snapshots of the local variables rather +than using the same shared dictionary cached on the frame object. Proposal ======== @@ -797,25 +830,6 @@ frame machinery will allow rebinding of local and nonlocal variable references in a way that is hidden from static analysis. -Retaining the internal frame value cache ----------------------------------------- - -Retaining the internal frame value cache results in some visible quirks when -frame proxy instances are kept around and re-used after name binding and -unbinding operations have been executed on the frame. - -The primary reason for retaining the frame value cache is to maintain backwards -compatibility with the ``PyEval_GetLocals()`` API. That API returns a borrowed -reference, so it must refer to persistent state stored on the frame object. -Storing a fast locals proxy object on the frame creates a problematic reference -cycle, so the cleanest option is to instead continue to return a frame value -cache, just as this function has done since optimised frames were first -introduced. - -With the frame value cache being kept around anyway, it then further made sense -to rely on it to simplify the fast locals proxy mapping implementation. - - What happens with the default args for ``eval()`` and ``exec()``? ----------------------------------------------------------------- @@ -840,9 +854,241 @@ namespace on each iteration). to make a list from the keys). +.. _pep-558-exec-eval-impact: + +Additional considerations for ``eval()`` and ``exec()`` in optimized scopes +--------------------------------------------------------------------------- + +Note: while implementing :pep:`667`, it was noted that neither that PEP nor this one +clearly explained the impact the ``locals()`` changes would have on code execution APIs +like ``exec()`` and ``eval()``. This section was added to this PEP's rationale to better +describe the impact and explain the intended benefits of the change. + +When ``exec()`` was converted from a statement to a builtin function +in Python 3.0 (part of the core language changes in :pep:`3100`), the +associated implicit call to ``PyFrame_LocalsToFast()`` was removed, so +it typically appears as if attempts to write to local variables with +``exec()`` in optimized frames are ignored:: + + >>> def f(): + ... x = 0 + ... exec("x = 1") + ... print(x) + ... print(locals()["x"]) + ... + >>> f() + 0 + 0 + +In truth, the writes aren't being ignored, they just aren't +being copied from the dictionary cache back to the optimized local +variable array. The changes to the dictionary are then overwritten +the next time the dictionary cache is refreshed from the array:: + + >>> def f(): + ... x = 0 + ... locals_cache = locals() + ... exec("x = 1") + ... print(x) + ... print(locals_cache["x"]) + ... print(locals()["x"]) + ... + >>> f() + 0 + 1 + 0 + +.. _pep-558-ctypes-example: + +The behaviour becomes even stranger if a tracing function +or another piece of code invokes ``PyFrame_LocalsToFast()`` before +the cache is next refreshed. In those cases the change *is* +written back to the optimized local variable array:: + + >>> from sys import _getframe + >>> from ctypes import pythonapi, py_object, c_int + >>> _locals_to_fast = pythonapi.PyFrame_LocalsToFast + >>> _locals_to_fast.argtypes = [py_object, c_int] + >>> def f(): + ... _frame = _getframe() + ... _f_locals = _frame.f_locals + ... x = 0 + ... exec("x = 1") + ... _locals_to_fast(_frame, 0) + ... print(x) + ... print(locals()["x"]) + ... print(_f_locals["x"]) + ... + >>> f() + 1 + 1 + 1 + +This situation was more common in Python 3.10 and earlier +versions, as merely installing a tracing function was enough +to trigger implicit calls to ``PyFrame_LocalsToFast()`` after +every line of Python code. However, it can still happen in Python +3.11+ depending on exactly which tracing functions are active +(e.g. interactive debuggers intentionally do this so that changes +made at the debugging prompt are visible when code execution +resumes). + +All of the above comments in relation to ``exec()`` apply to +*any* attempt to mutate the result of ``locals()`` in optimized +scopes, and are the main reason that the ``locals()`` builtin +docs contain this caveat: + + Note: The contents of this dictionary should not be modified; + changes may not affect the values of local and free variables + used by the interpreter. + +While the exact wording in the library reference is not entirely explicit, +both ``exec()`` and ``eval()`` have long used the results of calling +``globals()`` and ``locals()`` in the calling Python frame as their default +execution namespace. + +This was historically also equivalent to using the calling frame's +``frame.f_globals`` and ``frame.f_locals`` attributes, but this PEP maps +the default namespace arguments for ``exec()`` and ``eval()`` to +``globals()`` and ``locals()`` in the calling frame in order to preserve +the property of defaulting to ignoring attempted writes to the local +namespace in optimized scopes. + +This poses a potential compatibility issue for some code, as with the +previous implementation that returns the same dict when ``locals()`` is called +multiple times in function scope, the following code usually worked due to +the implicitly shared local variable namespace:: + + def f(): + exec('a = 0') # equivalent to exec('a = 0', globals(), locals()) + exec('print(a)') # equivalent to exec('print(a)', globals(), locals()) + print(locals()) # {'a': 0} + # However, print(a) will not work here + f() + +With ``locals()`` in an optimised scope returning the same shared dict for each call, +it was possible to store extra "fake locals" in that dict. While these aren't real +locals known by the compiler (so they can't be printed with code like ``print(a)``), +they can still be accessed via ``locals()`` and shared between multiple ``exec()`` +calls in the same function scope. Furthermore, because they're *not* real locals, +they don't get implicitly updated or removed when the shared cache is refreshed +from the local variable storage array. + +When the code in ``exec()`` tries to write to an existing local variable, the +runtime behaviour gets harder to predict:: + + def f(): + a = None + exec('a = 0') # equivalent to exec('a = 0', globals(), locals()) + exec('print(a)') # equivalent to exec('print(a)', globals(), locals()) + print(locals()) # {'a': None} + f() + +``print(a)`` will print ``None`` because the implicit ``locals()`` call in +``exec()`` refreshes the cached dict with the actual values on the frame. +This means that, unlike the "fake" locals created by writing back to ``locals()`` +(including via previous calls to ``exec()``), the real locals known by the +compiler can't easily be modified by ``exec()`` (it can be done, but it requires +both retrieving the ``frame.f_locals`` attribute to enable writes back to the frame, +and then invoking ``PyFrame_LocalsToFast()``, as :ref:`shown ` +using ``ctypes`` above). + +As noted in the :ref:`pep-558-motivation` section, this confusing side effect +happens even if the local variable is only defined *after* the ``exec()`` calls:: + + >>> def f(): + ... exec("a = 0") + ... exec("print('a' in locals())") # Printing 'a' directly won't work + ... print(locals()) + ... a = None + ... print(locals()) + ... + >>> f() + False + {} + {'a': None} + +Because ``a`` is a real local variable that is not currently bound to a value, it +gets explicitly removed from the dictionary returned by ``locals()`` whenever +``locals()`` is called prior to the ``a = None`` line. This removal is intentional, +as it allows the contents of ``locals()`` to be updated correctly in optimized +scopes when ``del`` statements are used to delete previously bound local variables. + +As noted in the ``ctypes`` :ref:`example `, the above behavioural +description may be invalidated if the CPython ``PyFrame_LocalsToFast()`` API gets invoked +while the frame is still running. In that case, the changes to ``a`` *might* become visible +to the running code, depending on exactly when that API is called (and whether the frame +has been primed for locals modification by accessing the ``frame.f_locals`` attribute). + +As described above, two options were considered to replace this confusing behaviour: + +* make ``locals()`` return write-through proxy instances (similar + to ``frame.f_locals``) +* make ``locals()`` return genuinely independent snapshots so that + attempts to change the values of local variables via ``exec()`` + would be *consistently* ignored without any of the caveats + noted above. + +The PEP chooses the second option for the following reasons: + +* returning independent snapshots in optimized scopes preserves + the Python 3.0 change to ``exec()`` that resulted in attempts + to mutate local variables via ``exec()`` being ignored in most + cases +* the distinction between "``locals()`` gives an instantaneous + snapshot of the local variables in optimized scopes, and + read/write access in other scopes" and "``frame.f_locals`` + gives read/write access to the local variables in all scopes, + including optimized scopes" allows the intent of a piece of + code to be clearer than it would be if both APIs granted + full read/write access in optimized scopes, even when write + access wasn't needed or desired +* in addition to improving clarity for human readers, ensuring + that name rebinding in optimized scopes remains lexically + visible in the code (as long as the frame introspection APIs + are not accessed) allows compilers and interpreters to apply + related performance optimizations more consistently +* only Python implementations that support the optional frame + introspection APIs will need to provide the new write-through + proxy support for optimized frames + +With the semantic changes to ``locals()`` in this PEP, it becomes much easier to explain +the behavior of ``exec()`` and ``eval()``: in optimized scopes, they will *never* implicitly +affect local variables; in other scopes, they will *always* implicitly affect local +variables. In optimized scopes, any implicit assignment to the local variables will be +discarded when the code execution API returns, since a fresh copy of the local variables +is used on each invocation. + + +Retaining the internal frame value cache +---------------------------------------- + +Retaining the internal frame value cache results in some visible quirks when +frame proxy instances are kept around and re-used after name binding and +unbinding operations have been executed on the frame. + +The primary reason for retaining the frame value cache is to maintain backwards +compatibility with the ``PyEval_GetLocals()`` API. That API returns a borrowed +reference, so it must refer to persistent state stored on the frame object. +Storing a fast locals proxy object on the frame creates a problematic reference +cycle, so the cleanest option is to instead continue to return a frame value +cache, just as this function has done since optimised frames were first +introduced. + +With the frame value cache being kept around anyway, it then further made sense +to rely on it to simplify the fast locals proxy mapping implementation. + +Note: the fact :pep:`667` *doesn't* use the internal frame value cache as part of the +write-through proxy implementation is the key Python level difference between the two PEPs. + + Changing the frame API semantics in regular operation ----------------------------------------------------- +Note: when this PEP was first written, it predated the Python 3.11 change to drop the +implicit writeback of the frame local variables whenever a tracing function was installed, +so making that change was included as part of the proposal. + Earlier versions of this PEP proposed having the semantics of the frame ``f_locals`` attribute depend on whether or not a tracing hook was currently installed - only providing the write-through proxy behaviour when a tracing hook diff --git a/peps/pep-0667.rst b/peps/pep-0667.rst index ae1c320780c..bde7d19dd92 100644 --- a/peps/pep-0667.rst +++ b/peps/pep-0667.rst @@ -51,7 +51,8 @@ rationale behind ``locals()`` being updated to return independent snapshots in from the original :pep:`558` discussions rather than being independently covered in this PEP. The PEP text has been updated to better cover this change, with additional updates to the Specification and Backwards Compatibility sections to cover the impact on code -execution APIs that default to executing code in the ``locals()`` namespace. +execution APIs that default to executing code in the ``locals()`` namespace. Additional +motivation and rationale details have also been added to :pep:`558`. .. _pep-667-motivation: @@ -89,32 +90,9 @@ the frame rather than to a cached dictionary snapshot. There are no compensating advantages for the Python 3.12 behavior; it is unreliable and slow. -The ``locals()`` builtin has its own undesirable behaviours, where -adding a new local variable can change the behaviour of code -executed with ``exec()`` in function scopes, even if that code -runs *before* the local variable is defined. +The ``locals()`` builtin has its own undesirable behaviours. Refer to :pep:`558` +for additional details on those concerns. -For example:: - - def f(): - exec("x = 1") - print(locals().get("x")) - f() - -prints ``1``, but:: - - def f(): - exec("x = 1") - print(locals().get("x")) - x = 0 - f() - -prints ``None`` (the default value from the ``.get()`` call). - -With this PEP both examples would print ``None``, as the call to -``exec()`` and the subsequent call to ``locals()`` would use -independent dictionary snapshots of the local variables rather -than using the same shared dictionary cached on the frame object. .. _pep-667-rationale: @@ -147,115 +125,29 @@ sync with the frame because it is a view of it, not a copy of it. Making the ``locals()`` builtin return independent snapshots ------------------------------------------------------------ -When ``exec`` was converted from a statement to a builtin function -in Python 3.0 (part of the core language changes in :pep:`3100`), the -associated implicit call to ``PyFrame_LocalsToFast()`` was removed, so -it typically appears as if attempts to write to local variables with -``exec`` in optimized frames are ignored:: - - >>> def f(): - ... x = 0 - ... exec("x = 1") - ... print(x) - ... print(locals()["x"]) - ... - >>> f() - 0 - 0 - -In truth, the writes aren't being ignored, they just aren't -being copied from the dictionary cache back to the optimized local -variable array. The changes to the dictionary are then overwritten -the next time the dictionary cache is refreshed from the array:: - - >>> def f(): - ... x = 0 - ... locals_cache = locals() - ... exec("x = 1") - ... print(x) - ... print(locals_cache["x"]) - ... print(locals()["x"]) - ... - >>> f() - 0 - 1 - 0 - -.. _pep-667-ctypes-example: - -The behaviour becomes even stranger if a tracing function -or another piece of code invokes ``PyFrame_LocalsToFast()`` before -the cache is next refreshed. In those cases the change *is* -written back to the optimized local variable array:: - - >>> from sys import _getframe - >>> from ctypes import pythonapi, py_object, c_int - >>> _locals_to_fast = pythonapi.PyFrame_LocalsToFast - >>> _locals_to_fast.argtypes = [py_object, c_int] - >>> def f(): - ... _frame = _getframe() - ... _f_locals = _frame.f_locals - ... x = 0 - ... exec("x = 1") - ... _locals_to_fast(_frame, 0) - ... print(x) - ... print(locals()["x"]) - ... print(_f_locals["x"]) - ... - >>> f() - 1 - 1 - 1 - -This situation was more common in Python 3.10 and earlier -versions, as merely installing a tracing function was enough -to trigger implicit calls to ``PyFrame_LocalsToFast()`` after -every line of Python code. However, it can still happen in Python -3.11+ depending on exactly which tracing functions are active -(e.g. interactive debuggers intentionally do this so that changes -made at the debugging prompt are visible when code execution -resumes). - -All of the above comments in relation to ``exec()`` apply to -*any* attempt to mutate the result of ``locals()`` in optimized -scopes, and are the main reason that the ``locals()`` builtin -docs contain this caveat: - - Note: The contents of this dictionary should not be modified; - changes may not affect the values of local and free variables - used by the interpreter. - -Two options were considered to replace this confusing behaviour: +:pep:`558` considered three potential options for standardising the behavior of the +``locals()`` builtin in optimized scopes: +* retain the historical behaviour of having each call to ``locals()`` on a given frame + update a single shared snapshot of the local variables * make ``locals()`` return write-through proxy instances (similar to ``frame.f_locals``) * make ``locals()`` return genuinely independent snapshots so that attempts to change the values of local variables via ``exec()`` - would be *consistently* ignored without any of the caveats - noted above. - -The PEP chooses the second option for the following reasons: - -* returning independent snapshots in optimized scopes preserves - the Python 3.0 change to ``exec()`` that resulted in attempts - to mutate local variables via ``exec()`` being ignored in most - cases -* the distinction between "``locals()`` gives an instantaneous - snapshot of the local variables in optimized scopes, and - read/write access in other scopes" and "``frame.f_locals`` - gives read/write access to the local variables in all scopes, - including optimized scopes" allows the intent of a piece of - code to be clearer than it would be if both APIs granted - full read/write access in optimized scopes, even when write - access wasn't needed or desired -* in addition to improving clarity for human readers, ensuring - that name rebinding in optimized scopes remains lexically - visible in the code (as long as the frame introspection APIs - are not accessed) allows compilers and interpreters to apply - related performance optimizations more consistently -* only Python implementations that support the optional frame - introspection APIs will need to provide the new write-through - proxy support for optimized frames + would be *consistently* ignored rather than being accepted in some circumstances + +The last option was chosen as the one which could most easily be explained in the +language reference, and memorised by users: + +* the ``locals()`` builtin gives an instantaneous snapshot of the local variables in + optimized scopes, and read/write access in other scopes; and +* ``frame.f_locals`` gives read/write access to the local variables in all scopes, + including optimized scopes + +This approach allows the intent of a piece of code to be clearer than it would be if both +APIs granted full read/write access in optimized scopes, even when write access wasn't +needed or desired. For additional details on this design decision, refer to :pep:`558`, +especially the :ref:`pep-558-motivation` section and :ref:`pep-558-exec-eval-impact`. This approach is not without its drawbacks, which are covered in the Backwards Compatibility section below. @@ -530,26 +422,9 @@ Even though this PEP does not modify ``exec()`` or ``eval()`` directly, the semantic change to ``locals()`` impacts the behavior of ``exec()`` and ``eval()`` as they default to running code in the calling namespace. -While the exact wording in the library reference is not entirely explicit, -both ``exec()`` and ``eval()`` have long used the results of calling -``globals()`` and ``locals()`` in the calling Python frame as their default -execution namespace. - -This was historically also equivalent to using the calling frame's -``frame.f_globals`` and ``frame.f_locals`` attributes, but this PEP maps -the default namespace arguments for ``exec()`` and ``eval()`` to -``globals()`` and ``locals()`` in the calling frame in order to preserve -the property of defaulting to ignoring attempted writes to the local -namespace in optimized scopes. - -However, as noted above for ``locals()``, this change has an additional -effect: each ``exec()`` call in an optimized scope will now run in a -*different* implicit namespace rather than a shared one. Furthermore, -separately calling ``locals()`` will also return a different namespace. - This poses a potential compatibility issue for some code, as with the previous implementation that returns the same dict when ``locals()`` is called -multiple times in function scope, the following code usually works due to +multiple times in function scope, the following code usually worked due to the implicitly shared local variable namespace:: def f(): @@ -559,69 +434,13 @@ the implicitly shared local variable namespace:: # However, print(a) will not work here f() -With ``locals()`` in an optimised scope returning the same shared dict for each call, -it is possible to store extra "fake locals" in that dict. While these aren't real -locals known by the compiler (so they can't be printed with code like ``print(a)``), -they can still be accessed via ``locals()`` and shared between multiple ``exec()`` -calls in the same function scope. Furthermore, because they're *not* real locals, -they don't get implicitly updated or removed when the shared cache is refreshed -from the local variable storage array. - -When the code in ``exec()`` tries to write to an existing local variable, the -runtime behaviour gets harder to predict:: - - def f(): - a = None - exec('a = 0') # equivalent to exec('a = 0', globals(), locals()) - exec('print(a)') # equivalent to exec('print(a)', globals(), locals()) - print(locals()) # {'a': None} - f() +With the semantic changes to ``locals()`` in this PEP, the ``exec('print(a)')'`` call +will fail with ``NameError``, and ``print(locals())`` will report an empty dictionary, as +each line will be using its own distinct snapshot of the local variables rather than +implicitly sharing a single cached snapshot stored on the frame object. -``print(a)`` will print ``None`` because the implicit ``locals()`` call in -``exec()`` refreshes the cached dict with the actual values on the frame. -This means that, unlike the "fake" locals created by writing back to ``locals()`` -(including via previous calls to ``exec()``), the real locals known by the -compiler can't easily be modified by ``exec()`` (it can be done, but it requires -both retrieving the ``frame.f_locals`` attribute to enable writes back to the frame, -and then invoking ``PyFrame_LocalsToFast()``, as :ref:`shown ` -using ``ctypes`` in the :ref:`pep-667-rationale` section above). - -As noted in the :ref:`pep-667-motivation` section, this confusing side effect -happens even if the local variable is only defined *after* the ``exec()`` calls:: - - >>> def f(): - ... exec("a = 0") - ... exec("print('a' in locals())") # Printing 'a' directly won't work - ... print(locals()) - ... a = None - ... print(locals()) - ... - >>> f() - False - {} - {'a': None} - -Because ``a`` is a real local variable that is not currently bound to a value, it -gets explicitly removed from the dictionary returned by ``locals()`` whenever -``locals()`` is called prior to the ``a = None`` line. This removal is intentional, -as it allows the contents of ``locals()`` to be updated correctly in optimized -scopes when ``del`` statements are used to delete previously bound local variables. - -As noted in the :ref:`pep-667-rationale` section, the above behavioural description may be -invalidated if the CPython ``PyFrame_LocalsToFast()`` API gets invoked while the frame -is still running. In that case, the changes to ``a`` *might* become visible to the -running code, depending on exactly when that API is called (and whether the frame -has been primed for locals modification by accessing the ``frame.f_locals`` attribute). - -With the semantic changes to ``locals()`` in this PEP, it becomes much easier to explain the -behavior of ``exec()`` and ``eval()``: in optimized scopes, they will *never* implicitly affect -local variables; in other scopes, they will *always* implicitly affect local variables. -In optimized scopes, any implicit assignment to the local variables will be discarded when -the code execution API returns, since a fresh copy of the local variables is used on each -invocation. - -A shared namespace across ``exec()`` calls can still be obtained by using explicit namespaces -rather than relying on the previously implicitly shared frame namespace:: +A shared namespace across ``exec()`` calls can still be obtained by using explicit +namespaces rather than relying on the previously implicitly shared frame namespace:: def f(): ns = {}