turbo: espressif boards loader enable native .mpy - #11392
Open
mikeysklar wants to merge 9 commits into
Open
mikeysklar wants to merge 9 commits into
mikeysklar wants to merge 9 commits into
Conversation
stop_mp() runs the heap finalisers in gc_deinit(). A port that keeps memory outside the VM heap which heap objects point into, such as native code copied into executable RAM, can only free it after that point. reset_port() runs before stop_mp(), which is too early. The weak default does nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The default MP_PLAT_ALLOC_EXEC places native code on the GC heap, which on these chips is DRAM or PSRAM and cannot be fetched as instructions. Define MP_PLAT_COMMIT_EXEC, used by py/persistentcode.c for native .mpy files, to copy the finished code into a heap_caps_malloc(MALLOC_CAP_EXEC) block, applying the relocations against the final address first. Blocks are kept on a list and freed together from port_gc_deinit(), once the heap finalisers have run. Modelled on esp_native_code_commit() in MicroPython's ports/esp32/main.c, including the ESP32-S2 esp_ptr_executable() workaround for espressif/esp-idf#14835. The copy is word-wise because Xtensa executable RAM is not byte-addressable. With CONFIG_ESP_SYSTEM_MEMPROT=y, the current default, the heap component disables CONFIG_HEAP_HAS_EXEC_HEAP and MALLOC_CAP_EXEC allocations return NULL, so loading native code fails with MemoryError rather than a fault. The sdkconfig side is the next commit. Only compiled when MICROPY_PERSISTENT_CODE_LOAD_NATIVE is set. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Add esp-idf-config/sdkconfig-native.defaults with CONFIG_ESP_SYSTEM_MEMPROT=n and include it in SDKCONFIG_DEFAULTS only when CIRCUITPY_LOAD_NATIVE=1 or CIRCUITPY_ENABLE_MPY_NATIVE=1. Nothing changes for builds without either. On ESP-IDF 6 the PMS (S2, S3, C3) and PMP (C6, C5) memory protection marks RAM non-executable and sets CONFIG_HEAP_HAS_EXEC_HEAP=n, so MALLOC_CAP_EXEC allocations always fail and native code cannot run. Turning it off is a security trade-off: a write-anywhere bug becomes an execute-anywhere bug. It is opt-in per build for that reason. MicroPython ships every esp32 board with CONFIG_ESP_SYSTEM_MEMPROT_FEATURE=n (ports/esp32/boards/sdkconfig.base). check-sdkconfig.py refuses a build that has either flag on while CONFIG_ESP_SYSTEM_MEMPROT is still set, so a board sdkconfig cannot silently re-enable it and produce firmware that faults on import. tools/update_sdkconfig.py loads the same defaults file for boards with either flag, so regenerating a board sdkconfig does not copy the setting into it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… loading MICROPY_EMIT_NATIVE_PRELUDE_SEPARATE_FROM_MACHINE_CODE is keyed on MICROPY_EMIT_XTENSAWIN alone. A build that loads native .mpy without the emitter on the ESP32-S2/S3 left it at 0, so persistentcode.c read the prelude of every loaded @micropython.native function byte-wise out of IRAM. Windowed Xtensa cannot do that: the first call of a native (non-viper) function took a LoadStoreError and the board reset. Viper functions carry no prelude, which is why viper alone passed. Key the define on the compiler target as well. Upstream has the same expression and does not hit this because its esp32 port always builds the emitter. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Turn on CIRCUITPY_LOAD_NATIVE for the two Xtensa boards. Host-compiled xtensawin viper mandelbrot (160x120, 64 iterations): Metro ESP32-S3 172 ms, Metro ESP32-S2 206 ms, against 4,867 ms and 7,520 ms float bytecode. @micropython.viper from source raises SyntaxError and an armv7emsp .mpy raises ValueError. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Turn on CIRCUITPY_LOAD_NATIVE for the RISC-V devkit. Host-compiled rv32imc viper mandelbrot (160x120, 64 iterations): 159 ms against 7,535 ms float bytecode and 3,569 ms fixed-point bytecode. @micropython.viper from source raises SyntaxError and an armv7emsp .mpy raises ValueError. Same code path as the Xtensa boards, no port change needed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Turn on CIRCUITPY_LOAD_NATIVE for the two Adafruit RISC-V boards. Same rv32imc loader path as the ESP32-C5-DevKitC-1-N8R8. Both build; neither was run on hardware. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Turn CIRCUITPY_LOAD_NATIVE on in mpconfigport.mk for every espressif chip and drop the five per-board lines. Boards without the flash space opt out in their own mpconfigboard.mk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The loader is now on by default in mpconfigport.mk, so a board file no longer names CIRCUITPY_LOAD_NATIVE. Default it to on here too and let a board's "= 0" turn it off, so the native sdkconfig defaults stay out of the board sdkconfig files. CIRCUITPY_ENABLE_MPY_NATIVE = 1 still counts on its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mikeysklar
force-pushed
the
espressif-load-native
branch
from
September 15, 2026 03:11
8bd9742 to
540f614
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Enables the turbo loader (
CIRCUITPY_LOAD_NATIVE) on all espressif boards. Follows #11332, #11355, #11356 and #11360.update_sdkconfig.pyfollows the loader defaultHow I tested it
Hardware ran
10.4.0-alpha.1-22-gcd6e5ec116on an Ubuntu host..mpyvs float bytecode@micropython.viperin sourceSyntaxError.mpyValueError: incompatible .mpy archAI assistance
Claude Code was used.