Add CPU overload detection and failsafe#826
Conversation
Track per-block render load on MCU platforms and, when it stays pinned at/above a threshold, reset the synth and play a descending bleep instead of wedging the host. - amy_overload_check(): platform render loops report busy vs wall time per block; a smoothed load estimate is kept in amy_global.render_load (readable via amy_get_render_load()). When it sits at/above config.overload_threshold (default 0.98, 0 disables) for config.overload_ms (default 250), the failsafe trips. - amy_overload_failsafe(): flushes the delta queue, resets all notes/oscs/sequencer, schedules a descending bleep, and calls the new config.amy_external_overload_hook so hosts (e.g. AMYboard/Tulip) can stop the running sketch and notify the user. - ESP32: measure busy vs blocked time in esp_fill_audio_buffer_task. When the i2s write stops blocking (past overload), vTaskDelay(1) so this max-priority task no longer starves USB CDC / MIDI / MicroPython on its core -- this alone prevents the board from wedging. - RP2040/RP2350, Teensy, and non-multithread ESP measure render time against the fixed block budget (AMY_BLOCK_US). - amy_get_us() is now available in all builds, not just AMY_DEBUG. - Also initialize the exec/update_file/reboot hooks in amy_default_config(), which were left as stack garbage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🎛️ AMYboard HW CIDispatched this PR's |
Per feedback: the previous descending bleep was too close to amy_bleep, which Tulip uses as its startup sound. Now: doot doot doot doot BLAWWW (A5-E5-C5-A4 sine doots, then a held A minor saw chord). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🎛️ HW CI (physical bench)Built from this AMY PR, pinned into the tulipcc AMYboard (USB-MIDI + AMY Tulip (TULIP4_R11; serial-REPL audio + WiFi screenshot): ✅ PASS — flashed this PR’s firmware; all checks matched the references. ⬇️ Artifacts: recordings · screenshot · serial logs · run logs Self-hosted bench. Audio spectral-compared to |
| // When rendering keeps up, this task spends most of each block parked in the | ||
| // i2s DMA write (or the update-sync wait) above, which is when lower-priority | ||
| // tasks on this core get to run. | ||
| amy_overload_check(busy_us, busy_us + blocked_us); |
There was a problem hiding this comment.
Not saying it's not right, but I don't understand why the denominator here is (time spent rendering + time spent waiting for I2S to unblock) instead of AMY_BLOCK_US (the total time available for each render block).
There was a problem hiding this comment.
From Claude:
"n equilibrium they're nearly the same — busy + blocked ≈ one block period. Two reasons for this form: (1) this loop isn't always paced by the nominal rate — on AMYboard the i2s is slave-mode off the external 512fs clock, and in the !AMY_HAS_I2S case the pacing is the host pulling blocks via the notify sync, which needn't be real-time at all, so measuring against the actual pacing (busy+blocked) stays correct while AMY_BLOCK_US wouldn't; and (2) it saturates at exactly 1.0 at exactly the condition that wedges the board (the write stopped blocking → this task never yields), rather than depending on the constant. Where there's no blocking wait to measure (non-multithread ESP, pico, Teensy render paths) the denominator is AMY_BLOCK_US."
⛓️ tulipcc integration PR openedThis merge was pinned into tulipcc for full-system CI: shorepine/tulipcc#1107 Test it there and merge that PR to move tulipcc onto this AMY. |
Pins the amy submodule to shorepine/amy#826 (CPU overload detection + failsafe) so the PR firmware/webapp builds carry it, and adds tulip.amy_render_load() -> float, AMY's smoothed 0..1 render load (~1.0 = overloaded), for trying the new measure from Python. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes #791.
It's easy to overwhelm an MCU with AMY (e.g. dpwe's Eno sketch on AMYboard: 18 voices of DX7 + big reverb + echo + chorus with 4-12 bar holds). Today that wedges the board:
amy_fb_taskruns atESP_TASK_PRIO_MAX - 1on core 1, and once a block takes longer than real time to render, the i2s DMA queue is permanently empty,i2s_channel_writenever blocks, and the task never yields — starvingtulip_mp_task(same core), which is what services USB CDC and MIDI. Hence the 96.86%amy_fb_task/ 1%tulip_mp_taskintulip.cpu()right before the board dies.This PR makes AMY detect the overload, stop itself in an orderly way, and tell the host:
Detection
Platform render loops call a new
amy_overload_check(busy_us, period_us)once per block. On ESP32 (multithread) we measure time spent working vs time spent blocked in the i2s DMA write (or input read / update sync) — on AMYboard the i2s is slave-mode off the external 512fs clock, so the blocked time is the ground-truth timebase, no budget constant needed. RP2040/RP2350, Teensy, and non-multithread ESP measure render time against the fixed block budget (AMY_BLOCK_US). The check keeps a smoothed load fraction inamy_global.render_load, readable any time viaamy_get_render_load()(e.g. for a CPU meter in AMYboard Online, or early warning at 0.85).Anti-wedge yield (ESP32)
Independent of the failsafe: when the i2s write didn't block (
blocked_us < 150), the fill-buffer task doesvTaskDelay(1). Audio is already breaking up at that point, but this keeps USB CDC / MIDI / MicroPython alive on the core, so the board degrades instead of wedging — you can still Ctrl-C.Failsafe
When
render_loadsits at/aboveconfig.overload_threshold(default 0.98, set 0 to disable) forconfig.overload_ms(default 250), AMY:RESET_ALL_NOTES | RESET_ALL_OSCS | RESET_SEQUENCER,config.amy_external_overload_hook(load)so hosts can react — Tulip/AMYboard can stop the running sketch and pop up "this sketch took too many resources, try again" in the web editor. (Hook is called from the render task; implementations should set a flag and return quickly.)Then it keeps rendering with the load measure reset, so the user can immediately try again. If overload somehow persists after the reset, it re-trips (~every 1.4s).
Also
amy_get_us()is now compiled in all builds, not justAMY_DEBUG(it's two timer reads per block).amy_default_config()now initializes theexec/update_file/reboothooks, which were left as uninitialized stack garbage.amy_grab_lock()/amy_release_lock()are now declared in amy.h.Testing
amy_overload_check, so no golden changes).The Tulip-side hook (stop sketch + popup) is a follow-up in shorepine/tulipcc.
🤖 Generated with Claude Code