Skip to content

Add CPU overload detection and failsafe#826

Merged
dpwe merged 3 commits into
mainfrom
overload-failsafe
Jul 8, 2026
Merged

Add CPU overload detection and failsafe#826
dpwe merged 3 commits into
mainfrom
overload-failsafe

Conversation

@bwhitman

@bwhitman bwhitman commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Fixes #791.

It's easy to overwhelm an MCU with AMY (e.g. dpwe's Eno sketch on AMYboard: 18 voices of DX7 + big reverb + echo + chorus with 4-12 bar holds). Today that wedges the board: amy_fb_task runs at ESP_TASK_PRIO_MAX - 1 on core 1, and once a block takes longer than real time to render, the i2s DMA queue is permanently empty, i2s_channel_write never blocks, and the task never yields — starving tulip_mp_task (same core), which is what services USB CDC and MIDI. Hence the 96.86% amy_fb_task / 1% tulip_mp_task in tulip.cpu() right before the board dies.

This PR makes AMY detect the overload, stop itself in an orderly way, and tell the host:

Detection

Platform render loops call a new amy_overload_check(busy_us, period_us) once per block. On ESP32 (multithread) we measure time spent working vs time spent blocked in the i2s DMA write (or input read / update sync) — on AMYboard the i2s is slave-mode off the external 512fs clock, so the blocked time is the ground-truth timebase, no budget constant needed. RP2040/RP2350, Teensy, and non-multithread ESP measure render time against the fixed block budget (AMY_BLOCK_US). The check keeps a smoothed load fraction in amy_global.render_load, readable any time via amy_get_render_load() (e.g. for a CPU meter in AMYboard Online, or early warning at 0.85).

Anti-wedge yield (ESP32)

Independent of the failsafe: when the i2s write didn't block (blocked_us < 150), the fill-buffer task does vTaskDelay(1). Audio is already breaking up at that point, but this keeps USB CDC / MIDI / MicroPython alive on the core, so the board degrades instead of wedging — you can still Ctrl-C.

Failsafe

When render_load sits at/above config.overload_threshold (default 0.98, set 0 to disable) for config.overload_ms (default 250), AMY:

  1. flushes the delta queue (which by then holds hundreds of pending note-offs that would re-wedge it),
  2. schedules RESET_ALL_NOTES | RESET_ALL_OSCS | RESET_SEQUENCER,
  3. plays "doot doot doot doot BLAWWW" — a descending A-minor arpeggio then a held A-minor saw chord, deliberately nothing like the startup bleep — so a headless user knows AMY stopped on purpose,
  4. calls the new config.amy_external_overload_hook(load) so hosts can react — Tulip/AMYboard can stop the running sketch and pop up "this sketch took too many resources, try again" in the web editor. (Hook is called from the render task; implementations should set a flag and return quickly.)

Then it keeps rendering with the load measure reset, so the user can immediately try again. If overload somehow persists after the reset, it re-trips (~every 1.4s).

Also

  • amy_get_us() is now compiled in all builds, not just AMY_DEBUG (it's two timer reads per block).
  • amy_default_config() now initializes the exec / update_file / reboot hooks, which were left as uninitialized stack garbage.
  • amy_grab_lock() / amy_release_lock() are now declared in amy.h.

Testing

  • All 103 WAV tests pass bit-exact (desktop never calls amy_overload_check, so no golden changes).
  • Offline harness against the new logic: healthy load (0.34) never trips; saturated load trips after ~111 blocks (~0.9s including EMA climb) with reset+bleep deltas queued and the hook called; a 120ms 3x-budget spike does not trip; threshold 0 disables.
  • Web (emscripten) build compiles clean. ESP32/pico/teensy paths ride on the arduino.yml CI matrix.
  • Not yet tested on real AMYboard hardware — that's the ask here: run the Eno sketch and watch for the bleep instead of the wedge.

The Tulip-side hook (stop sketch + popup) is a follow-up in shorepine/tulipcc.

🤖 Generated with Claude Code

Track per-block render load on MCU platforms and, when it stays pinned
at/above a threshold, reset the synth and play a descending bleep
instead of wedging the host.

- amy_overload_check(): platform render loops report busy vs wall time
  per block; a smoothed load estimate is kept in amy_global.render_load
  (readable via amy_get_render_load()). When it sits at/above
  config.overload_threshold (default 0.98, 0 disables) for
  config.overload_ms (default 250), the failsafe trips.
- amy_overload_failsafe(): flushes the delta queue, resets all
  notes/oscs/sequencer, schedules a descending bleep, and calls the new
  config.amy_external_overload_hook so hosts (e.g. AMYboard/Tulip) can
  stop the running sketch and notify the user.
- ESP32: measure busy vs blocked time in esp_fill_audio_buffer_task.
  When the i2s write stops blocking (past overload), vTaskDelay(1) so
  this max-priority task no longer starves USB CDC / MIDI / MicroPython
  on its core -- this alone prevents the board from wedging.
- RP2040/RP2350, Teensy, and non-multithread ESP measure render time
  against the fixed block budget (AMY_BLOCK_US).
- amy_get_us() is now available in all builds, not just AMY_DEBUG.
- Also initialize the exec/update_file/reboot hooks in
  amy_default_config(), which were left as stack garbage.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@bwhitman

bwhitman commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator Author

🎛️ AMYboard HW CI

Dispatched this PR's amy SHA to the tulipcc bench — it builds the AMYboard firmware with your change and runs it on the physical board (audio spectral-compared to a committed reference). Results post here in a few minutes.

bwhitman and others added 2 commits July 7, 2026 12:04
Per feedback: the previous descending bleep was too close to amy_bleep,
which Tulip uses as its startup sound. Now: doot doot doot doot BLAWWW
(A5-E5-C5-A4 sine doots, then a held A minor saw chord).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@bwhitman

bwhitman commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator Author

🎛️ HW CI (physical bench)

Built from this AMY PR, pinned into the tulipcc amy submodule.

AMYboard (USB-MIDI + AMY zP → audio; built-in tones + AMYboard World sketches acid/house/woodpiano over the SysEx control API): ✅ PASS — flashed this PR’s firmware; all checks matched the references.

Tulip (TULIP4_R11; serial-REPL audio + WiFi screenshot): ✅ PASS — flashed this PR’s firmware; all checks matched the references.

⬇️ Artifacts: recordings · screenshot · serial logs · run logs

Self-hosted bench. Audio spectral-compared to ref/hwci_basic.wav, the AMYboard World sketch refs (ref/{acid_generator,house_generator,woodpiano}.wav) + ref/tulip_basic.wav; Tulip screenshot pixel-compared to ref/tulip_screenshot.png. Both analog outs share one capture card, so the tests run sequentially.

Comment thread src/i2s.c
// When rendering keeps up, this task spends most of each block parked in the
// i2s DMA write (or the update-sync wait) above, which is when lower-priority
// tasks on this core get to run.
amy_overload_check(busy_us, busy_us + blocked_us);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not saying it's not right, but I don't understand why the denominator here is (time spent rendering + time spent waiting for I2S to unblock) instead of AMY_BLOCK_US (the total time available for each render block).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From Claude:
"n equilibrium they're nearly the same — busy + blocked ≈ one block period. Two reasons for this form: (1) this loop isn't always paced by the nominal rate — on AMYboard the i2s is slave-mode off the external 512fs clock, and in the !AMY_HAS_I2S case the pacing is the host pulling blocks via the notify sync, which needn't be real-time at all, so measuring against the actual pacing (busy+blocked) stays correct while AMY_BLOCK_US wouldn't; and (2) it saturates at exactly 1.0 at exactly the condition that wedges the board (the write stopped blocking → this task never yields), rather than depending on the constant. Where there's no blocking wait to measure (non-multithread ESP, pico, Teensy render paths) the denominator is AMY_BLOCK_US."

@dpwe dpwe merged commit 067c95c into main Jul 8, 2026
10 checks passed
@bwhitman

bwhitman commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator Author

⛓️ tulipcc integration PR opened

This merge was pinned into tulipcc for full-system CI: shorepine/tulipcc#1107

Test it there and merge that PR to move tulipcc onto this AMY.

pull Bot pushed a commit to manmuqingshan/tulipcc that referenced this pull request Jul 8, 2026
Pins the amy submodule to shorepine/amy#826 (CPU overload detection +
failsafe) so the PR firmware/webapp builds carry it, and adds
tulip.amy_render_load() -> float, AMY's smoothed 0..1 render load
(~1.0 = overloaded), for trying the new measure from Python.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Failsafe mode when CPU overload detected

2 participants