fix(desktop): supervise and re-arm relay-mesh runtime#2823
Conversation
Relay-mesh agents went silent with no error if the embedded mesh runtime exited or wedged after launch: `mesh_llm_runtime` stayed `Some` while the local OpenAI ingress on :9337 was dead, so `ensure_relay_mesh_for_record` took the "already running" fast path and `wait_for_mesh_inference` just timed out against a dead endpoint (#2062). Add a fast liveness probe (`mesh_ingress_is_live`: single 3s `GET /v1/models`, the same call the issue used to confirm the ingress was down). When a runtime handle is present but the ingress is unreachable, drop the stale runtime (best-effort stop; never block re-arm on a wedged runtime — the zombie-guard motivation) and fall through to re-arm it via the normal bootstrap path. Resolves the reported symptom (option 1 + zombie guard from the issue): long-lived sessions recover shared-compute agents on the next dispatch instead of staying silent until a manual restart. Additive; the healthy fast path (live ingress) is unchanged. Refs #2062 Signed-off-by: Bartok9 <danielrpike9@gmail.com>
Brad #2304: mesh_ingress_is_live inside ensure_relay_mesh_for_record only ran on start/restore — not after launch. Local agents hit :9337 themselves, so there is no desktop turn-dispatch hook; add a bounded coordinator watchdog (15s base, double on failure to 120s) that probes ingress, drops a zombie runtime handle, and re-arms via ensure_relay_mesh_for_record for relay-mesh agents. Failed re-arm writes an actionable \"Buzz shared compute offline\" last_error. Share drop_stale_mesh_runtime_if_ingress_dead with ensure path. Unit tests cover dead-port probe, noop drop without handle, and failure copy. Refs #2062 Signed-off-by: Bartok9 <danielrpike9@gmail.com>
…opped agents Addresses Brad's #2304 review of head 6d20d20: 1. Bounded stale stop() — a wedged runtime could hang stop().await and block the watchdog forever, defeating the never-block-re-arm intent. Wrap in a 3s tokio::time::timeout (matches probe budget); log and drop the handle on timeout/error so the watchdog keeps making progress. 2. Probe/evict race — capture the runtime identity (new monotonic DesktopMeshRuntime::id) before the ingress probe .await and only evict if the same handle is still installed on lock reacquire, so a concurrent stop/start replacement is never evicted. 3. Stopped/manual records — re-arm now filters to local relay-mesh agents whose own process is actually running (runtime_pid + process_is_running), so deliberately stopped agents are not resurrected. Adds unit tests for the running/stopped/non-mesh re-arm target filter. Signed-off-by: Bartok9 <danielrpike9@gmail.com>
Address remaining review gaps on the watchdog re-arm path: 1. Keep bounded stop timeout (already present) and document the never-block-re-arm invariant. 2. Extract identity-compare helper + injectable ingress probe so probe/evict never drops a concurrent replacement runtime. 3. Intersect relay-mesh records with live managed_agent_processes pubkeys (not every configured mesh record / pid-only heuristic). 4. persist/clear mesh last_error under managed_agents_store_lock, bump updated_at, preserve unrelated errors, surface save failures. Tests: process-map filter, identity skip, stop budget, error classifier; hardware-gated kill-:9337 recovery documented as Signed-off-by: Bartok9 <danielrpike9@gmail.com> #[ignore] for manual mesh machines.
…micspiral #2304) Address the mesh-applicability review on the re-arm watchdog: 1. Consecutive-failure debounce before eviction (merge-blocker). A single dead /v1/models probe past the 3s budget (transient stall: model load/reload, VRAM alloc, GC/mmap pause, inference saturation) no longer cold-restarts a healthy runtime. Require DEAD_PROBE_EVICT_THRESHOLD (2) consecutive dead probes on the watchdog cadence; a live probe or an identity-mismatch replacement resets the streak. Counter lives on AppState (AtomicU32). 2. clear_mesh_last_error_if_set now matches a sentinel prefix (MESH_REARM_ERROR_SENTINEL) instead of the loose "shared compute" substring, so recovery only clears errors this watchdog set. 3. Documented the serve->client re-arm mode transition as intentional-by-design (safe fail-safe; serve restoration stays restore_mesh_sharing's job). Tests: added eviction_debounces_transient_dead_probe; updated error classifier + actionable-copy tests for the sentinel. 17 passed, 2 ignored. Signed-off-by: Bartok9 <danielrpike9@gmail.com>
CI on #2304 failed: cargo fmt --check on mesh_llm paths, and desktop file-size gate (app_state + mesh_llm growth for watchdog). Signed-off-by: Bartok9 <danielrpike9@gmail.com>
The AppState field is #[cfg(feature = "mesh-llm")] but its initializer was unconditional, breaking non-mesh builds under -D warnings. Gate the initializer and fully-qualify AtomicU32 so the import is not left unused (micspiral, #2304). Signed-off-by: Bartok9 <danielrpike9@gmail.com>
Signed-off-by: Michael Neale <michael.neale@gmail.com>
Signed-off-by: Michael Neale <michael.neale@gmail.com>
Contributor note from #2304 (Bartok9)@micspiral — thank you for taking this over after validating against a real two-machine setup. Moving here as a fellow contributor (happy for you to keep driving the code). Credit: Clean and appreciated. PR body credits #2304; 7 commits remain authored/signed as Bartok9 ( What I dual-checked against the #2304 intent + your.additions
Checks: Desktop/E2E/DCO/lint green from here; MERGEABLE, waiting on review protections. Optional polish (your call — zero pressure; offering eyes only):
I will close #2304 as superseded by this PR (full cherry-pick + your ops layer). Happy to review further revisions, reproduce dogfood nits, or ship a small follow-up PR if you want a fix coded passively — or stay in review-only mode if you prefer to implement everything yourself. |
…unity-policy-fetch * origin/main: Fix formatting in README.md diagram (block#2284) fix(desktop): make Linux AppImage GStreamer work on non-Debian distros (block#2176) refactor(desktop): remove Agent directory section from Agents page (block#2290) fix(desktop): enable arboard Wayland backend so Linux copies reach the Wayland clipboard (block#2904) fix(desktop): supervise and re-arm relay-mesh runtime (block#2823) fix(agents): run live Databricks discovery instead of the fallback list (block#2890) fix(desktop): retire prepend mode on every reader wheel (block#2913) fix(desktop): consolidate prepend scroll correction (block#2855) docs(buzz-acp): correct agent key generation instructions (block#2875) fix(desktop): track concurrent agent turns up to the harness maximum (block#2882) docs(contributing): trim to goose-scale minimal intake surface (block#2780) fix(relay): preserve reconnect backoff (block#2759)
…ding-harnesses * origin/main: (47 commits) fix(desktop): remove bundled libsystemd from AppImage (#2353) docs: document required DCO sign-off and add commit-msg sign-off hook (#2993) fix(desktop): make agent definition authoritative for model/provider/prompt (#1968) chore(desktop): delete dead persona catalog UI cluster (#2886) fix(desktop): surface install failures hidden by curl-pipe exit codes (#2892) fix(mobile): validate invite relay destinations (#2986) Refactor managed-agent runtime into cohesive modules (#2974) Refine mobile settings and themes (#2844) Fix formatting in README.md diagram (#2284) fix(desktop): make Linux AppImage GStreamer work on non-Debian distros (#2176) refactor(desktop): remove Agent directory section from Agents page (#2290) fix(desktop): enable arboard Wayland backend so Linux copies reach the Wayland clipboard (#2904) fix(desktop): supervise and re-arm relay-mesh runtime (#2823) fix(agents): run live Databricks discovery instead of the fallback list (#2890) fix(desktop): retire prepend mode on every reader wheel (#2913) fix(desktop): consolidate prepend scroll correction (#2855) docs(buzz-acp): correct agent key generation instructions (#2875) fix(desktop): track concurrent agent turns up to the harness maximum (#2882) docs(contributing): trim to goose-scale minimal intake surface (#2780) fix(relay): preserve reconnect backoff (#2759) ... Signed-off-by: morgmart <98432065+morgmart@users.noreply.github.com> # Conflicts: # desktop/src/features/onboarding/ui/SetupStep.tsx
Uses MeshLLM built-in `mesh` collective intelligence / Mixture of Agents when Buzz Auto sees two or more distinct physical models. With zero or one model, Auto remains ordinary `auto`. This is an alternative way to improve tool responses, accuracy, and resistance to hallucination when a high-latency distributed mesh contains diverse models. Models and members may come and go: collective routing enables only after stable capacity, drops on confirmed contraction, and can recover later. Mesh-specific failures retry once through ordinary Auto. This update also pins MeshLLM to a v0.73.1-compatible backport of [MeshLLM #1074](Mesh-LLM/mesh-llm#1074), so client-only Buzz nodes cannot enter model election or download a remote provider model. Buzz preserves the selected local sharing model and switches an existing client to sharing across a controlled app restart, retaining one runtime and one `:9337` / `:3131` pair per machine. Validation: - Full local `just ci` passes on the cleaned branch. - MeshLLM host-runtime suite: 1,568 passed, 0 failed; strict Clippy passes. - Buzz desktop Tauri suite with `mesh-llm`: 1,721 passed, 0 failed; strict feature Clippy passes. - Playwright covers client-to-share using the saved local model and no destructive stop. - Packaged two-machine testing proved single-model routing, dual-model collective routing, tool-markup fallback, and runtime reuse. - Packaged client-only recheck routed a real Mini Buzz turn through M5 while Mini stayed `is_client=true`, `is_host=false`, hosted no models, and created no Gemma cache. Builds on the recovery work merged in #2823; this PR does not duplicate it. --------- Signed-off-by: Michael Neale <michael.neale@gmail.com> Signed-off-by: Tyler Longwell <tlongwell@block.xyz> Co-authored-by: Michael Neale <michael.neale@gmail.com> Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
Refs #2062
This carries forward the relay-mesh recovery work from #2304 by @Bartok9 (cherry-picked with original authorship/sign-offs) and adds the startup/readiness supervision and packaging fixes found while validating it against a real two-machine Buzz setup.
From #2304
:9337) after launch and re-arm a stale relay-mesh runtime.Added here
:9337), rather than management port:3131, as Buzz's client-readiness boundary. A usable client no longer fails or holds agent-save open merely because management startup is still pending.buzz-acp,buzz-agent, and tool sidecars to be reported as missing.Validation
Automated:
just ci— passed, including formatting, Clippy with warnings denied, desktop/web/mobile checks and tests, and builds.-D warnings— passed.mesh-llm— built successfully; every bundled sidecar passed executable-mode and deep code-signature verification.Live two-machine E2E:
unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M.buzz-agent/relay-mesh agent.:9337accepted inference and the saved ACP harness started with no error while:3131was still unavailable.CORRECTED-MINI-E2E-OK).:3131was still unavailable.AFTER-RESTART-E2E-OK).@C55message traversed desktop →buzz-acp→buzz-agent→ mini:9337→ M5 compute and published the requested reply successfully.