Provider maintenance idle-shutdown timer is never scheduled when a maintenance request never settles

[dfaivre-pcs]

Summary

Provider bridge workers spawned in ~/.bb/provider-maintenance-workspace accumulate and are never retired, despite PROVIDER_MAINTENANCE_IDLE_TIMEOUT_MS being 60 s. On a box that had been up ~13 h with 7–9 live threads, 7–8 of them were resident simultaneously, in cohorts up to ~12 minutes old, holding ~700–770 MB. They are npm version probes (npm view @openai/codex version, npm list @anthropic-ai/claude-code, npm list @earendil-works/pi-coding-agent), so each is a fresh ~85–136 MB node process that has no work left to do.

Verified live against bb-app 0.42.1, node v24.13.1, WSL2 (6.6.87.2), 4 installed providers (provider-acp, provider-claude-code, provider-codex, provider-pi).

This is distinct from #2308 and #2289. The thread-serving bridge workers are healthy on this build: startIdleProviderSessionReaper fires as designed, and the daemon log carries 17 Reaped idle provider sessions records with idleForMs of 1.85–2.09 M (31–35 min), matching IDLE_PROVIDER_SESSION_REAP_AFTER_MS. There is exactly one worker per live thread environment. Only the maintenance workspace accumulates.

Evidence

One snapshot, taken by reading /proc/<pid>/cwd and /proc/<pid>/cmdline (process titles are rewritten, so pgrep/ps cannot see any of this — pgrep -c node reports 0 on this box):

pid 30688   provider-acp               522s old    84 MB
pid 30689   provider-acp               522s old    85 MB
pid 30695   provider-acp               522s old    85 MB
pid 30790   provider-acp               522s old    86 MB
pid 8381    provider-claude-code       741s old   136 MB
pid 8383    provider-codex             741s old   110 MB
pid 8388    provider-acp               741s old    86 MB

All are children of the single host daemon pid, all with cwd = ~/.bb/provider-maintenance-workspace, all running bb-provider-bridge-worker.mjs.

The two cohorts are what makes this conclusive. They are 219 s apart in a single snapshot: the 741 s cohort was still alive when the 522 s cohort was spawned, and both are far past the 60 s idle timeout. Working retirement would have removed the older cohort long before the newer one existed. An earlier snapshot the same day showed 8 simultaneous workers in the same workspace at ages 370–460 s, spawned in bursts of three — consistent with mapProviderMaintenanceRequests' concurrency limit of 3.

provider-acp accumulates fastest: 5 of 8 workers in one snapshot, 4 of them in a single cohort, suggesting the ACP bridge spawns one per configured ACP agent and none is retired.

What the logs say

~/.bb/logs/host-daemon.20.log contains zero occurrences of either warning on the shutdown path:

• Failed to shut down idle provider maintenance runtime — 0
• Failed to shut down provider maintenance runtime during invalidation — 0

So shutdownProviderMaintenanceRuntime() is not failing loudly. Either it never runs, or it completes without terminating the worker processes.

Two candidate mechanisms

I did not narrow this further — offering both rather than asserting one:

1. The idle timer is never scheduled. withProviderMaintenanceRuntime schedules it only in its finally, and only when providerMaintenanceActiveRequests reaches 0 (apps/host-daemon/src/runtime-manager.ts, scheduleProviderMaintenanceIdleShutdown). A probe request that never settles — npm view against a slow or unreachable registry has no timeout of its own — keeps the counter above 0 forever, so the 60 s timer is never armed and the workers outlive every subsequent probe round.
2. The shutdown does not reach the workers. If the timer does fire, runtime.shutdown() resolving without terminating the spawned bb-provider-bridge-worker.mjs children would leave exactly this residue, silently, since nothing on that path logs success.

The cohort structure slightly favours (1): each round leaves its whole burst behind, rather than some workers dying and others surviving.

Suggested direction

Whatever the cause, the shape of the work looks over-provisioned for what it produces. A version probe is a short, cacheable, read-only question, and the current design pays a fresh ~100 MB node process per provider per round to ask it. Three things would each help independently:

• Cache the answer across rounds with a TTL, so an idle box stops re-probing at all.
• Bound the probe with its own timeout, so a hung npm view cannot pin providerMaintenanceActiveRequests above 0 indefinitely.
• Log the shutdown, not only its failure. The silence on this path is why the accumulation is invisible until someone counts processes by /proc/<pid>/exe.

Impact

On a 48 GB dev box shared with Docker and a browser this was ~700 MB of the ~10 GB the whole agent stack held, and it is the component that grows with uptime rather than with thread count — so it is the one that gets worse the longer the machine stays up. bb-app stop clears it.

Workaround

bb-app stop and restart. There is no way to retire them without taking the daemon down.

---

Downstream tracking issue, with the census tooling used to find this: https://github.com/PremierCrop/pcs/issues/1341

[bb-slop-cop]

🚨 SLOP COP 🚨 · new-issue-autopilot

• Classification: Type Bug · Priority Medium · Effort Low · labels host, providers, perf
• Report: https://get-bb.github.io/reports/issues/3230.html
• Verdict: PARTIALLY REPRODUCED · confidence medium · reproduction label partial-repro
• Root cause: a provider-maintenance request that never settles keeps the active-request counter positive, so the idle-shutdown timer is never scheduled. This reproduced in two clean trusted checkouts; the reported WSL/npm process behavior did not.
• Pull request: none; a generic timeout could interrupt legitimate provider install/update operations, so the verified partial mechanism needs a cancellation-boundary decision before a safe fix.

[dfaivre-pcs]

I'm sorry — this issue should not have been filed in the shape it was. I led with dramatic process and memory numbers taken from a single uncontrolled snapshot on one machine, presented "unbounded growth and no reaping" as established fact, and buried the one part I had actually reasoned my way to as a secondary hypothesis. Your autopilot then had to spend a repro cycle separating the two. That was avoidable with one more measurement before filing.

Your autopilot got this right, and it split my report exactly where it should be split. Retraction of the rest below.

Confirmed, and the only part that should stay open: a provider-maintenance request that never settles keeps providerMaintenanceActiveRequests positive, so scheduleProviderMaintenanceIdleShutdown — which only arms in withProviderMaintenanceRuntime's finally, and only when the counter reaches 0 — never schedules the timer. You reproduced that in two clean checkouts. The one observation from my report that supports it and still stands is that ~/.bb/logs/host-daemon.*.log carried zero Failed to shut down idle provider maintenance runtime warnings: consistent with the timer never being armed, rather than with the shutdown failing.

Retracted — everything about process accumulation and memory. Specifically:

• the 7–8 workers resident simultaneously, the two-cohort structure, and the ~700–770 MB total;
• the per-worker ~85–136 MB figures and the "each npm invocation is a fresh node process" framing;
• provider-acp accumulating fastest (5 of 8 in one snapshot);
• the claim that this is the component that "grows with uptime rather than thread count";
• the whole Impact section.

Two reasons, and the first is mine, not yours:

1. I never established it was steady state. Every number came from one uncontrolled snapshot on one WSL2 box. When I re-measured the same box later with no intervention, the count had fallen from ~65 processes / 10 GB to 43 / 7.0 GB. That is transient behavior, and it invalidates the premise I built the report on. I should have taken a series before filing, not a snapshot.
2. The likely cause is ours, not bb-app's. We have in-flight work contributing ZDOTDIR to provider processes so that non-interactive zsh -c tool shells load our directory environment via eval "$(direnv export zsh)". That is a fork-amplification hazard by construction, and the box was showing 154 Host plugin worker ready / 154 stopped in a 310-line daemon log while I was measuring. I can't prove the loop — the storm was over before I looked and I never captured it live — but it is a much better explanation for my process pile than a bb-app leak, and it means you should not chase those numbers.

Separately, the impact framing was wrong on its own terms: I wrote it against a 48 GB machine, and our team's machines are 96 GB. At that size the magnitude I described wouldn't matter even if it had reproduced.

On the fix: your cancellation-boundary objection is well taken and my "bound the probe with its own timeout" suggestion was too blunt — a generic timeout interrupting a legitimate install or update is a worse failure than the one it prevents. I have no better proposal, and no standing to push one given the above.

I've retitled the issue to the mechanism you actually confirmed so it doesn't keep advertising the unreproduced part. Please feel free to close, relabel, or rescope it however suits you — including closing it outright if a never-settling maintenance request isn't worth tracking on its own.

[dfaivre-pcs]

Hey, Human @dfaivre-pcs here... sorry about that sloppy issue file. I shouldn't code after 3pm... This project is amazing and I'm loving using it. If there is a good way to help on it as I find issues/features/etc - let me know -- I don't have any experience in contributing to open source projects. Thank you for all the hard work!