Skip to content

fix(coding-agent): recover supervisor ownership when its registry entry is pruned - #1149

Closed
gbusto wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
gbusto:fix/1148-supervisor-registry-entry-pruned
Closed

fix(coding-agent): recover supervisor ownership when its registry entry is pruned#1149
gbusto wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
gbusto:fix/1148-supervisor-registry-entry-pruned

Conversation

@gbusto

@gbusto gbusto commented Aug 10, 2026

Copy link
Copy Markdown

Fixes #1148.

The supervisor registry lives under os.tmpdir(), and owner.json was only written on acquire and on a phase change. macOS prunes /var/folders entries untouched for about three days, so a healthy long-lived supervisor loses its own registry entry. assertCurrent treated that missing record exactly like a takeover, so every journaled command failed with supervisor_generation_stale for the rest of the process lifetime. Nothing could repair it from outside either, because sameOwnerRecord compares a token that existed only in the deleted file and in the supervisor's memory.

Changes

  • assertCurrent now distinguishes a vanished record from a stolen one. A record belonging to another generation is still fatal. A missing record is restored under withDaemonSupervisorRegistryGuard, re-reading inside the lock, and only after scanning for a live supervisor that has claimed the same socket or descriptor directory. If one exists, ownership really is gone and the error is still raised, so two supervisors can never both consider themselves the owner.
  • Added an hourly unref'd refresh that rewrites the record, keeping its mtime well inside the pruning window so the entry does not age out to begin with. Cleared on release().

Both halves matter: without the recovery path an already-broken daemon stays broken, and without the refresh a healthy daemon still stalls once every three days before healing itself.

0.4.0 fixed the startup side of this through readOwnerRecordForScope, but that self-heal only runs when a new supervisor acquires ownership, never for the live process.

Testing

packages/coding-agent/test/suite/regressions/1148-supervisor-registry-entry-pruned.test.ts covers four cases: recovery when owner.json is deleted, recovery when the whole owner directory is removed, and the two paths that must still fail, namely a foreign record in our slot and a live supervisor holding our socket.

Reverting only the source change fails exactly the two recovery cases and leaves the two ownership-lost cases passing, so the test pins the bug without loosening the fatal path.

npm run check is clean. The new test passes, as do 4600-supervisor-singleton, daemon-supervisor-admission, and daemon-supervisor-monitor (70 tests). I did not run daemon-supervisor-process, which is excluded from test:ci.

Note

assertDaemonSupervisorOwnerCurrent has the same missing-record-is-fatal logic on the worker validation path. It recovers indirectly once the supervisor rewrites the record, so I left it alone to keep this diff focused. Happy to include it if you would prefer.

Note

Fix supervisor ownership recovery when OS prunes its registry entry from temp directory

  • assertCurrent() in DaemonSupervisorOwnership now treats a missing owner record as recoverable: if the current token matches, it rewrites the scope and owner records instead of throwing supervisor_generation_stale.
  • A periodic refresh (every ~1 hour) rewrites the owner record proactively to reduce the chance of OS temp pruning causing issues for long-running daemons.
  • release() cancels and awaits any in-flight refresh before performing the cleanup rename-and-delete sequence.
  • Adds a regression test suite covering recovery from deleted files, deleted directories, token conflicts, and competing live supervisors.

Macroscope summarized ecde15f.

…ry is pruned

The supervisor registry lives under the OS temp directory, and `owner.json` was
only written on acquire and on a phase change. macOS prunes `/var/folders`
entries untouched for about three days, so a healthy long-lived supervisor lost
its own registry entry, and `assertCurrent` treated the missing record exactly
like a takeover. Every journaled command then failed with
`supervisor_generation_stale` for the rest of the process lifetime, with no
possible repair: `sameOwnerRecord` compares a token that existed only in the
deleted file and in memory.

Distinguish a vanished record from a stolen one. A record belonging to another
generation is still fatal, but a missing record is now restored under the
registry guard, after confirming no live supervisor claimed this socket or
descriptor directory. Also rewrite the record hourly so it does not age out of
the temp directory to begin with.

fixes PrimeIntellect-ai#1148
@sethkarten

Copy link
Copy Markdown
Contributor

Thank you for the report and proposed work. This root cause is now covered by maintainer-owned stacked PR #1161, authored independently from upstream/main.

We did not inspect or reuse this PR's diff, branch, commits, implementation code, or tests; its public description/comments were used only as a bug report. To keep one review surface, this PR is superseded by #1161 and is being closed.

The complete review stack is #1158#1165. It is being left unmerged for human review after CI and review-bot findings are cleared.

@sethkarten sethkarten closed this Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Daemon supervisor permanently fails with supervisor_generation_stale after the OS prunes its registry entry from the temp directory

2 participants