fix(coding-agent): recover supervisor ownership when its registry entry is pruned - #1149
Closed
gbusto wants to merge 1 commit into
Closed
Conversation
…ry is pruned The supervisor registry lives under the OS temp directory, and `owner.json` was only written on acquire and on a phase change. macOS prunes `/var/folders` entries untouched for about three days, so a healthy long-lived supervisor lost its own registry entry, and `assertCurrent` treated the missing record exactly like a takeover. Every journaled command then failed with `supervisor_generation_stale` for the rest of the process lifetime, with no possible repair: `sameOwnerRecord` compares a token that existed only in the deleted file and in memory. Distinguish a vanished record from a stolen one. A record belonging to another generation is still fatal, but a missing record is now restored under the registry guard, after confirming no live supervisor claimed this socket or descriptor directory. Also rewrite the record hourly so it does not age out of the temp directory to begin with. fixes PrimeIntellect-ai#1148
gbusto
force-pushed
the
fix/1148-supervisor-registry-entry-pruned
branch
from
August 10, 2026 15:47
ae3348c to
ecde15f
Compare
Contributor
|
Thank you for the report and proposed work. This root cause is now covered by maintainer-owned stacked PR #1161, authored independently from We did not inspect or reuse this PR's diff, branch, commits, implementation code, or tests; its public description/comments were used only as a bug report. To keep one review surface, this PR is superseded by #1161 and is being closed. The complete review stack is #1158–#1165. It is being left unmerged for human review after CI and review-bot findings are cleared. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1148.
The supervisor registry lives under
os.tmpdir(), andowner.jsonwas only written on acquire and on a phase change. macOS prunes/var/foldersentries untouched for about three days, so a healthy long-lived supervisor loses its own registry entry.assertCurrenttreated that missing record exactly like a takeover, so every journaled command failed withsupervisor_generation_stalefor the rest of the process lifetime. Nothing could repair it from outside either, becausesameOwnerRecordcompares a token that existed only in the deleted file and in the supervisor's memory.Changes
assertCurrentnow distinguishes a vanished record from a stolen one. A record belonging to another generation is still fatal. A missing record is restored underwithDaemonSupervisorRegistryGuard, re-reading inside the lock, and only after scanning for a live supervisor that has claimed the same socket or descriptor directory. If one exists, ownership really is gone and the error is still raised, so two supervisors can never both consider themselves the owner.unref'd refresh that rewrites the record, keeping its mtime well inside the pruning window so the entry does not age out to begin with. Cleared onrelease().Both halves matter: without the recovery path an already-broken daemon stays broken, and without the refresh a healthy daemon still stalls once every three days before healing itself.
0.4.0fixed the startup side of this throughreadOwnerRecordForScope, but that self-heal only runs when a new supervisor acquires ownership, never for the live process.Testing
packages/coding-agent/test/suite/regressions/1148-supervisor-registry-entry-pruned.test.tscovers four cases: recovery whenowner.jsonis deleted, recovery when the whole owner directory is removed, and the two paths that must still fail, namely a foreign record in our slot and a live supervisor holding our socket.Reverting only the source change fails exactly the two recovery cases and leaves the two ownership-lost cases passing, so the test pins the bug without loosening the fatal path.
npm run checkis clean. The new test passes, as do4600-supervisor-singleton,daemon-supervisor-admission, anddaemon-supervisor-monitor(70 tests). I did not rundaemon-supervisor-process, which is excluded fromtest:ci.Note
assertDaemonSupervisorOwnerCurrenthas the same missing-record-is-fatal logic on the worker validation path. It recovers indirectly once the supervisor rewrites the record, so I left it alone to keep this diff focused. Happy to include it if you would prefer.Note
Fix supervisor ownership recovery when OS prunes its registry entry from temp directory
assertCurrent()inDaemonSupervisorOwnershipnow treats a missing owner record as recoverable: if the current token matches, it rewrites the scope and owner records instead of throwingsupervisor_generation_stale.release()cancels and awaits any in-flight refresh before performing the cleanup rename-and-delete sequence.Macroscope summarized ecde15f.