diff --git a/.the-framework/LAYOUT b/.the-framework/LAYOUT index 4ec1f8e4b..52cc94569 100644 --- a/.the-framework/LAYOUT +++ b/.the-framework/LAYOUT @@ -1,8 +1,7 @@ framework-dir: .the-framework data-branch: agent-data -logs-branch: agents-logs -archive-dir: agents events-file: events.jsonl meta-file: agent.json tickets-dir: tickets queue-file: TODO_AGENTS.md +runs-dir: agents diff --git a/FEATURES-SPEC.md b/FEATURES-SPEC.md index 40917c255..cb917de6c 100644 --- a/FEATURES-SPEC.md +++ b/FEATURES-SPEC.md @@ -89,7 +89,7 @@ happens while nobody is at the keyboard. - Hottest tickets - Projects sidebar - A project whose directory was renamed or deleted leaves the sidebar on the next refresh, and comes back when the directory does; the registration is kept -- Project errors: a project whose bookkeeping branches cannot reach origin (push rejected, or no remote) — the `agent-data` branch or the `agents-logs` branch — is flagged with a red dot in the sidebar and a banner on its page, until a sync converges +- Project errors: a project whose bookkeeping branch cannot reach origin (push rejected, or no remote) — the `agent-data` branch — is flagged with a red dot in the sidebar and a banner on its page, until a sync converges ## Tickets @@ -119,6 +119,8 @@ happens while nobody is at the keyboard. - `tickets` on every agent's PATH — the skill's command line: `list` and `show ` read the tickets with their plans and their holders, `put ` writes a ticket, a plan or the import stamp, `close ` removes a ticket with its plan and claim, `claim ` and `release ` are the claim; every one of them is one commit pushed to the `agent-data` branch - The `queue` skill: every checkout The Framework creates carries the `@gemstack/skill-queue` package's `SKILL.md` beside the other two (`.claude/skills/queue`, `.agents/skills/queue`), and the presets tell the agent to use it — the agent queue is on the `agent-data` branch, read and changed with the command; an agent anywhere else gets the same git bridge as for the tickets (#1750) - `queue` on every agent's PATH — the skill's command line: `queue` reads the queue in the order it is worked, `queue add [--priority N]` puts an entry on it in its priority section, `queue done ` takes an entry off; every one of them is one commit pushed to the `agent-data` branch +- The `logs` skill: every checkout The Framework creates carries the `@gemstack/skill-logs` package's `SKILL.md` beside the other three (`.claude/skills/logs`, `.agents/skills/logs`); it tells the agent to read a ticket's past runs before planning or working it — a stopped or failed run says what to avoid, a done run with a PR says the work may be there; an agent anywhere else gets the skill's text through the same bridge as for the tickets (#1769) +- `logs` on every agent's PATH — the skill's command line, read-only: `logs` lists the runs newest first (the newest 20 unless `--limit N`), `--ticket ` the runs that worked one ticket, `--branch ` the runs on one branch, `logs show ` one run's card with what the agent said, its result, its ending and its cost; the daemon's own bookkeeping on the card is never printed - The session name is the branch: an agent is labelled by its `agent-` branch, read from git after every turn — nothing to signal, nothing to record twice - Dependency directories shared from the parent checkout instead of reinstalled — as directories of links, so an agent's own install stays in its checkout and never rewrites or purges the parent's - A checkout whose work is not on the remote is kept — and a publish-nothing (`handoff: local`) agent's is kept until you publish or delete it @@ -130,7 +132,7 @@ happens while nobody is at the keyboard. - Empty agents publish nothing - Handoff panel: push / open PR / merge, as buttons - A withheld merge is reported with its reason -- Agent history archived on the `agents-logs` branch under per-user directories — pushed the moment a session settles +- Agent history recorded on the `agent-data` branch as the `logs` skill's runs, under per-person directories — pushed the moment a session settles; the framework's own fields ride inside the skill's card under one key, and its other events inside the diary beside the four kinds the skill knows - Post-merge quality follow-ups queued (maintainability / security) - Knowledge folded back into `DECISIONS.md` / `FACTS.md` / `INSIGHTS.md` at merge @@ -148,7 +150,7 @@ happens while nobody is at the keyboard. - Reclaim the checkout of an agent whose work is on the remote — never by publishing what a `handoff: local` agent refused to - An agent that committed nothing leaves no branch behind: its empty branch goes with its checkout, never pushed — and the run-id branch it started on goes too, once the branch it moved to holds everything the run-id branch did - A directory under `.branches/` that git does not know as a worktree is never committed, pushed, linked or deleted through — it is reported and left alone, so a leftover can never stand in for your own checkout -- One triage at a time, across machines: a routine lock (`routines/.lock.md` on the `agents-logs` branch) taken by the daemon before the run starts and released when it ends, whatever the ending; a held lock stands the routine down naming the machine holding it, with no agent spent; a lock left by a dead machine expires after four hours, and a daemon frees its own on boot +- One triage at a time, across machines: a routine lock (`routines/.lock.md` on the `agent-data` branch) taken by the daemon before the run starts and released when it ends, whatever the ending; a held lock stands the routine down naming the machine holding it, with no agent spent; a lock left by a dead machine expires after four hours, and a daemon frees its own on boot - The agent drains its own TODO backlog, one entry per turn ## Spending @@ -190,7 +192,7 @@ happens while nobody is at the keyboard. - Answer a cloud agent's question from the dashboard (typed back into claude.ai) — the same gate panel a local agent gets, multi-select and stop options included, listed with every other open question - Browser-bridge token setting - A cloud run's row follows the session's real branch and PR, with its armed draft PR opened when the session opens none -- Another machine's runs on the shared `agents-logs` branch are told apart: their rows carry a glyph naming the machine that started them (the Overview's working-now card spells it out), and a run is listed once even when two checkouts share its archive +- Another machine's runs on the shared `agent-data` branch are told apart: their rows carry a glyph naming the machine that started them (the Overview's working-now card spells it out), and a run is listed once even when two checkouts share its archive ## Notifications diff --git a/SPEC.md b/SPEC.md index 081ecb5ca..2995c46d6 100644 --- a/SPEC.md +++ b/SPEC.md @@ -25,7 +25,7 @@ Every user-facing feature is enumerated in `FEATURES-SPEC.md`. - **The agent is the unit of work** - one task, in its own git worktree on its own branch, streaming everything it does as events; finished work is pushed and leaves as a pull request. - **Black-box driving** - the framework prompts the wrapped coding-agent CLI, lets the CLI's own loop run a full turn, and learns everything from the turn's final message: the session name the agent invented, the questions it stops to ask, and the ready-for-merge signal. - **Autonomy bounded by the account's own quota** - unattended work runs only while the account is under its pro-rated quota boundary; work a human asks for is never blocked, and a running agent is never interrupted over quota. -- **Nothing bookkeeping-shaped on a code branch** - the tickets and the agent queue live on the `agent-data` branch, the `tickets` skill's and the `queue` skill's; The Framework's own records of its runs — the agent archives, the routine locks — live on the `agents-logs` branch. The default branch stays 100% code. +- **Nothing bookkeeping-shaped on a code branch** - the tickets, the agent queue and the record of every run live on the `agent-data` branch, the `tickets`, `queue` and `logs` skills'; The Framework's routine locks sit beside them. The default branch stays 100% code. ## Business logic @@ -47,7 +47,7 @@ See `## User story`: the daemon keeps working while nobody is around, within the #### Business logic -On a shared clock the daemon runs its background jobs: Auto PM works the agent queue down and refills it by triaging tickets and planning the ones without plans; the CI watch merges the framework's pull requests once their checks pass and starts a fix agent when checks go red; sweeps reclaim finished agents' checkouts (only what is already on the remote) and keep bookkeeping healthy, and a routine that must not run twice is guarded by a routine lock on the `agents-logs` branch. Each unattended start checks the quota boundary first. +On a shared clock the daemon runs its background jobs: Auto PM works the agent queue down and refills it by triaging tickets and planning the ones without plans; the CI watch merges the framework's pull requests once their checks pass and starts a fix agent when checks go red; sweeps reclaim finished agents' checkouts (only what is already on the remote) and keep bookkeeping healthy, and a routine that must not run twice is guarded by a routine lock on the `agent-data` branch. Each unattended start checks the quota boundary first. ## Before modifying/creating SPEC.md files diff --git a/package.json b/package.json index f6bbf3bdd..f6655d9d2 100644 --- a/package.json +++ b/package.json @@ -11,8 +11,8 @@ "preview": "pnpm -C packages/framework preview", "dev": "pnpm -C packages/framework dev", "typecheck": "pnpm -C packages/framework typecheck && pnpm -C packages/the-framework.ai typecheck", - "clean": "pnpm -C packages/agent-data clean && pnpm -C packages/skill-branches clean && pnpm -C packages/skill-tickets clean && pnpm -C packages/skill-queue clean && pnpm -C packages/agent-driver clean && pnpm -C packages/framework clean", - "test": "pnpm -C packages/agent-data test && pnpm -C packages/skill-branches test && pnpm -C packages/skill-tickets test && pnpm -C packages/skill-queue test && pnpm -C packages/agent-driver test && pnpm -C packages/framework test", + "clean": "pnpm -C packages/agent-data clean && pnpm -C packages/skill-branches clean && pnpm -C packages/skill-tickets clean && pnpm -C packages/skill-queue clean && pnpm -C packages/skill-logs clean && pnpm -C packages/agent-driver clean && pnpm -C packages/framework clean", + "test": "pnpm -C packages/agent-data test && pnpm -C packages/skill-branches test && pnpm -C packages/skill-tickets test && pnpm -C packages/skill-queue test && pnpm -C packages/skill-logs test && pnpm -C packages/agent-driver test && pnpm -C packages/framework test", "========= Website": "", "// Develop website https://the-framework.ai locally": "", "website": "cd packages/the-framework.ai/ && pnpm run dev", @@ -25,6 +25,7 @@ "@gemstack/skill-branches": "workspace:*", "@gemstack/skill-tickets": "workspace:*", "@gemstack/skill-queue": "workspace:*", + "@gemstack/skill-logs": "workspace:*", "typescript": "^7.0.2" } } diff --git a/packages/SPEC.md b/packages/SPEC.md index b8406d184..8dbd7a74c 100644 --- a/packages/SPEC.md +++ b/packages/SPEC.md @@ -5,11 +5,12 @@ The deliverables The Framework ships, one directory each: - `skill-branches/` — the `@gemstack/skill-branches` npm package: the git conventions and operations behind an agent's own checkout, with the skill's instructions (`SKILL.md`) and the `branches` command an agent follows them with: the first skill of the skills-plus architecture (#1725). The product depends on it, and so does the skill-tickets package; it depends on agent-data. - `skill-tickets/` — the `@gemstack/skill-tickets` npm package: the project's tickets on the `agent-data` branch of its own repository, with the skill's instructions (`SKILL.md`) and the `tickets` command an agent reads, writes and claims them with: the second skill of the skills-plus architecture (#1748). The product depends on it; it depends on agent-data for the branch it keeps everything on. - `skill-queue/` — the `@gemstack/skill-queue` npm package: the project's agent queue on the `agent-data` branch of its own repository, with the skill's instructions (`SKILL.md`) and the `queue` command an agent reads and changes it with: the third skill, split out of skill-tickets (#1750). The product depends on it; it depends on agent-data only. +- `skill-logs/` — the `@gemstack/skill-logs` npm package: the record of every run agents made on a project, on the `agent-data` branch of its own repository, with the skill's instructions (`SKILL.md`) and the `logs` command an agent reads it with: the fourth skill (#1769). The product writes every run through it and reads its history back the same way; it depends on agent-data only. - `agent-driver/` — the `agent-driver` npm package: the driver seam — start a coding-agent CLI in a directory, prompt it for one full turn, stream what it does, resume it later — with the Claude Code, Codex, GitHub Actions and fake implementations. The product depends on it and adds its own cloud-session implementation behind the same contract. - `chrome-extension/` — the Claude web bridge, a companion Chrome extension that connects Claude Code cloud sessions on claude.ai back to the local dashboard. - `the-framework.ai/` — the marketing website. -The product depends on the agent-data, skill-branches, skill-tickets, skill-queue and agent-driver packages in code; the extension and the website depend on the product only in what they present. See the root `SPEC.md` for how they relate as a product. +The product depends on the agent-data, skill-branches, skill-tickets, skill-queue, skill-logs and agent-driver packages in code; the extension and the website depend on the product only in what they present. See the root `SPEC.md` for how they relate as a product. ## Before modifying/creating SPEC.md files diff --git a/packages/framework/SPEC.md b/packages/framework/SPEC.md index 2308a6089..2ff980e3b 100644 --- a/packages/framework/SPEC.md +++ b/packages/framework/SPEC.md @@ -2,7 +2,7 @@ The product: the `framework` npm package. One CLI, `the-framework`, runs a foreg How the package's pieces relate: -- `src/` — everything that runs in Node: the CLI, the daemon, the agent lifecycle (checkouts through the `skill-branches` package, gates, handoff), the drivers (the agent-driver package's, plus the product's own hand-off to a Claude Code cloud session), the `agents-logs` branch its own records live on (a branch used as a file store through the `@gemstack/agent-data` package; the tickets are the `skill-tickets` package's and the queue the `skill-queue` package's), autonomy (Auto PM, sweeps, CI watch), and the server side of the dashboard. +- `src/` — everything that runs in Node: the CLI, the daemon, the agent lifecycle (checkouts through the `skill-branches` package, gates, handoff), the drivers (the agent-driver package's, plus the product's own hand-off to a Claude Code cloud session), its own records — the runs are the `skill-logs` package's, the tickets the `skill-tickets` package's and the queue the `skill-queue` package's, all on the `agent-data` branch, a branch used as a file store through the `@gemstack/agent-data` package, autonomy (Auto PM, sweeps, CI watch), and the server side of the dashboard. - `dashboard/` — the browser app the daemon serves: a single-page app that is a pure projection of the files the daemon writes, reading over `POST /_rpc/` and a live event stream, and steering agents back through the same daemon. - `prompts/` — every prompt The Framework sends an agent, as markdown: the built-in system prompt, the protocols agents answer through, and the presets. The markdown is the only source of truth for agent-facing text. - `scripts/` — the build steps that compile the prompts into importable strings and run the package's test suites. diff --git a/packages/framework/dashboard/components/AgentHistory.SPEC.md b/packages/framework/dashboard/components/AgentHistory.SPEC.md index 547e982b4..b71247930 100644 --- a/packages/framework/dashboard/components/AgentHistory.SPEC.md +++ b/packages/framework/dashboard/components/AgentHistory.SPEC.md @@ -52,7 +52,7 @@ Scanning the list, the user needs to tell at a glance which agents need them, wh Each row shows a status word with a matching dot: a working agent pulses and reads its running status; an agent parked on the user reads "waiting" with a still dot, because it used to pulse identically whether it was mid-edit or had been idle for an hour; an agent that ended clean but has not yet reported its handoff reads "publishing…" with a pulsing dot, matching the wording on its own page; an agent that handed its task to a cloud session reads, once its local half is done, what the cloud side is doing per the cloud state rule (`cloud-run-state`): "waiting" with a still dot when the browser bridge holds a question its session is parked on, "in cloud" while the session is assumed to still be working, "merged" once the framework merged its pull request, and "done" once it has a pull request or the session can no longer be working — never "in cloud" forever. Otherwise the row shows the agent's final status. -Beside that: how long ago it started (preceded by its project name on the Overview), and, at the end of the line, a glyph naming the machine when another machine's daemon started the agent — the `agents-logs` branch the archives live on is shared, so other machines' agents are listed here, and one that looked like this daemon's own was a mystery; a glyph rather than a word, since the rail's fixed width would cut a machine name short — a device glyph naming the device when the agent runs on another machine, a cloud glyph when it is a cloud session, and the logo of the driver that ran it. Underneath sits the agent's task; a task too long for the rail's fixed width is faded at its end and shows in full on hover, while one that fits is shown plainly with no hover at all. +Beside that: how long ago it started (preceded by its project name on the Overview), and, at the end of the line, a glyph naming the machine when another machine's daemon started the agent — the `agent-data` branch the runs live on is shared, so other machines' agents are listed here, and one that looked like this daemon's own was a mystery; a glyph rather than a word, since the rail's fixed width would cut a machine name short — a device glyph naming the device when the agent runs on another machine, a cloud glyph when it is a cloud session, and the logo of the driver that ran it. Underneath sits the agent's task; a task too long for the rail's fixed width is faded at its end and shows in full on hover, while one that fits is shown plainly with no hover at all. ### Destinations and what is highlighted diff --git a/packages/framework/dashboard/components/AgentHistory.tsx b/packages/framework/dashboard/components/AgentHistory.tsx index 2e5b28d20..072394fa6 100644 --- a/packages/framework/dashboard/components/AgentHistory.tsx +++ b/packages/framework/dashboard/components/AgentHistory.tsx @@ -626,7 +626,7 @@ function AgentHistoryRow({ being decorative. */} {(remote || cloud || picked || startedOn) && ( - {/* Another machine's daemon started this run (#1648): the shared agents-logs branch lists every + {/* Another machine's daemon started this run (#1648): the shared agent-data branch lists every machine's runs here, and one that looked exactly like this daemon's own was a mystery solved only by reading the archive. A glyph, not a word in the meta line: the rail's fixed width truncated a hostname to "from…". */} diff --git a/packages/framework/dashboard/components/ProjectErrorBanner.SPEC.md b/packages/framework/dashboard/components/ProjectErrorBanner.SPEC.md index 931f2dcce..f647dc182 100644 --- a/packages/framework/dashboard/components/ProjectErrorBanner.SPEC.md +++ b/packages/framework/dashboard/components/ProjectErrorBanner.SPEC.md @@ -2,7 +2,7 @@ The warning banner at the top of a project's page, listing every project error t ## Glossary -- **project error** - a condition the daemon has found broken about a project and keeps recorded until the condition is gone. Today the only kind is one of the project's two bookkeeping branches failing to converge with origin — the `agent-data` branch, holding the tickets and the agent queue, or the `agents-logs` branch, holding the agent archives and the routine locks — headlined "Not syncing with the remote". +- **project error** - a condition the daemon has found broken about a project and keeps recorded until the condition is gone. Today the only kind is the project's bookkeeping branch failing to converge with origin — the `agent-data` branch, holding the tickets, the agent queue, the runs and the routine locks — headlined "Not syncing with the remote". ## Business logic — TL;DR diff --git a/packages/framework/dashboard/components/ProjectErrorBanner.tsx b/packages/framework/dashboard/components/ProjectErrorBanner.tsx index 51a695576..b02a89cb4 100644 --- a/packages/framework/dashboard/components/ProjectErrorBanner.tsx +++ b/packages/framework/dashboard/components/ProjectErrorBanner.tsx @@ -3,7 +3,7 @@ import type { ProjectError, ProjectErrorCode } from '../../src/index.js' import { formatAge } from '../lib/format-date.js' // What the daemon currently finds wrong with the project (#1500), at the top of its page. The -// daemon records the state and clears it when the condition is gone (#1599: a tickets or agents-logs branch that +// daemon records the state and clears it when the condition is gone (#1599: an agent-data branch that // cannot reach origin), so this renders exactly what the project list carries — no state of its // own, and nothing to dismiss: the way to make it go away is to fix the thing it names. diff --git a/packages/framework/dashboard/components/ProjectHome.tsx b/packages/framework/dashboard/components/ProjectHome.tsx index 2ffe38125..d87399878 100644 --- a/packages/framework/dashboard/components/ProjectHome.tsx +++ b/packages/framework/dashboard/components/ProjectHome.tsx @@ -49,7 +49,7 @@ export function ProjectHome({ return ( - {/* Above the start form, because an agent started on a project whose tickets or agents-logs branch cannot + {/* Above the start form, because an agent started on a project whose agent-data branch cannot reach origin (#1599) works from stale tickets and a queue nobody else will see. */} { diff --git a/packages/framework/src/SPEC.md b/packages/framework/src/SPEC.md index 7b5385585..11f3c7d83 100644 --- a/packages/framework/src/SPEC.md +++ b/packages/framework/src/SPEC.md @@ -6,8 +6,8 @@ Everything of The Framework that runs in Node: the CLI entry, the daemon, the ag - **The agent lifecycle** - an agent gets a worktree and a branch, is framed with the built-in system prompt, works turn by turn through the driver, parks on gates when it needs the user, works the agent queue when its main task settles, and publishes itself per its handoff level when done. - **The driver seam** - the wrapped coding-agent CLI is a black box (the `agent-driver` package; `driver/` holds only the product's own cloud-session implementation): prompt in, full turn out, everything learned by parsing the turn's final message. Which CLI (Claude Code, Codex) and where it runs (this device, a GitHub Actions runner, a Claude Code cloud session) are two separate axes. - **Files are the seam** - an agent appends events to `.the-framework/events.jsonl`; steering flows back through `.the-framework/control.jsonl`; the agent's status lives in `agent.json`. The daemon and every surface are projections of these files (`store/`) — there is no process-to-process IPC. -- **Two branches carry the records** - the tickets and the agent queue belong to the `tickets` and `queue` skills (the `@gemstack/skill-tickets` and `@gemstack/skill-queue` packages) and live on the project's `agent-data` branch; The Framework's own records — the agent archives and the routine locks — live on its `agents-logs` branch. Both are branches used as file stores (the `@gemstack/agent-data` package's): nobody edits them in a working tree, and every change goes through one serialized sync → apply → commit → push cycle, so every machine and cloud session sees the same thing while the default branch stays 100% code. -- **Autonomy, bounded** - on one shared background clock the daemon runs Auto PM (drain the agent queue, triage, plan — at most one firing per routine at a time, guarded by a routine lock on the `agents-logs` branch), the CI watch (merge on green, fix on red), and the sweeps (reclaim pushed checkouts, adopt cloud work, expire dead refs) — each start gated by the quota boundary. +- **One branch carries the records** - the tickets, the agent queue and the runs belong to the `tickets`, `queue` and `logs` skills (the `@gemstack/skill-tickets`, `@gemstack/skill-queue` and `@gemstack/skill-logs` packages) and live on the project's `agent-data` branch, with The Framework's routine locks beside them. It is a branch used as a file store (the `@gemstack/agent-data` package's): nobody edits it in a working tree, and every change goes through one serialized sync → apply → commit → push cycle, so every machine and cloud session sees the same thing while the default branch stays 100% code. +- **Autonomy, bounded** - on one shared background clock the daemon runs Auto PM (drain the agent queue, triage, plan — at most one firing per routine at a time, guarded by a routine lock on the `agent-data` branch), the CI watch (merge on green, fix on red), and the sweeps (reclaim pushed checkouts, adopt cloud work, expire dead refs) — each start gated by the quota boundary. - **The dashboard's server side** - `dashboard/` (in this directory) serves the built browser app and implements its reads, live event stream, actions, and daemon-to-daemon relay; `dashboard-rpc/` is the RPC surface itself. ## Glossary diff --git a/packages/framework/src/agent-archive.BUG-ANALYSIS.md b/packages/framework/src/agent-archive.BUG-ANALYSIS.md deleted file mode 100644 index 21e8ae498..000000000 --- a/packages/framework/src/agent-archive.BUG-ANALYSIS.md +++ /dev/null @@ -1 +0,0 @@ -Last analyzed: [25 August 2026](https://github.com/framework/the-framework/pull/1698) diff --git a/packages/framework/src/agent-archive.SPEC.md b/packages/framework/src/agent-archive.SPEC.md deleted file mode 100644 index 10f298f26..000000000 --- a/packages/framework/src/agent-archive.SPEC.md +++ /dev/null @@ -1,60 +0,0 @@ -Decides which per-user directory a project's archives are filed under on the logs branch (`agents-logs`), so that every finished agent's lasting record survives the repo being cleaned and two people working the same repo never write to the same paths. - -## Business logic — TL;DR - -- **Archives are filed per user** - a finished agent's archive lands under `agents//` on the logs branch, where `` comes from the git identity the repo already commits with. -- **The directory name can never climb out of the archive** - the identity is reduced to a conservative, lowercased name that must start with a letter or digit; anything that cannot be made to fit is filed under `anonymous` instead. -- **A missing identity still gets a home** - when git has no configured identity, the archive goes to `anonymous` rather than being dropped. -- **The identity is read once per repo** - resolved on the first archive and reused for the rest of the daemon's life, with an explicit way to forget it. - -## Business logic - -### Archives are filed per user - -#### User story - -Several people work the same project from their own machines. Each of them wants their finished agents kept, and none of them wants a merge conflict every time somebody else finishes an agent. - -#### Business logic - -An archive is written under a directory named after the user, inside the archive directory `agents/` on the logs branch. Because each person's archives sit under their own directory, two people's histories sit side by side instead of overwriting the same paths. The whole team can see everybody's list; that visibility is intended. - -The identity used is the email that `git config user.email` reports for the project, so there is nothing extra for the user to configure and the directory matches the name that appears on the commits. - -#### Rationale - -Agent state used to be written to `.the-framework/agents/`, which the install-time `.gitignore` keeps untracked. An ordinary `git clean -fdx` therefore deleted every agent a project had ever run, and nothing was recoverable because nothing had ever been committed. Keeping the lasting copy on the logs branch is what makes the history survive. - -### The directory name can never climb out of the archive - -#### User story - -The user's git identity is repo configuration — the framework treats it as untrusted input, because it is joined onto a filesystem path. - -#### Business logic - -The email is trimmed, lowercased, and every character outside a conservative allowed set is replaced by `-`. The resulting name is accepted only if it is non-empty, no longer than 64 characters, and starts with a letter or digit. Requiring a leading letter or digit is what rules out `.`, `..` and dotfile-style names, so no identity can produce a directory that escapes the archive. A name that fails any of these checks falls back to `anonymous` rather than to a guess. The 64-character limit is well past any real address and keeps the archive paths inside the path length limits of every supported platform. - -### A missing identity still gets a home - -#### User story - -A user runs an agent in a repo where they never configured a git identity, and still expects to find the finished agent afterwards. - -#### Business logic - -When the identity is missing or cannot be read, the archive is filed under `anonymous`. Keeping the history under a placeholder is preferred over dropping it. - -### The identity is read once per repo - -#### User story - -The daemon archives many agents over a long uptime and should not pay for re-reading configuration that essentially never changes. - -#### Business logic - -The resolved directory name is remembered per project for the daemon's life. A git identity changes about as often as never, and an agent that outlived such a change would only mean that the next agent files itself correctly. The remembered names can be forgotten explicitly, which is what a daemon that outlives a configuration change uses. - -## Before modifying/creating SPEC.md files - -You must always read and respect https://raw.githubusercontent.com/brillout/sdd/refs/heads/main/sdd.md diff --git a/packages/framework/src/agent-archive.test.BUG-ANALYSIS.md b/packages/framework/src/agent-archive.test.BUG-ANALYSIS.md deleted file mode 100644 index 21e8ae498..000000000 --- a/packages/framework/src/agent-archive.test.BUG-ANALYSIS.md +++ /dev/null @@ -1 +0,0 @@ -Last analyzed: [25 August 2026](https://github.com/framework/the-framework/pull/1698) diff --git a/packages/framework/src/agent-archive.test.SPEC.md b/packages/framework/src/agent-archive.test.SPEC.md deleted file mode 100644 index f4267efcb..000000000 --- a/packages/framework/src/agent-archive.test.SPEC.md +++ /dev/null @@ -1,5 +0,0 @@ -What the tests cover: an email becomes its own archive directory name, trimmed and lowercased; a hostile identity can never produce a name that starts with a dot or holds a path separator, so `.`, `..` and absolute-looking values all fall back to `anonymous`; a missing, blank or absurdly long identity also falls back to `anonymous` rather than dropping the history; the identity is read from git once per project and reused afterwards; a repo where git reports no identity falls back instead of failing. A final test runs against real git and verifies that in an activated project everything under `.the-framework/` — the agent's live event log, the transient archive, and the agent checkouts — stays untracked, with the `.the-framework/.gitignore` marker itself as the one tracked file, so the framework never dirties the default branch. - -## Before modifying/creating SPEC.md files - -You must always read and respect https://raw.githubusercontent.com/brillout/sdd/refs/heads/main/sdd.md diff --git a/packages/framework/src/agent-archive.test.ts b/packages/framework/src/agent-archive.test.ts deleted file mode 100644 index f4056c7b0..000000000 --- a/packages/framework/src/agent-archive.test.ts +++ /dev/null @@ -1,82 +0,0 @@ -import { strict as assert } from 'node:assert' -import { test } from 'node:test' -import { join } from 'node:path' -import type { GitRunner } from '@gemstack/agent-data' -import { userDirName, resolveUserDir, forgetUserDirs, ANONYMOUS_USER_DIR } from './agent-archive.js' -import { frameworkGitignore } from './framework-gitignore.js' - -test('an email becomes its own directory name (#1179)', () => { - assert.equal(userDirName('git@brillout.com'), 'git@brillout.com') - assert.equal(userDirName(' Git@Brillout.COM '), 'git@brillout.com', 'trimmed and lowercased') -}) - -test('a name that could climb out of the directory is refused (#1179)', () => { - // The value comes from repo configuration and is joined onto a path, so this is the one property - // that has to hold: the result can never be `.`, `..`, or anything that starts with a dot. - for (const hostile of ['..', '.', '../../etc/passwd', '.hidden', '/absolute', '..@evil.com']) { - const dir = userDirName(hostile) - assert.ok(!dir.startsWith('.'), `${hostile} -> ${dir} must not start with a dot`) - assert.ok(!dir.includes('/'), `${hostile} -> ${dir} must not hold a separator`) - } - assert.equal(userDirName('..'), ANONYMOUS_USER_DIR) - assert.equal(userDirName('/absolute'), ANONYMOUS_USER_DIR, 'a leading separator has nothing safe left') -}) - -test('no identity still gets a directory, rather than dropping the history (#1179)', () => { - assert.equal(userDirName(undefined), ANONYMOUS_USER_DIR) - assert.equal(userDirName(''), ANONYMOUS_USER_DIR) - assert.equal(userDirName(' '), ANONYMOUS_USER_DIR) - assert.equal(userDirName('a'.repeat(200)), ANONYMOUS_USER_DIR, 'an absurd length is not a directory name') -}) - -test('the identity comes from git, and is read once per repo (#1179)', async () => { - forgetUserDirs() - let calls = 0 - const git: GitRunner = async () => { - calls++ - return 'git@brillout.com\n' - } - assert.equal(await resolveUserDir('/repo', git), 'git@brillout.com') - assert.equal(await resolveUserDir('/repo', git), 'git@brillout.com') - assert.equal(calls, 1, 'cached: this is read on every archive') -}) - -test('git with no identity configured falls back rather than throwing (#1179)', async () => { - forgetUserDirs() - const git: GitRunner = async () => { - throw new Error('no user.email') - } - assert.equal(await resolveUserDir('/nowhere', git), ANONYMOUS_USER_DIR) -}) - -test('against real git: everything under .the-framework is transient on main (#1582)', async () => { - // The lasting records live on the data branch now, so the ignore file is "ignore it all" — a - // session's live state, the transient archive, and the run checkouts must never dirty main. - const { mkdtemp, mkdir, writeFile, rm } = await import('node:fs/promises') - const { tmpdir } = await import('node:os') - const { execFileSync } = await import('node:child_process') - - const repo = await mkdtemp(join(tmpdir(), 'fw-sessions-')) - const git = (...args: string[]): string => execFileSync('git', args, { cwd: repo, encoding: 'utf8' }) - try { - git('init', '-q') - git('config', 'user.email', 'git@example.com') - git('config', 'user.name', 'Test') - - const fw = join(repo, '.the-framework') - await mkdir(join(fw, 'agents'), { recursive: true }) - await mkdir(join(fw, 'branches', 'agent-r9'), { recursive: true }) - await writeFile(join(fw, '.gitignore'), frameworkGitignore()) - await writeFile(join(fw, 'agents', 'old.json'), '{}\n') - await writeFile(join(fw, 'events.jsonl'), '\n') - await writeFile(join(fw, 'branches', 'agent-r9', 'file.txt'), 'x\n') - - const status = git('status', '--porcelain', '-uall') - assert.ok(!status.includes('.the-framework/agents/'), 'the transient archive stays ignored') - assert.ok(!status.includes('.the-framework/events.jsonl'), 'the live log stays ignored') - assert.ok(!status.includes('.branches/'), 'a run checkout stays ignored') - assert.ok(status.includes('.the-framework/.gitignore'), 'the ignore file itself is the one tracked thing') - } finally { - await rm(repo, { recursive: true, force: true }) - } -}) diff --git a/packages/framework/src/agent-archive.ts b/packages/framework/src/agent-archive.ts deleted file mode 100644 index 054e70993..000000000 --- a/packages/framework/src/agent-archive.ts +++ /dev/null @@ -1,72 +0,0 @@ -import { ARCHIVE_DIR } from './store/index.js' -import { nodeGitRunner, type GitRunner } from '@gemstack/agent-data' -/** - * Committed session history (#1179): where a project's finished agents are archived so they survive - * the repo being cleaned. - * - * The bug this exists for: agent state was written to `.the-framework/agents/`, which the install-time - * `.gitignore` keeps untracked, so `git clean -fdx` — an ordinary thing to do to a repo — deleted - * every session a project had ever run. Nothing was recoverable, because nothing had ever been - * committed. Since #1582 the lasting copy lives on the `agents-logs` branch (#1748), as `agents//`. - * - * Scoped per user rather than one shared directory. Two people working the same repo would - * otherwise write the same paths from different machines and conflict on every merge; under their - * own directory their histories simply sit side by side. The list being visible to the whole team - * is the intended outcome, not a leak — see the issue. - * - * The identity is the git `user.email` already configured in the repo, so there is nothing new to - * set up and the directory matches the name on the commits. - */ - -/** The directory, under a user's own directory, that holds their archived agents. */ -export { ARCHIVE_DIR } - -/** Where an agent's history goes when git has no identity configured. */ -export const ANONYMOUS_USER_DIR = 'anonymous' - -/** - * Longest directory name we will make from an email. Well past any real address, and short enough - * that the archive paths under it stay inside the path limits of every platform we run on. - */ -const MAX_USER_DIR = 64 - -/** - * An email as a directory name: lowercased, with anything outside a conservative set replaced by - * `-`. The result must start with a letter or digit, which is what rules out `.`, `..` and dotfile - * names — this value comes from repo configuration and is joined onto a path, so a name that could - * climb out of the directory is the one thing that must be impossible. Anything that cannot be made - * to fit falls back to {@link ANONYMOUS_USER_DIR} rather than to a guess. - */ -export function userDirName(email: string | undefined): string { - const cleaned = (email ?? '') - .trim() - .toLowerCase() - .replace(/[^a-z0-9@._+-]/g, '-') - return cleaned.length > 0 && cleaned.length <= MAX_USER_DIR && /^[a-z0-9][a-z0-9@._+-]*$/.test(cleaned) - ? cleaned - : ANONYMOUS_USER_DIR -} - -/** - * The directory name for the identity this repo commits under, from `git config user.email`. - * - * Cached per repo for the process's life: it is read on every archive, it changes about as often - * as a git identity does, and an agent that outlived a config change would only mean the next agent - * files itself correctly. A missing or unreadable identity yields {@link ANONYMOUS_USER_DIR}, so - * history is still kept — filing it under a placeholder is strictly better than dropping it. - */ -const cache = new Map() - -export async function resolveUserDir(cwd: string, git: GitRunner = nodeGitRunner()): Promise { - const hit = cache.get(cwd) - if (hit !== undefined) return hit - const email = await git(['config', 'user.email'], cwd).catch(() => '') - const dir = userDirName(email.trim()) - cache.set(cwd, dir) - return dir -} - -/** Drop the {@link resolveUserDir} cache. For tests, and for a daemon that outlives a config change. */ -export function forgetUserDirs(): void { - cache.clear() -} diff --git a/packages/framework/src/archived-agent-patch.BUG-ANALYSIS.md b/packages/framework/src/archived-agent-patch.BUG-ANALYSIS.md deleted file mode 100644 index 21e8ae498..000000000 --- a/packages/framework/src/archived-agent-patch.BUG-ANALYSIS.md +++ /dev/null @@ -1 +0,0 @@ -Last analyzed: [25 August 2026](https://github.com/framework/the-framework/pull/1698) diff --git a/packages/framework/src/archived-agent-patch.SPEC.md b/packages/framework/src/archived-agent-patch.SPEC.md deleted file mode 100644 index abd9aaa04..000000000 --- a/packages/framework/src/archived-agent-patch.SPEC.md +++ /dev/null @@ -1,7 +0,0 @@ -Records a late fact onto a finished agent's archive — its record on the logs branch — by putting the patch through that branch's single write cycle: sync with origin, apply, commit, push. Adoption uses this to record a cloud session's actual branch and PR onto the waiting agent's archive. - -The funnel is the point: a patch written straight into the branch's checkout is not a fact yet — the next sync refuses a dirty tree and hard-resets it, so the edit would vanish within a minute without any other machine ever seeing it. The outcome is true when the record now carries the patch, committed; a push that could not go out rides the next cycle, per the funnel's own owed-push rule. An agent with no archive to patch reports false, and nothing is committed. - -## Before modifying/creating SPEC.md files - -You must always read and respect https://raw.githubusercontent.com/brillout/sdd/refs/heads/main/sdd.md diff --git a/packages/framework/src/archived-agent-patch.test.BUG-ANALYSIS.md b/packages/framework/src/archived-agent-patch.test.BUG-ANALYSIS.md deleted file mode 100644 index 21e8ae498..000000000 --- a/packages/framework/src/archived-agent-patch.test.BUG-ANALYSIS.md +++ /dev/null @@ -1 +0,0 @@ -Last analyzed: [25 August 2026](https://github.com/framework/the-framework/pull/1698) diff --git a/packages/framework/src/archived-agent-patch.test.SPEC.md b/packages/framework/src/archived-agent-patch.test.SPEC.md deleted file mode 100644 index 5cbb10f47..000000000 --- a/packages/framework/src/archived-agent-patch.test.SPEC.md +++ /dev/null @@ -1,5 +0,0 @@ -What the tests cover: patching a finished agent's archive lands as a commit on the logs branch — pushed, leaving that branch's checkout clean — and the patched record (branch and PR) survives the next sync of the branch; patching an agent that has no archive reports not-patched and commits nothing. - -## Before modifying/creating SPEC.md files - -You must always read and respect https://raw.githubusercontent.com/brillout/sdd/refs/heads/main/sdd.md diff --git a/packages/framework/src/archived-agent-patch.test.ts b/packages/framework/src/archived-agent-patch.test.ts deleted file mode 100644 index 998fde1de..000000000 --- a/packages/framework/src/archived-agent-patch.test.ts +++ /dev/null @@ -1,74 +0,0 @@ -import { strict as assert } from 'node:assert' -import { test } from 'node:test' -import { mkdtemp, mkdir, readFile, rm, writeFile } from 'node:fs/promises' -import { tmpdir } from 'node:os' -import { join } from 'node:path' -import { execFile } from 'node:child_process' -import { promisify } from 'node:util' -import { fileBranchPath, withFileBranch } from '@gemstack/agent-data' -import { LOGS_BRANCH } from './framework-dir.js' -import { patchArchivedAgentOnDataBranch } from './archived-agent-patch.js' - -const git = promisify(execFile) - -/** A real repo with a bare origin and one archived run seeded on the data branch. */ -async function repoWithArchive(): Promise<{ project: string; cleanup: () => Promise }> { - const project = await mkdtemp(join(tmpdir(), 'framework-archive-patch-')) - const origin = await mkdtemp(join(tmpdir(), 'framework-archive-patch-origin-')) - await git('git', ['init', '-q', '--bare'], { cwd: origin }) - await git('git', ['init', '-q', '-b', 'main'], { cwd: project }) - await git('git', ['config', 'user.email', 'test@example.com'], { cwd: project }) - await git('git', ['config', 'user.name', 'Test'], { cwd: project }) - await git('git', ['config', 'commit.gpgsign', 'false'], { cwd: project }) - await git('git', ['remote', 'add', 'origin', origin], { cwd: project }) - const seeded = await withFileBranch(project, LOGS_BRANCH, 'seed', async dir => { - await mkdir(join(dir, 'agents', 'u'), { recursive: true }) - const meta = { version: 1, status: 'done', id: 'r1', startedAt: '2026-08-20T10:00:00.000Z', updatedAt: '2026-08-20T10:00:00.000Z', branch: 'agent-r1' } - await writeFile(join(dir, 'agents', 'u', 'r1.json'), JSON.stringify(meta)) - await writeFile(join(dir, 'agents', 'u', 'r1.jsonl'), '') - }) - assert.ok(seeded.ok && seeded.pushed, 'the fixture archive must land on origin') - return { - project, - cleanup: async () => { - await rm(project, { recursive: true, force: true }) - await rm(origin, { recursive: true, force: true }) - }, - } -} - -test('an archive patch lands as a commit on the data branch, pushed, leaving the checkout clean (#1601)', async () => { - const { project, cleanup } = await repoWithArchive() - try { - const dir = fileBranchPath(project, LOGS_BRANCH) - assert.equal( - await patchArchivedAgentOnDataBranch(project, 'r1', { branch: 'claude/fix-it', pr: { number: 7, url: 'https://x/pull/7' } }, '[The Framework] adopt r1'), - true, - ) - const meta = JSON.parse(await readFile(join(dir, 'agents', 'u', 'r1.json'), 'utf8')) as { branch: string; pr: { number: number } } - assert.equal(meta.branch, 'claude/fix-it') - assert.equal(meta.pr.number, 7) - // Committed, not merely written: a dirty checkout is what the next sync hard-resets. - assert.equal((await git('git', ['status', '--porcelain'], { cwd: dir })).stdout.trim(), '') - assert.equal((await git('git', ['log', '-1', '--format=%s'], { cwd: dir })).stdout.trim(), '[The Framework] adopt r1') - assert.equal((await git('git', ['rev-list', '--count', `origin/${LOGS_BRANCH}..${LOGS_BRANCH}`], { cwd: dir })).stdout.trim(), '0', 'and pushed') - // The sync the daemon runs a minute later keeps it. - await withFileBranch(project, LOGS_BRANCH, 'sync', async () => {}) - const after = JSON.parse(await readFile(join(dir, 'agents', 'u', 'r1.json'), 'utf8')) as { branch: string } - assert.equal(after.branch, 'claude/fix-it') - } finally { - await cleanup() - } -}) - -test('a run with no archive is reported as not patched, and nothing is committed (#1601)', async () => { - const { project, cleanup } = await repoWithArchive() - try { - const dir = fileBranchPath(project, LOGS_BRANCH) - const before = (await git('git', ['rev-parse', 'HEAD'], { cwd: dir })).stdout.trim() - assert.equal(await patchArchivedAgentOnDataBranch(project, 'nope', { branch: 'claude/x' }, '[The Framework] adopt nope'), false) - assert.equal((await git('git', ['rev-parse', 'HEAD'], { cwd: dir })).stdout.trim(), before) - } finally { - await cleanup() - } -}) diff --git a/packages/framework/src/archived-agent-patch.ts b/packages/framework/src/archived-agent-patch.ts deleted file mode 100644 index e6e562820..000000000 --- a/packages/framework/src/archived-agent-patch.ts +++ /dev/null @@ -1,27 +0,0 @@ -import { withFileBranch } from '@gemstack/agent-data' -import { LOGS_BRANCH } from './framework-dir.js' -import { patchArchivedAgent, type ArchivePatch } from './store/index.js' - -/** - * Patch a settled run's archived record on the logs branch (#1601): synced with origin, patched, - * committed, pushed — the same funnel every other write to the branch goes through (#1582). - * - * A patch written straight into the branch's checkout is not a fact yet: the next sync's rebase - * refuses a dirty tree and the funnel hard-resets it, so the patch is gone within a minute and - * no other machine ever saw it. Seen live on the cloud-work adoption before it went through here. - * - * True when the record now carries the patch, committed; a push that could not go out rides the - * next cycle, which is the funnel's owed-push rule. False when the run has no archive to patch. - */ -export async function patchArchivedAgentOnDataBranch( - cwd: string, - agentId: string, - patch: ArchivePatch, - message: string, -): Promise { - let patched = false - const result = await withFileBranch(cwd, LOGS_BRANCH, message, async () => { - patched = await patchArchivedAgent(cwd, agentId, patch) - }) - return patched && (result.ok || result.committed) -} diff --git a/packages/framework/src/auto-pm.SPEC.md b/packages/framework/src/auto-pm.SPEC.md index bd284ab23..21c7304be 100644 --- a/packages/framework/src/auto-pm.SPEC.md +++ b/packages/framework/src/auto-pm.SPEC.md @@ -9,7 +9,7 @@ Auto PM: the policy that decides whether the daemon may spend leftover quota on ## Glossary - **ended dry** - work Auto PM already gave an agent that finished without producing anything to hand off. It is not handed out again for the rest of the daemon's lifetime. -- **routine lock** - a routine's lock file on The Framework's `agents-logs` branch, naming the machine that is running that routine and since when, so no two machines run it at once. A routine declares whether it holds one. +- **routine lock** - a routine's lock file on the `agent-data` branch, naming the machine that is running that routine and since when, so no two machines run it at once. A routine declares whether it holds one. ## Business logic — TL;DR diff --git a/packages/framework/src/auto-pm.ts b/packages/framework/src/auto-pm.ts index cbaca6ce4..36d29d823 100644 --- a/packages/framework/src/auto-pm.ts +++ b/packages/framework/src/auto-pm.ts @@ -201,7 +201,7 @@ export interface AutoPmJob { entry?: string /** * The routine lock this job holds while it runs (#1659), as `routines/.lock.md` on the - * agents-logs branch: the triage routines rewrite the shared queue and may take hours, so no two may + * agent-data branch: the triage routines rewrite the shared queue and may take hours, so no two may * run at once, on any machine. The sweep mints it before the start and releases it when the * run ends. Declared as data on the job, like {@link AutoPmJob.drains}, so the sweep never * matches on {@link AutoPmJob.name} at the call site. @@ -544,7 +544,7 @@ export interface AutoPmDeps { start(project: AutoPmProject, job: AutoPmJob): Promise /** * Take a job's {@link AutoPmJob.lock} before its run starts (#1659): `routines/.lock.md` - * on the agents-logs branch, pushed, so every machine sharing it sees the routine as taken. `ok: false` + * on the agent-data branch, pushed, so every machine sharing it sees the routine as taken. `ok: false` * stands the job down with the reason — the lock's holder, or a write that could not land. * Omitted, the job starts unguarded. */ diff --git a/packages/framework/src/cli.SPEC.md b/packages/framework/src/cli.SPEC.md index bb4c51036..f1a9e0de1 100644 --- a/packages/framework/src/cli.SPEC.md +++ b/packages/framework/src/cli.SPEC.md @@ -216,7 +216,7 @@ The pull request the framework opens carries the agent's own title and descripti The outcome is both recorded and narrated: the pull request's number and URL are recorded so every later surface reads them off the agent instead of re-deriving them from branch names and timestamps, and the terminal gets a line for what happened — opened, pushed, auto-merge armed, merge-on-green watched, merged, withheld with the reason, or failed. -The quality step runs first, the handoff second, and the archive last. Both must precede the archive because archiving copies the event log to the logs branch, and an outcome recorded afterwards would be missing from the copy the dashboard's history reads; and the handoff must follow the quality step so whatever that step committed is part of what gets published. +The quality step runs first, the handoff second, and the archive last. Both must precede the archive because archiving records the event log on the data branch, and an outcome recorded afterwards would be missing from the copy the dashboard's history reads; and the handoff must follow the quality step so whatever that step committed is part of what gets published. #### Rationale diff --git a/packages/framework/src/cli.test.ts b/packages/framework/src/cli.test.ts index d3f879d3d..37df0821b 100644 --- a/packages/framework/src/cli.test.ts +++ b/packages/framework/src/cli.test.ts @@ -304,7 +304,7 @@ test('runCli refuses a session whose repo records another layout, before writing const { io, err } = capture() const cwd = await mkdtemp(join(tmpdir(), 'framework-layout-skew-')) await mkdir(join(cwd, '.the-framework'), { recursive: true }) - await writeFile(layoutMarkerPath(cwd), layoutMarker().replace('archive-dir: agents', 'archive-dir: sessions')) + await writeFile(layoutMarkerPath(cwd), layoutMarker().replace('runs-dir: agents', 'runs-dir: sessions')) const code = await runAgentCli({ prompt: 'say hi', kind: 'prompt', cwd }, io) assert.equal(code, 1) assert.ok(err.some(l => /#1575/.test(l))) diff --git a/packages/framework/src/cloud-work.SPEC.md b/packages/framework/src/cloud-work.SPEC.md index b6fe810e0..41c8ed418 100644 --- a/packages/framework/src/cloud-work.SPEC.md +++ b/packages/framework/src/cloud-work.SPEC.md @@ -7,7 +7,7 @@ Adoption: the pass that matches a cloud session's actual `claude/*` branch back ## Business logic — TL;DR - **Ancestry makes the match exact** - a `claude/*` head on origin belongs to the agent whose hand-off anchor it descends from; exactly one descendant adopts, zero or several adopt nothing and are retried next pass. -- **What is learned lands as one commit** - the branch (first time only) and the PR (once known) are recorded onto the agent's archive on the logs branch; nothing learned, nothing written. +- **What is learned lands as one commit** - the branch (first time only) and the PR (once known) are recorded onto the agent's run card on the data branch, through the `logs` skill; nothing learned, nothing written. - **The armed draft PR finally opens** - an agent set to open a PR whose session pushed work but never opened one gets its draft PR opened by this pass. - **"None" and "could not tell" never look alike** - a PR listing that fails records the branch but opens nothing this pass, so a transient failure can never produce a second PR on a branch that already has one. - **Bounded and quiet** - only settled `web`-target agents started within the last 48 hours are asked about, the archive is read by that window so old history costs nothing, and only adoptions and failures are logged. @@ -36,7 +36,7 @@ The publish level the user armed the agent with must still come true when the se For the matched branch, the session's own PR is looked up from the branch's PR history, filtered by the agent's start time so a predecessor's PR on a reused branch name is never this agent's, latest one wins. When the listing succeeds and finds none, the agent finished, its handoff includes the PR stage, and the head carries commits beyond the anchor itself, the pass opens the draft PR the agent's own epilogue never could (it saw only the empty agent branch) — the armed handoff finally resolving against the facts. A head that *is* the anchor gets no PR: the session pushed nothing, and a PR over nothing helps nobody. A listing that fails is a reported failure, records the branch (a fact regardless), opens nothing this pass, and leaves the agent to be asked again — "the session opened no PR" and "the listing could not be read" must not look alike, because guessing cost a duplicate draft PR once. -Whatever the pass learned — the branch on first adoption, the PR once known — is recorded onto the agent's archive as one commit on the logs branch. A record that cannot be written is a reported failure and is retried. +Whatever the pass learned — the branch on first adoption, the PR once known — is recorded onto the agent's run card as one commit on the data branch, through the `logs` skill. A record that cannot be written is a reported failure and is retried. ### The adoption pass as a daemon service diff --git a/packages/framework/src/cloud-work.ts b/packages/framework/src/cloud-work.ts index b0f254707..fd931b1f3 100644 --- a/packages/framework/src/cloud-work.ts +++ b/packages/framework/src/cloud-work.ts @@ -2,8 +2,8 @@ import { nodeGitRunner, type GitRunner } from '@gemstack/agent-data' import { agentBranchName } from '@gemstack/skill-branches' import { ghPrsForBranchOrThrow, pickAgentPr, type LinkedPr } from './dashboard/gh.js' import { openRemoteBranchPullRequest, type HandoffResult } from './dashboard/agent-handoff.js' -import { listAgents, nodeStoreFs, startedAtFromAgentId, type AgentMeta, type ArchivePatch } from './store/index.js' -import { patchArchivedAgentOnDataBranch } from './archived-agent-patch.js' +import { listAgents, nodeStoreFs, startedAtFromAgentId, type AgentMeta } from './store/index.js' +import { patchRun, type RunPatch } from '@gemstack/skill-logs' import { errorMessage } from './error-message.js' import { startProjectPass, type ProjectPass, type ProjectsSource } from './project-pass.js' @@ -55,8 +55,8 @@ export interface CloudWorkDeps { prs?: (cwd: string, branch: string) => Promise /** The project's run records, none older than `since` in epoch ms (default {@link listAgents}). */ agents?: (cwd: string, since: number) => Promise - /** Record the adopted branch and PR on the run's archive (default {@link patchArchivedAgentOnDataBranch}). */ - patch?: (cwd: string, agentId: string, patch: ArchivePatch, message: string) => Promise + /** Record the adopted branch and PR on the run's card, as one commit on the data branch (default the `logs` skill's {@link patchRun}). */ + patch?: (cwd: string, agentId: string, patch: RunPatch) => Promise /** Open the armed draft PR for a remote-only branch (default {@link openRemoteBranchPullRequest}). */ openPr?: (cwd: string, agent: AgentMeta, branch: string) => Promise /** The current time in ms (injected so tests can age runs deterministically). */ @@ -149,7 +149,7 @@ export async function adoptCloudWork(cwd: string, deps: CloudWorkDeps = {}): Pro const git = deps.git ?? nodeGitRunner() const prs = deps.prs ?? ghPrsForBranchOrThrow const agents = deps.agents ?? ((project: string, since: number) => listAgents(project, nodeStoreFs(), since)) - const patchArchive = deps.patch ?? patchArchivedAgentOnDataBranch + const patchArchive = deps.patch ?? patchRun const openPr = deps.openPr ?? openRemoteBranchPullRequest const now = deps.now ? deps.now() : Date.now() const result: CloudWorkResult = { adopted: [], failed: [] } @@ -207,12 +207,12 @@ export async function adoptCloudWork(cwd: string, deps: CloudWorkDeps = {}): Pro // One commit on the data branch carries whatever this pass learned: the branch (first time // only), the PR (once known). Nothing learned, nothing written — and nothing announced. - const patch: ArchivePatch = { + const patch: RunPatch = { ...(onBirthBranch(run) ? { branch } : {}), ...(pr && run.pr === undefined ? { pr: { number: pr.number, url: pr.url } } : {}), } if (Object.keys(patch).length === 0) continue - if (!(await patchArchive(cwd, run.id, patch, `[The Framework] adopt session ${run.id}'s cloud work`))) { + if (!(await patchArchive(cwd, run.id, patch))) { result.failed.push({ agentId: run.id, error: `could not record ${branch} on the run's archive` }) continue } diff --git a/packages/framework/src/daemon-runtime.SPEC.md b/packages/framework/src/daemon-runtime.SPEC.md index 0a2f83991..ad810759f 100644 --- a/packages/framework/src/daemon-runtime.SPEC.md +++ b/packages/framework/src/daemon-runtime.SPEC.md @@ -15,7 +15,7 @@ Starting, supervising and retiring agents on behalf of the daemon: every agent t - **One agent per checkout** - a second agent aimed at a checkout that is already busy is refused as busy; different checkouts never wait on each other. - **An agent that never booted is reported as failed** - when the spawned process dies before writing its own status, the daemon writes the `failed` status for it and puts the process's error output into the agent's event log — unless the checkout is gone, where only the terminal records it. - **One more try after a transient driver death** - an agent killed by a dropped connection or an overloaded API is continued automatically, at most twice; any other failure stands. -- **Retiring a finished agent** - its history is archived onto the logs branch, then its checkout is removed once its work has reached the remote, and kept otherwise; any branch that went with it is named. +- **Retiring a finished agent** - its history is recorded on the data branch as the `logs` skill's run, then its checkout is removed once its work has reached the remote, and kept otherwise; any branch that went with it is named. - **Running an agent on another device** - a start aimed at a device is handed to that daemon over the relay, its events are streamed back, and read/steer requests for it are forwarded there. - **Adding projects** - the repo at a given path is activated and registered in one go. - **Shutdown stops every agent it spawned** - Ctrl-C terminates each agent's process — an agent that will not go is killed together with its whole process group, its browser included — and waits for its teardown to finish before the daemon lets go of the repo; starts landing during shutdown are refused. @@ -50,7 +50,7 @@ The user runs several agents on the same project at once, and keeps working in t Each agent is given its own git worktree under the project's `.branches/`, on its own `agent-` branch. Concurrent agents on one project therefore never fight over a working tree, and the user's own checkout — uncommitted work included — is left untouched. -A fresh worktree has no installed dependencies, since those are not tracked by git, so the project's are mirrored in (the `skill-branches` package's dependency linking). The `tickets` and `queue` skills are linked into every checkout the daemon creates or re-attaches, where the agent's harness looks for skills — a temporary arrangement, until skills are committed into the repository itself and every checkout carries them as tracked files. +A fresh worktree has no installed dependencies, since those are not tracked by git, so the project's are mirrored in (the `skill-branches` package's dependency linking). The `tickets`, `queue` and `logs` skills are linked into every checkout the daemon creates or re-attaches, where the agent's harness looks for skills — a temporary arrangement, until skills are committed into the repository itself and every checkout carries them as tracked files. #### Rationale @@ -150,7 +150,7 @@ An agent finishes. Its work must survive, its history must stay visible in the d #### Business logic -When an agent's process exits, its history — which lives inside its own checkout — is copied out into the project first, as an archive filed under the identity the repo commits as, on the logs branch, through the logs branch's single write cycle so it is committed and pushed the moment it lands. The branch the work ended on is recorded with it, because the branch outlives the checkout and is the only handle the dashboard has left on a finished agent. That branch is read only from a directory git knows as a checkout in its own right: a leftover directory would answer with the *enclosing* repository's branch, and the archive would record the user's own branch as the agent's. +When an agent's process exits, its history — which lives inside its own checkout — is read out first and recorded on the data branch as the `logs` skill's run: the agent meta as the run's card, with the skill's eleven fields on top and the rest of the meta under the one key the skill keeps for its caller, and the event log as the run's diary, with what the agent said, its result, its ending and its cost as the four kinds of line the skill knows and every other event as it is. The skill files it under the identity the repo commits as and writes it through the branch's single write cycle, so it is committed and pushed the moment it lands. The branch the work ended on is recorded with it, because the branch outlives the checkout and is the only handle the dashboard has left on a finished agent. That branch is read only from a directory git knows as a checkout in its own right: a leftover directory would answer with the *enclosing* repository's branch, and the archive would record the user's own branch as the agent's. Then the checkout goes, under one rule: it is removed once its work is on the remote, whatever state the agent ended in — pending changes are committed and the branch is pushed, unless everything the checkout holds is already there, and only then does the checkout come off disk. A push that cannot land keeps the checkout, and a later sweep retries it. Teardown, the sweep and the dashboard's Remove button are the same behaviour by construction. When branches went with the checkout — one that provably held nothing the remote lacks, or the `agent-` branch the agent had branched away from — the teardown names them, because a branch disappearing unannounced reads as a bug. diff --git a/packages/framework/src/daemon-runtime.ts b/packages/framework/src/daemon-runtime.ts index 70b84987a..b7d0ad2df 100644 --- a/packages/framework/src/daemon-runtime.ts +++ b/packages/framework/src/daemon-runtime.ts @@ -2,12 +2,13 @@ import { spawn, type ChildProcess } from 'node:child_process' import { closeSync, mkdirSync, openSync } from 'node:fs' import { basename, delimiter, dirname, join, resolve } from 'node:path' import { appendFile, mkdir, readFile, rm, stat, writeFile } from 'node:fs/promises' -import { agentIdFromStartedAt, startedAtFromAgentId, archiveWorktreeAgent, restoreArchivedAgent, listAgents, findAgent, archivedAgentPaths, readLiveMetas, readLiveMeta, resolveAgentEventsPath, EVENTS_FILE, META_FILE, isPidAlive, type AgentMeta } from './store/index.js' -import { isGitRepo, nodeGitRunner, isGitTimeout, withFileBranch } from '@gemstack/agent-data' +import { agentIdFromStartedAt, startedAtFromAgentId, readWorktreeAgent, restoreArchivedAgent, listAgents, findAgent, archivedAgentPaths, readLiveMetas, readLiveMeta, resolveAgentEventsPath, EVENTS_FILE, META_FILE, isPidAlive, toRunCard, diaryOf, fromDiaryLine, type AgentMeta } from './store/index.js' +import { isGitRepo, nodeGitRunner, isGitTimeout } from '@gemstack/agent-data' import { createCheckout, attachCheckout, agentBranchName, worktreePath, worktreeBranch, removeWorktree, pruneWorktrees, agentIdFromWorktreeDir, CLI_BIN_DIR as BRANCHES_BIN_DIR } from '@gemstack/skill-branches' import { isTicketPath, CLI_BIN_DIR as TICKETS_BIN_DIR, SKILL_DIR as TICKETS_SKILL_DIR, SKILL_NAME as TICKETS_SKILL_NAME, AGENT_ID_ENV } from '@gemstack/skill-tickets' import { CLI_BIN_DIR as QUEUE_BIN_DIR, SKILL_DIR as QUEUE_SKILL_DIR, SKILL_NAME as QUEUE_SKILL_NAME } from '@gemstack/skill-queue' -import { LOGS_BRANCH, THE_FRAMEWORK_DIR } from './framework-dir.js' +import { CLI_BIN_DIR as LOGS_BIN_DIR, SKILL_DIR as LOGS_SKILL_DIR, SKILL_NAME as LOGS_SKILL_NAME, writeRun, type AnyDiaryLine } from '@gemstack/skill-logs' +import { THE_FRAMEWORK_DIR } from './framework-dir.js' import type { FrameworkEvent } from './events.js' import { removeAgentSpec, writeAgentSpec } from './agent-spec.js' import type { StartAgentKind, StartAgentOptions, StartAgentResult, AddProjectResult } from './dashboard/index.js' @@ -16,7 +17,6 @@ import { RelayedAgents, startRemoteAgent } from './dashboard/remote-run.js' import { agentBranchFor } from './dashboard/agent-handoff.js' import { dispatchRelayRpc } from './dashboard-rpc/relay-dispatch.js' import { tailEvents, tailAgentEvents } from './dashboard-rpc/events-tail.js' -import { resolveUserDir } from './agent-archive.js' import { removeProjectWorktree } from './worktrees.js' import { describeDeleted } from './merged-worktrees.js' import { scopedKey, parseScopedKey, keyBelongsTo } from './runtime-keys.js' @@ -107,25 +107,27 @@ function spawnDetached(binPath: string, specPath: string, stderrFile?: string, e } /** - * A spawned run's environment: ours, with the `branches`, `tickets` and `queue` commands on its - * PATH (#1725/#1748) — the agent names its session, checks its tree, and reads and changes the - * tickets and the queue through the same packages the daemon does — its id as `AGENT_ID`, so a - * claim it makes names the agent and not its branch, plus the daemon's URL when it has one (#1328). + * A spawned run's environment: ours, with the `branches`, `tickets`, `queue` and `logs` commands on + * its PATH (#1725/#1748/#1769) — the agent names its session, checks its tree, reads and changes + * the tickets and the queue, and reads past runs through the same packages the daemon does — its + * id as `AGENT_ID`, so a claim it makes names the agent and not its branch, plus the daemon's URL + * when it has one (#1328). */ function childEnv(daemonUrl: string | undefined, agentId: string | undefined, base: NodeJS.ProcessEnv = process.env): NodeJS.ProcessEnv { - const env: NodeJS.ProcessEnv = { ...base, PATH: [BRANCHES_BIN_DIR, TICKETS_BIN_DIR, QUEUE_BIN_DIR, base['PATH']].filter(Boolean).join(delimiter) } + const env: NodeJS.ProcessEnv = { ...base, PATH: [BRANCHES_BIN_DIR, TICKETS_BIN_DIR, QUEUE_BIN_DIR, LOGS_BIN_DIR, base['PATH']].filter(Boolean).join(delimiter) } if (agentId) env[AGENT_ID_ENV] = agentId return daemonUrl ? { ...env, [DAEMON_URL_ENV]: daemonUrl } : env } /** - * TEMPORARY (#1748): the `tickets` and `queue` skills linked into every checkout beside the + * TEMPORARY (#1748): the `tickets`, `queue` and `logs` skills linked into every checkout beside the * `branches` skill, through the branches package's caller-given list. Gone when use-npm-skills * commits the skills into the repository, where a checkout carries them as tracked files. */ const CHECKOUT_SKILLS = [ { name: TICKETS_SKILL_NAME, dir: TICKETS_SKILL_DIR }, { name: QUEUE_SKILL_NAME, dir: QUEUE_SKILL_DIR }, + { name: LOGS_SKILL_NAME, dir: LOGS_SKILL_DIR }, ] /** Where a spawned agent's stderr lands (#1261), so a child that dies at boot leaves a trace. */ @@ -200,7 +202,8 @@ export function lastAgentFailureDetail(eventsJsonl: string): string | undefined for (const line of eventsJsonl.split('\n')) { if (!line.trim()) continue try { - const event = JSON.parse(line) as { kind?: string; ok?: boolean; detail?: string } + // A live log or a recorded diary: the run's ending reads the same through the skill's mapping. + const event = fromDiaryLine(JSON.parse(line) as AnyDiaryLine) if (event.kind === 'end') detail = event.ok === false ? event.detail : undefined } catch { // A malformed line is not this reader's problem; the events around it still count. @@ -593,20 +596,17 @@ export function createProjectRuntime({ cwd, env, binPath, retryDelayMs, driverPr // only from a worktree root (#1654): a directory that is no longer one would answer with // the enclosing repo's branch, and the archive would record the user's `main` as the run's. const branch = await worktreeBranch(worktree) - // Filed under the identity this repo commits as, onto the logs branch (#1179/#1582) - // through its write funnel: the archive is committed and pushed the moment it lands — - // durable without a human, and never a commit on main. - const user = await resolveUserDir(projectCwd) - const archived = await withFileBranch( - projectCwd, - LOGS_BRANCH, - `[The Framework] archive session ${agentId ?? agentIdFromWorktreeDir(basename(worktree))}`, - async () => { - await archiveWorktreeAgent(worktree, projectCwd, undefined, branch, user) - }, - ) - if (!archived.ok && !archived.committed) - console.log(`[framework] could not archive session ${basename(worktree)}: ${archived.error}`) + // Recorded as the `logs` skill's run on the data branch (#1179/#1582/#1769), under the + // identity this repo commits as, through the branch's write funnel: the record is committed + // and pushed the moment it lands — durable without a human, and never a commit on main. + // The card is the skill's shape with the rest of the meta under `caller`; the diary is the + // event log with the four kinds the skill knows mapped onto its lines. + const run = await readWorktreeAgent(worktree, undefined, branch) + if (run) { + const archived = await writeRun(projectCwd, toRunCard(run.meta), diaryOf(run.events)) + if (!archived.ok && !archived.committed) + console.log(`[framework] could not archive session ${basename(worktree)}: ${archived.error}`) + } // One rule (E5): the checkout goes once its work is on the remote, whatever state the agent // ended in. `removeProjectWorktree` owns the whole sequence — keep a dirty checkout, push // the branch, remove only if the remote has it — so teardown, the sweep and the dashboard's diff --git a/packages/framework/src/daemon-services.SPEC.md b/packages/framework/src/daemon-services.SPEC.md index 1c124d5fd..6a7b339cb 100644 --- a/packages/framework/src/daemon-services.SPEC.md +++ b/packages/framework/src/daemon-services.SPEC.md @@ -13,7 +13,7 @@ The user closes the dashboard and walks away. While nobody is watching, the mach - **Auto PM's wiring** - which routines may run, how many agents at once, which routine holds its lock while it runs, which tickets may be claimed, who writes the claims, and when a drained queue entry is taken off the queue. - **The CI watch's fix half asks the same consent as Auto PM** - merging a green pull request is ungated, but starting a fix agent needs the Auto PM preference and quota headroom. - **Discord notifications, rebuilt on demand** - a token pasted into the dashboard takes effect immediately, and toggling the preference never replays the backlog. -- **Keeping every project's branches converged** - each project's `agent-data` branch and `agents-logs` branch are converged with origin regularly, and a project that cannot converge either of them is recorded as an error for the dashboard to show. +- **Keeping every project's branch converged** - each project's `agent-data` branch is converged with origin regularly, and a project that cannot converge it is recorded as an error for the dashboard to show. - **A project's agent options** - the user's own settings, with the repo's committed config on top; the same options a hand-started agent gets. - **Quiescing before shutdown** - everything that could start or steer an agent stops first, and the turn already in flight is waited out. @@ -41,7 +41,7 @@ The job list and its cadences: - **worktree sweep**, every ten minutes — reclaims the checkout of an agent whose work has reached the remote; the branch and the agent's record are kept, so this frees disk rather than discarding work. It is the retry for a push that could not land at teardown. - **branches view**, every ten minutes — keeps one symlink per checkout under `.branches/`, named after its branch. -- **data sync**, every other tick — converges each project's two branches with origin: first the `agent-data` branch, which the `tickets` skill sets up as well as pulls (its checkout, the repository-root link to the tickets) and the `queue` skill then seeds and pulls (the queue file), then the `agents-logs` branch. This machine ends up seeing what other machines and cloud sessions pushed, and anything a failed write cycle left local is carried out. +- **data sync**, every other tick — converges each project's `agent-data` branch with origin: the `tickets` skill sets it up as well as pulls it (its checkout, the repository-root link to the tickets) and the `queue` skill then seeds and pulls it (the queue file); the runs and the routine locks ride on the same branch, so one pull carries them. This machine ends up seeing what other machines and cloud sessions pushed, and anything a failed write cycle left local is carried out. - **CI watch**, every other tick — roughly a minute, which is the latency chosen for noticing a check result. - **Discord watchers**, every other tick. - **Auto PM**, every ten minutes. @@ -89,7 +89,7 @@ Auto PM is given everything it needs to decide, and everything it decides is app - **Consent.** The `autoPm` preference is read on every turn rather than at start-up, so switching it off takes effect at the next turn without a restart. The routines the user unticked are read the same way. Both are global rather than per project — the rotation is one schedule for the machine, not one per repo — as is the cap on how many agents a routine may keep going per project. - **What the projects look like.** The queue is handed over as its open entries rather than as a bare "empty or not", because a batch of concurrent drains pins one entry each and the decision needs the entries themselves. The agents currently held on a project are handed over as one label per slot — the agent's id with its process id — so a stand-down or fan-out line names what it was measured against instead of printing a bare number. - **The maintenance sweep's schedule** is a file in the project's checkout rather than in-memory state, because unlike the routine rotation it must survive a daemon restart: a machine rebooted daily would otherwise sweep every morning and never reach its interval. -- **Locking a routine.** A routine that must not run twice at once takes its routine lock on the `agents-logs` branch before its agent starts, and drops it when its run ends, with both outcomes logged — including a release that could not be committed, which stays held for the next turn to retry. On a project's first turn, the routine locks a previous daemon on this machine left behind are released as well, unless one of this machine's own agents started since a given lock was taken is still running. +- **Locking a routine.** A routine that must not run twice at once takes its routine lock on the `agent-data` branch before its agent starts, and drops it when its run ends, with both outcomes logged — including a release that could not be committed, which stays held for the next turn to retry. On a project's first turn, the routine locks a previous daemon on this machine left behind are released as well, unless one of this machine's own agents started since a given lock was taken is still running. - **Which tickets may be planned.** Only tickets that have no plan yet and are not claimed by a lock, most important first by priority. Locks are never expired on a timer: a lock stands until the agent holding it lifts it — releasing the ticket once its plan is written, or closing the ticket once its work lands — or a human releases it from the dashboard. - **Who writes the claims.** The daemon claims the tickets through the `tickets` skill and pushes them, never the agent — an agent only pushes at the end onto its own branch, and a claim that stayed local would not reach the other machines it exists for. The holder each claim names is the id of the agent the sweep is about to start with it. A drain claims what it is about to implement under the same rule, except that it skips only on an existing lock: the plan it also finds is the drain's input, not a rival. - **Releasing a dead claim.** The one claim the daemon can know is dead is the one whose agent settled with nothing to hand off — the pull request that would have deleted the lock is never coming — so that lock is released, and the outcome is logged either way. @@ -130,7 +130,7 @@ The user works from two machines, and some agents run in the cloud. Tickets, the #### Business logic -Every registered project's `agent-data` branch is converged with origin on a schedule, and then its `agents-logs` branch; a project whose tickets could not be converged is not asked for its logs, since the first failure is already the answer. On success the project's data-sync error is cleared unconditionally, so the error lives exactly as long as the condition does — the turn after the user fixes the remote, it is gone. On failure the reason is recorded against that project for the dashboard to show, and said on the daemon's log as well, so a user watching the daemon rather than the dashboard sees it too. +Every registered project's `agent-data` branch is converged with origin on a schedule; a project whose tickets could not be converged is not asked for its queue, since the first failure is already the answer. On success the project's data-sync error is cleared unconditionally, so the error lives exactly as long as the condition does — the turn after the user fixes the remote, it is gone. On failure the reason is recorded against that project for the dashboard to show, and said on the daemon's log as well, so a user watching the daemon rather than the dashboard sees it too. ### A project's agent options diff --git a/packages/framework/src/daemon-services.ts b/packages/framework/src/daemon-services.ts index abcc046b9..9bcf2c285 100644 --- a/packages/framework/src/daemon-services.ts +++ b/packages/framework/src/daemon-services.ts @@ -19,11 +19,9 @@ import { maintenanceDue, readMaintenanceState, mergeMaintenanceState } from './m import { claimTickets, readTickets, releaseTicket, syncTickets, TICKETS_DIR, ticketFromQueueEntry, ticketsDir } from '@gemstack/skill-tickets' import { queueDone, readQueueEntries, syncQueue } from '@gemstack/skill-queue' import { nextQueuedTicket } from './todo-loop.js' -import { LOGS_BRANCH } from './framework-dir.js' import type { ProjectErrors } from './project-errors.js' import { readFile, writeFile } from 'node:fs/promises' import { startMergedWorktreeSweep, type MergedSweepOptions } from './merged-worktrees.js' -import { pullFileBranch } from '@gemstack/agent-data' import { reconcileBranchLinks } from '@gemstack/skill-branches' import { startProjectPass } from './project-pass.js' import { startCloudScratchSweep } from './cloud-scratch-refs.js' @@ -118,17 +116,16 @@ export interface BackgroundServiceDeps { } /** - * One project's data-sync turn (#1599): converge the `agent-data` branch (the skills': the - * `tickets` skill's checkout, root link and pull, then the `queue` skill's seed and pull) and the - * `agents-logs` branch (the archives and routine locks) with origin, and set or clear the - * project's `data-sync` error by the outcome. The clear is unconditional on success, so the error - * lives exactly as long as the condition — the next tick after the user fixes the remote, it is - * gone. + * One project's data-sync turn (#1599): converge the `agent-data` branch — the skills' branch, + * which also carries the `logs` skill's runs and the routine locks — with origin (the `tickets` + * skill's checkout, root link and pull, then the `queue` skill's seed and pull), and set or clear + * the project's `data-sync` error by the outcome. The clear is unconditional on success, so the + * error lives exactly as long as the condition — the next tick after the user fixes the remote, + * it is gone. */ export async function syncProjectData(path: string, errors: ProjectErrors, log: (message: string) => void): Promise { const tickets = await syncTickets(path, { log }) - const queue = tickets.ok ? await syncQueue(path, { log }) : tickets - const result = queue.ok ? await pullFileBranch(path, LOGS_BRANCH, { log }) : queue + const result = tickets.ok ? await syncQueue(path, { log }) : tickets if (result.ok) errors.clear(path, 'data-sync') else { log(`[framework] data sync: ${result.error}`) diff --git a/packages/framework/src/daemon-workspace.test.SPEC.md b/packages/framework/src/daemon-workspace.test.SPEC.md index 7b2490f0a..68037ebb1 100644 --- a/packages/framework/src/daemon-workspace.test.SPEC.md +++ b/packages/framework/src/daemon-workspace.test.SPEC.md @@ -12,7 +12,7 @@ Refusing a start the driver cannot serve: a start on a logged-out driver is refu Held slots: a live agent's slot names the agent and its process id, and stopping the agents reports the same name and clears the slot. -The agent's environment: the spawned process's PATH starts with the `skill-branches`, `skill-tickets` and `skill-queue` packages' executable directories, in that order, and continues with the daemon's own; the `branches` command found through it lists the very checkout the daemon allocated; and that checkout links the three skills, each under its name, where every harness looks for skills, each link pointing at the package holding its `SKILL.md`. +The agent's environment: the spawned process's PATH starts with the `skill-branches`, `skill-tickets`, `skill-queue` and `skill-logs` packages' executable directories, in that order, and continues with the daemon's own; the `branches` command found through it lists the very checkout the daemon allocated; and that checkout links the four skills, each under its name, where every harness looks for skills, each link pointing at the package holding its `SKILL.md`. ## Before modifying/creating SPEC.md files diff --git a/packages/framework/src/daemon-workspace.test.ts b/packages/framework/src/daemon-workspace.test.ts index ea55bbbd7..9d4e3c44d 100644 --- a/packages/framework/src/daemon-workspace.test.ts +++ b/packages/framework/src/daemon-workspace.test.ts @@ -18,6 +18,7 @@ import { BRANCHES_DIR, nodeGitRunner, GitTimeoutError } from '@gemstack/agent-da import { worktreePath, agentBranchName, CLI_BIN_DIR, HARNESS_SKILL_DIRS, SKILL_DIR as BRANCHES_SKILL_DIR } from '@gemstack/skill-branches' import { CLI_BIN_DIR as TICKETS_BIN_DIR, SKILL_DIR as TICKETS_SKILL_DIR } from '@gemstack/skill-tickets' import { CLI_BIN_DIR as QUEUE_BIN_DIR, SKILL_DIR as QUEUE_SKILL_DIR } from '@gemstack/skill-queue' +import { CLI_BIN_DIR as LOGS_BIN_DIR, SKILL_DIR as LOGS_SKILL_DIR } from '@gemstack/skill-logs' import { THE_FRAMEWORK_DIR } from './framework-dir.js' import { addProject, projectId } from './registry.js' import type { AgentSpec } from './agent-spec.js' @@ -665,8 +666,8 @@ test('a spawned agent finds the `branches` command on its PATH (#1725)', async ( recorded = await readFile(log, 'utf8').catch(() => '') } const path = recorded.trim() - assert.deepEqual(path.split(delimiter).slice(0, 3), [CLI_BIN_DIR, TICKETS_BIN_DIR, QUEUE_BIN_DIR], 'the packages\' bin dirs come first: branches, tickets, queue (#1748)') - assert.equal(path.split(delimiter).slice(3).join(delimiter), process.env['PATH'], "after the daemon's own") + assert.deepEqual(path.split(delimiter).slice(0, 4), [CLI_BIN_DIR, TICKETS_BIN_DIR, QUEUE_BIN_DIR, LOGS_BIN_DIR], 'the packages\' bin dirs come first: branches, tickets, queue, logs (#1748/#1769)') + assert.equal(path.split(delimiter).slice(4).join(delimiter), process.env['PATH'], "after the daemon's own") // By name, the way the agent's shell resolves it, against the project the daemon started it in. const listed = await new Promise((resolvePromise, rejectPromise) => execFile('branches', ['list'], { cwd, env: { ...process.env, PATH: path } }, (err, stdout) => (err ? rejectPromise(err) : resolvePromise(stdout))), @@ -676,9 +677,9 @@ test('a spawned agent finds the `branches` command on its PATH (#1725)', async ( [result.agentId], 'and it reports the checkout the daemon allocated', ) - // The three skills are in the checkout too, where each harness looks for them (#1739/#1748). + // The four skills are in the checkout too, where each harness looks for them (#1739/#1748/#1769). for (const harnessDir of HARNESS_SKILL_DIRS) { - for (const [name, dir] of [['branches', BRANCHES_SKILL_DIR], ['tickets', TICKETS_SKILL_DIR], ['queue', QUEUE_SKILL_DIR]] as const) { + for (const [name, dir] of [['branches', BRANCHES_SKILL_DIR], ['tickets', TICKETS_SKILL_DIR], ['queue', QUEUE_SKILL_DIR], ['logs', LOGS_SKILL_DIR]] as const) { const target = await readlink(join(worktreePath(cwd, result.agentId!), harnessDir, name)) assert.equal(await realpath(resolve(join(worktreePath(cwd, result.agentId!), harnessDir), target)), await realpath(dir), `${harnessDir}/${name} links the package holding its SKILL.md`) } diff --git a/packages/framework/src/daemon.SPEC.md b/packages/framework/src/daemon.SPEC.md index 27dce84ab..fb9b03ba7 100644 --- a/packages/framework/src/daemon.SPEC.md +++ b/packages/framework/src/daemon.SPEC.md @@ -132,7 +132,7 @@ The user presses Ctrl-C. Nothing may be started while everything is being stoppe On an interrupt or termination signal, the shutdown runs in a fixed order. The background services quiesce first, so that Auto PM or an arriving notification cannot start an agent while the rest are being stopped. Then the agents this daemon spawned are stopped, and the ids of those that were still alive are logged by name — a process still alive at shutdown that the dashboard showed as finished is the one fact that explains a busy slot nothing else can account for, and a bare count hides it. Finally the quota meter is stopped, the bridge browser is closed — left running it would serve a daemon that is gone, and hold the profile the next daemon needs — the runtime is disposed, and the server is closed. -Finished agents' archives need no flushing step here: each one is committed and pushed through the logs branch's write cycle the moment that agent settles. +Finished agents' archives need no flushing step here: each one is committed and pushed through the data branch's write cycle, as the `logs` skill's run, the moment that agent settles. ## Before modifying/creating SPEC.md files diff --git a/packages/framework/src/dashboard-rpc/control.SPEC.md b/packages/framework/src/dashboard-rpc/control.SPEC.md index ba09b005c..3669ad8ed 100644 --- a/packages/framework/src/dashboard-rpc/control.SPEC.md +++ b/packages/framework/src/dashboard-rpc/control.SPEC.md @@ -64,7 +64,7 @@ An agent finishes and the user decides to publish its work: push the branch, or #### Business logic -Both actions address the agent's own branch as recorded on its agent meta, and act on the agent's own checkout, because for an agent that never committed, its checkout is the only place its work exists. Nothing is committed on the agent's behalf: what is pushed is the branch as the agent committed it, and work it never committed stays in its checkout, named on its page. Opening a pull request pushes the branch first when the remote does not have it, and takes its title and body from what the agent already recorded — the session name it chose and the intent the user asked for — inventing nothing and asking the user for nothing. The number and URL of the opened pull request are written onto the agent's archive on the logs branch, so every surface reads the same fact from the same place instead of re-deriving it from branch names. +Both actions address the agent's own branch as recorded on its agent meta, and act on the agent's own checkout, because for an agent that never committed, its checkout is the only place its work exists. Nothing is committed on the agent's behalf: what is pushed is the branch as the agent committed it, and work it never committed stays in its checkout, named on its page. Opening a pull request pushes the branch first when the remote does not have it, and takes its title and body from what the agent already recorded — the session name it chose and the intent the user asked for — inventing nothing and asking the user for nothing. The number and URL of the opened pull request are written onto the agent's run card on the data branch, through the `logs` skill, so every surface reads the same fact from the same place instead of re-deriving it from branch names. #### Rationale diff --git a/packages/framework/src/dashboard-rpc/control.ts b/packages/framework/src/dashboard-rpc/control.ts index 88bd44acc..da4838c6a 100644 --- a/packages/framework/src/dashboard-rpc/control.ts +++ b/packages/framework/src/dashboard-rpc/control.ts @@ -13,7 +13,7 @@ import { pushBranch } from '@gemstack/agent-data' import { isSafeAgentId, worktreePath } from '@gemstack/skill-branches' import { withAgentLock } from '../agent-locks.js' import { removeProjectWorktree, deleteProjectAgent } from '../worktrees.js' -import { patchArchivedAgentOnDataBranch } from '../archived-agent-patch.js' +import { patchRun } from '@gemstack/skill-logs' import { mergeAgentPr, openAgentPullRequest, agentBranchFor, type HandoffResult } from '../dashboard/agent-handoff.js' import type { ChoiceBy } from '../events.js' import { isHandoffLevel, type HandoffLevel } from '../handoff-level.js' @@ -277,7 +277,7 @@ export async function sendOpenPullRequest(projectId: string, agentId: string): P // stream to carry the fact — but it is the same fact, and every surface reads it from the same // place either way rather than re-deriving it from branch names. if (opened.ok && opened.number !== undefined && opened.url) { - await patchArchivedAgentOnDataBranch(target.cwd, agentId, { pr: { number: opened.number, url: opened.url } }, `[The Framework] record the PR of session ${agentId}`) + await patchRun(target.cwd, agentId, { pr: { number: opened.number, url: opened.url } }) } return opened }, { ok: false, error: 'could not reach the device' }) diff --git a/packages/framework/src/dashboard-rpc/events.ts b/packages/framework/src/dashboard-rpc/events.ts index 7e1922dfb..7fc441729 100644 --- a/packages/framework/src/dashboard-rpc/events.ts +++ b/packages/framework/src/dashboard-rpc/events.ts @@ -1,4 +1,5 @@ -import { resolveAgentEventsPath } from '../store/index.js' +import { fromDiaryLine, resolveAgentEventsPath } from '../store/index.js' +import type { AnyDiaryLine } from '@gemstack/skill-logs' import { contextEventsSource, resolveProjectPath } from './context.js' import type { FrameworkEvent } from '../events.js' import { tailAgentEvents } from './events-tail.js' @@ -75,7 +76,8 @@ export async function streamAgentEvents( // initial attach stays permissive: a fallback agent (non-git project) legitimately lives there. const rootJournal = agentId === undefined ? undefined : await resolveEventsPath(projectId, undefined) let initial = true - return tailAgentEvents( + // An ended run's file is the `logs` skill's diary (#1769): its lines come back as the framework's events. + return tailAgentEvents( async () => { const next = await resolveEventsPath(projectId, agentId) if (initial) { @@ -84,7 +86,7 @@ export async function streamAgentEvents( } return rootJournal !== undefined && next === rootJournal ? undefined : next }, - send, + line => send(fromDiaryLine(line)), () => send({ kind: 'stream-sync' }), ) } diff --git a/packages/framework/src/dashboard-rpc/reads.SPEC.md b/packages/framework/src/dashboard-rpc/reads.SPEC.md index 67e480157..a19419c6e 100644 --- a/packages/framework/src/dashboard-rpc/reads.SPEC.md +++ b/packages/framework/src/dashboard-rpc/reads.SPEC.md @@ -47,7 +47,7 @@ The user starts an agent and sees it appear in the sidebar as running straight a #### Business logic -The list is the project's archived agents, plus every agent currently live in its own worktree, plus the agents this daemon is relaying from devices — which exist only in the daemon's memory and would otherwise be lost on a reload. There is exactly one row per agent id: a relayed copy beats a local one, and a live copy beats an archived one, because the live copy is the current truth. Nothing is filtered out by status, so an agent whose record was just corrected does not flicker out of the list for a poll. A `web`-target agent whose cloud session the browser bridge reports as waiting on a human — parked on a question it holds, or shown awaiting input by claude.ai's session list — is marked as waiting on the way out — the record on disk cannot know, only the daemon does — so its row says "waiting" rather than "in cloud". Likewise an agent whose recorded host is not this machine is marked as from another host: the logs branch is shared precisely so that other machines' agents appear here, and a row that looked exactly like one of this daemon's own left the user reading the archive by hand to learn whose it was. A record with no host at all is left alone. The same marks are applied to the Overview's pooled recent agents. +The list is the project's archived agents, plus every agent currently live in its own worktree, plus the agents this daemon is relaying from devices — which exist only in the daemon's memory and would otherwise be lost on a reload. There is exactly one row per agent id: a relayed copy beats a local one, and a live copy beats an archived one, because the live copy is the current truth. Nothing is filtered out by status, so an agent whose record was just corrected does not flicker out of the list for a poll. A `web`-target agent whose cloud session the browser bridge reports as waiting on a human — parked on a question it holds, or shown awaiting input by claude.ai's session list — is marked as waiting on the way out — the record on disk cannot know, only the daemon does — so its row says "waiting" rather than "in cloud". Likewise an agent whose recorded host is not this machine is marked as from another host: the data branch is shared precisely so that other machines' agents appear here, and a row that looked exactly like one of this daemon's own left the user reading the archive by hand to learn whose it was. A record with no host at all is left alone. The same marks are applied to the Overview's pooled recent agents. ### Where an agent is working diff --git a/packages/framework/src/dashboard/overview.SPEC.md b/packages/framework/src/dashboard/overview.SPEC.md index 75acaa35a..349c36901 100644 --- a/packages/framework/src/dashboard/overview.SPEC.md +++ b/packages/framework/src/dashboard/overview.SPEC.md @@ -24,7 +24,7 @@ The user wants to know, without opening any project, which agents are running an #### Business logic -Every project's live agents are read and only those still running are listed, one entry per agent — a single project can have several in flight, each in its own worktree. Every project's archived agents are read too, for the `web`-target agents whose cloud side is still at work per the cloud state rule (`cloud-run-state`): in cloud, or waiting on a human per the browser bridge — a question it holds, or claude.ai's session list showing the session awaiting input. Those are listed as well, marked with that state, keyed to the project's own path since their checkout may be gone, and once each even when two registered checkouts of one repository share the archive that holds them; a cloud session at work is an agent at work, and "no agents working" over a session waiting on the user was a lie. Each entry carries the project it belongs to, the agent's own checkout so its git and file status is read from the worktree it actually edits, what the user asked for, when the agent last spoke, the session name the agent chose if it has picked one, whether it has signalled ready for merge (which drives the building-versus-ready indicator), and the name of the machine whose daemon started it when that is not this machine — the logs branch is shared, so another machine's agents appear here too. The list is ordered by last activity, most recent first. +Every project's live agents are read and only those still running are listed, one entry per agent — a single project can have several in flight, each in its own worktree. Every project's archived agents are read too, for the `web`-target agents whose cloud side is still at work per the cloud state rule (`cloud-run-state`): in cloud, or waiting on a human per the browser bridge — a question it holds, or claude.ai's session list showing the session awaiting input. Those are listed as well, marked with that state, keyed to the project's own path since their checkout may be gone, and once each even when two registered checkouts of one repository share the archive that holds them; a cloud session at work is an agent at work, and "no agents working" over a session waiting on the user was a lie. Each entry carries the project it belongs to, the agent's own checkout so its git and file status is read from the worktree it actually edits, what the user asked for, when the agent last spoke, the session name the agent chose if it has picked one, whether it has signalled ready for merge (which drives the building-versus-ready indicator), and the name of the machine whose daemon started it when that is not this machine — the data branch is shared, so another machine's agents appear here too. The list is ordered by last activity, most recent first. ### Backlog size diff --git a/packages/framework/src/framework-dir.SPEC.md b/packages/framework/src/framework-dir.SPEC.md index 558b9bda7..c6b30536f 100644 --- a/packages/framework/src/framework-dir.SPEC.md +++ b/packages/framework/src/framework-dir.SPEC.md @@ -1,4 +1,4 @@ -Names `.the-framework/` — the directory under a project root where The Framework keeps its own files — and the logs branch, `agents-logs`, which carries what The Framework records about its own runs (the agent archives and the routine locks), with the place its checkout sits under a project: `.branches/agents-logs`, beside the agent checkouts and named as its branch like each of them. Nothing of the product rides there: the tickets and the agent queue are the `tickets` and `queue` skills', on their own branch. Kept on its own so browser-side surfaces (the dashboard renders preset file paths built from it) can name paths under it without touching any node-only module. +Names `.the-framework/` — the directory under a project root where The Framework keeps its own files. Nothing of the product rides on a branch of its own: the tickets and the agent queue are the `tickets` and `queue` skills', the runs are the `logs` skill's, and the routine locks sit beside them, all on the shared `agent-data` branch. Kept on its own so browser-side surfaces (the dashboard renders preset file paths built from it) can name paths under it without touching any node-only module. ## Before modifying/creating SPEC.md files diff --git a/packages/framework/src/framework-dir.ts b/packages/framework/src/framework-dir.ts index 40757a4d9..996715617 100644 --- a/packages/framework/src/framework-dir.ts +++ b/packages/framework/src/framework-dir.ts @@ -1,23 +1,12 @@ -import { BRANCHES_DIR } from '@gemstack/agent-data/names' - /** * The directory, under a project root, that holds The Framework's own files. * * Its own module because it is the one piece of `logs.ts` the browser needs: the preset registry * builds `tf.presets..filePath` from it (#874), and the dashboard renders presets in the * browser (#520), where `logs.ts` cannot go — it imports `node:path`. + * + * Nothing of the product's rides on a branch of its own any more: the agent archives are the + * `logs` skill's runs and the routine locks sit beside them, all on the shared `agent-data` + * branch that `@gemstack/agent-data` names (#1769). */ export const THE_FRAMEWORK_DIR = '.the-framework' - -/** - * The branch holding what The Framework itself records about its runs (#1582/#1748): the agent - * archives and the routine locks. The tickets and the queue are the `tickets` and `queue` skills', - * on the shared `agent-data` branch — nothing of the product's rides there. - */ -export const LOGS_BRANCH = 'agents-logs' - -/** - * The logs branch's checkout under a project, relative to the project root: `.branches/agents-logs`, - * beside the agent checkouts and named as its branch like each of them (#1736). - */ -export const LOGS_CHECKOUT_DIR = `${BRANCHES_DIR}/${LOGS_BRANCH}` diff --git a/packages/framework/src/framework-gitignore.SPEC.md b/packages/framework/src/framework-gitignore.SPEC.md index 9f0d38a75..537acdcec 100644 --- a/packages/framework/src/framework-gitignore.SPEC.md +++ b/packages/framework/src/framework-gitignore.SPEC.md @@ -1,4 +1,4 @@ -The `.the-framework/.gitignore` written at install — one file, one fixed content, and the file whose presence marks a project as activated. It ignores everything under `.the-framework/` except itself and the layout marker: agent state there (the event log, the agent meta, the worktrees) is transient on the default branch, and the lasting records live on the `agents-logs` branch — so the default branch stays 100% code plus the one tracked gate file. +The `.the-framework/.gitignore` written at install — one file, one fixed content, and the file whose presence marks a project as activated. It ignores everything under `.the-framework/` except itself and the layout marker: agent state there (the event log, the agent meta, the worktrees) is transient on the default branch, and the lasting records — every run, the `logs` skill's — live on the `agent-data` branch — so the default branch stays 100% code plus the one tracked gate file. ## Before modifying/creating SPEC.md files diff --git a/packages/framework/src/framework-gitignore.test.SPEC.md b/packages/framework/src/framework-gitignore.test.SPEC.md new file mode 100644 index 000000000..728e13c14 --- /dev/null +++ b/packages/framework/src/framework-gitignore.test.SPEC.md @@ -0,0 +1,5 @@ +What the tests cover: the `.the-framework/.gitignore` written at install, against real git — everything under `.the-framework/` (the transient archive, the live event log, a run checkout) stays ignored on the default branch, and the ignore file itself is the one tracked thing. + +## Before modifying/creating SPEC.md files + +You must always read and respect https://raw.githubusercontent.com/brillout/sdd/refs/heads/main/sdd.md diff --git a/packages/framework/src/framework-gitignore.test.ts b/packages/framework/src/framework-gitignore.test.ts new file mode 100644 index 000000000..c26e0edef --- /dev/null +++ b/packages/framework/src/framework-gitignore.test.ts @@ -0,0 +1,36 @@ +import { strict as assert } from 'node:assert' +import { test } from 'node:test' +import { join } from 'node:path' +import { frameworkGitignore } from './framework-gitignore.js' + +test('against real git: everything under .the-framework is transient on main (#1582)', async () => { + // The lasting records live on the data branch, so the ignore file is "ignore it all" — a + // session's live state, the transient archive, and the run checkouts must never dirty main. + const { mkdtemp, mkdir, writeFile, rm } = await import('node:fs/promises') + const { tmpdir } = await import('node:os') + const { execFileSync } = await import('node:child_process') + + const repo = await mkdtemp(join(tmpdir(), 'fw-sessions-')) + const git = (...args: string[]): string => execFileSync('git', args, { cwd: repo, encoding: 'utf8' }) + try { + git('init', '-q') + git('config', 'user.email', 'git@example.com') + git('config', 'user.name', 'Test') + + const fw = join(repo, '.the-framework') + await mkdir(join(fw, 'agents'), { recursive: true }) + await mkdir(join(fw, 'branches', 'agent-r9'), { recursive: true }) + await writeFile(join(fw, '.gitignore'), frameworkGitignore()) + await writeFile(join(fw, 'agents', 'old.json'), '{}\n') + await writeFile(join(fw, 'events.jsonl'), '\n') + await writeFile(join(fw, 'branches', 'agent-r9', 'file.txt'), 'x\n') + + const status = git('status', '--porcelain', '-uall') + assert.ok(!status.includes('.the-framework/agents/'), 'the transient archive stays ignored') + assert.ok(!status.includes('.the-framework/events.jsonl'), 'the live log stays ignored') + assert.ok(!status.includes('.branches/'), 'a run checkout stays ignored') + assert.ok(status.includes('.the-framework/.gitignore'), 'the ignore file itself is the one tracked thing') + } finally { + await rm(repo, { recursive: true, force: true }) + } +}) diff --git a/packages/framework/src/framework-gitignore.ts b/packages/framework/src/framework-gitignore.ts index 307c957ba..7e8d375e4 100644 --- a/packages/framework/src/framework-gitignore.ts +++ b/packages/framework/src/framework-gitignore.ts @@ -1,5 +1,6 @@ import { join } from 'node:path' -import { LOGS_BRANCH, THE_FRAMEWORK_DIR } from './framework-dir.js' +import { DATA_BRANCH } from '@gemstack/agent-data/names' +import { THE_FRAMEWORK_DIR } from './framework-dir.js' import { LAYOUT_FILE } from './layout.js' /** @@ -7,7 +8,7 @@ import { LAYOUT_FILE } from './layout.js' * * Everything under `.the-framework/` is transient on main — `events.jsonl`, `agent.json`, the * worktrees — and would otherwise turn every session into a dirty checkout. The lasting records - * (the session archives) live on the data branch (#1582), so the only tracked entry left is the + * (the runs, the `logs` skill's) live on the data branch (#1582/#1769), so the only tracked entry left is the * layout marker (#1575): main is 100% code plus that one gate file. */ @@ -18,5 +19,5 @@ export function gitignorePath(cwd: string): string { /** The whole file: everything under `.the-framework/` is transient except the layout marker (#1582, #1575). */ export function frameworkGitignore(): string { - return `# The Framework: agent state is transient; the lasting records live on the ${LOGS_BRANCH} branch.\n*\n!.gitignore\n!${LAYOUT_FILE}\n` + return `# The Framework: agent state is transient; the lasting records live on the ${DATA_BRANCH} branch.\n*\n!.gitignore\n!${LAYOUT_FILE}\n` } diff --git a/packages/framework/src/install.test.SPEC.md b/packages/framework/src/install.test.SPEC.md index c6dd4b51f..4ada9986c 100644 --- a/packages/framework/src/install.test.SPEC.md +++ b/packages/framework/src/install.test.SPEC.md @@ -2,7 +2,7 @@ What the tests cover: activating a repository. - Activating a clean repository writes the `.the-framework/.gitignore` marker and produces exactly one install commit. - The quality presets are written out during activation, so a queued quality follow-up points at a document that really exists. -- The seeded ignore file ignores everything transient under `.the-framework/` and un-ignores only itself and the layout marker — the lasting records live on the logs branch, so nothing else is kept. +- The seeded ignore file ignores everything transient under `.the-framework/` and un-ignores only itself and the layout marker — the lasting records live on the data branch, so nothing else is kept. - The layout marker is written and is tracked, which is what lets a build with a different layout refuse to run in the repository. - A repository with uncommitted work keeps it uncommitted: only `.the-framework/` is added, and the one commit is the activation. - An already-activated repository is left completely untouched and reported as already activated. diff --git a/packages/framework/src/layout.SPEC.md b/packages/framework/src/layout.SPEC.md index 2f7977825..b49540571 100644 --- a/packages/framework/src/layout.SPEC.md +++ b/packages/framework/src/layout.SPEC.md @@ -2,7 +2,7 @@ The layout gate: a build of The Framework refuses to run in a project whose reco ## Business logic — TL;DR -- **The project records the layout it uses** - the layout marker is a tracked file under `.the-framework/`, so every clone and every worktree of the project carries it. It records every name a committed artifact's path hangs off: the framework directory, the `agent-data` branch, the logs branch, the archive directory, the event log and agent meta file names, the tickets directory, and the agent queue file. +- **The project records the layout it uses** - the layout marker is a tracked file under `.the-framework/`, so every clone and every worktree of the project carries it. It records every name a committed artifact's path hangs off: the framework directory, the `agent-data` branch, the event log and agent meta file names, the tickets directory, the agent queue file, and the runs directory. - **A mismatch is refused outright** - a build whose layout differs from the project's records refuses to run there, and says both layouts in full plus the fix for each direction: update The Framework to a build that matches the project, or, when the build is the newer side, rewrite the marker as part of the rename itself. - **An unmarked project is ungated** - a project with no marker runs without the check. Activation writes the marker, so every newly activated project is gated from the start. - **Refusal is a reported outcome** - the check reports its verdict for the caller to act on rather than crashing. diff --git a/packages/framework/src/layout.test.ts b/packages/framework/src/layout.test.ts index 8cf02a636..e0628cbcd 100644 --- a/packages/framework/src/layout.test.ts +++ b/packages/framework/src/layout.test.ts @@ -33,14 +33,14 @@ test('a marker matching this build passes', async () => { }) test('a mismatched marker refuses, naming both layouts and the fix (#1575)', async () => { - const recorded = layoutMarker().replace('archive-dir: agents', 'archive-dir: sessions') + const recorded = layoutMarker().replace('runs-dir: agents', 'runs-dir: sessions') const result = await checkLayout('/proj', markerFs(recorded)) assert.equal(result.ok, false) if (result.ok) return // The refusal carries what a stranded session's log needs: the file, both sides, the cause. assert.match(result.error, /\.the-framework\/LAYOUT/) - assert.match(result.error, /archive-dir: agents/) - assert.match(result.error, /archive-dir: sessions/) + assert.match(result.error, /runs-dir: agents/) + assert.match(result.error, /runs-dir: sessions/) assert.match(result.error, /#1575/) assert.match(result.error, /update/i) }) diff --git a/packages/framework/src/layout.ts b/packages/framework/src/layout.ts index a2cd2c226..7a0a3b58e 100644 --- a/packages/framework/src/layout.ts +++ b/packages/framework/src/layout.ts @@ -1,8 +1,9 @@ import { join } from 'node:path' -import { LOGS_BRANCH, THE_FRAMEWORK_DIR } from './framework-dir.js' -import { ARCHIVE_DIR, EVENTS_FILE, META_FILE, nodeStoreFs, type StoreFs } from './store/index.js' +import { THE_FRAMEWORK_DIR } from './framework-dir.js' +import { EVENTS_FILE, META_FILE, nodeStoreFs, type StoreFs } from './store/index.js' import { TICKETS_DIR } from '@gemstack/skill-tickets/names' import { QUEUE_FILE } from '@gemstack/skill-queue/names' +import { RUNS_DIR } from '@gemstack/skill-logs/names' import { DATA_BRANCH } from '@gemstack/agent-data/names' /** @@ -39,12 +40,11 @@ export function layoutMarker(): string { return [ `framework-dir: ${THE_FRAMEWORK_DIR}`, `data-branch: ${DATA_BRANCH}`, - `logs-branch: ${LOGS_BRANCH}`, - `archive-dir: ${ARCHIVE_DIR}`, `events-file: ${EVENTS_FILE}`, `meta-file: ${META_FILE}`, `tickets-dir: ${TICKETS_DIR}`, `queue-file: ${QUEUE_FILE}`, + `runs-dir: ${RUNS_DIR}`, '', ].join('\n') } diff --git a/packages/framework/src/project-errors.SPEC.md b/packages/framework/src/project-errors.SPEC.md index 89378affa..d88198491 100644 --- a/packages/framework/src/project-errors.SPEC.md +++ b/packages/framework/src/project-errors.SPEC.md @@ -2,7 +2,7 @@ Holds the daemon's per-project error state: when a background job finds a projec ## Business logic — TL;DR -- **One slot per project and error kind** - each project holds at most one live error per kind; today the only kind is `data-sync`, raised when a project's branches — the `agent-data` branch and the `agents-logs` branch — cannot be synced (the push to origin is rejected, or the repo has no origin at all). +- **One slot per project and error kind** - each project holds at most one live error per kind; today the only kind is `data-sync`, raised when a project's `agent-data` branch cannot be synced (the push to origin is rejected, or the repo has no origin at all). - **The age of a problem is preserved** - re-reporting the same error kind refreshes its detail message but keeps the timestamp it was first seen, so the dashboard can say how long the condition has lasted instead of restarting the clock on every re-check. - **Errors live only as long as the daemon does** - nothing is written to disk; a restarted daemon starts clean and re-learns each error on the next check by the job that raised it. diff --git a/packages/framework/src/routine-locks.SPEC.md b/packages/framework/src/routine-locks.SPEC.md index 50634c839..5a0ce6563 100644 --- a/packages/framework/src/routine-locks.SPEC.md +++ b/packages/framework/src/routine-locks.SPEC.md @@ -1,4 +1,4 @@ -The routine lock: holding one of Auto PM's routines for one machine while it runs, by writing a lock file for that routine on The Framework's own logs branch, `agents-logs`, so no two machines sharing that branch ever run the same routine at once. +The routine lock: holding one of Auto PM's routines for one machine while it runs, by writing a lock file for that routine on the shared `agent-data` branch, beside the skills' files, so no two machines sharing that branch ever run the same routine at once. ## User story @@ -8,12 +8,12 @@ The routine lock: holding one of Auto PM's routines for one machine while it run ## Glossary -- **routine lock** — a routine's `routines/.lock.md` file on the logs branch. It names the machine that holds the routine and the moment that machine took it, one line each: `CLAIMED: ` and `SINCE: `. Content missing either line names no holder and is not a lock. -- **write funnel** — the logs branch's single write cycle (sync, apply the change, commit, push), which re-runs the change against origin's fresher state when a push loses a race, and restores the checkout when a cycle fails whole. +- **routine lock** — a routine's `routines/.lock.md` file on the data branch. It names the machine that holds the routine and the moment that machine took it, one line each: `CLAIMED: ` and `SINCE: `. Content missing either line names no holder and is not a lock. +- **write funnel** — the data branch's single write cycle (sync, apply the change, commit, push), which re-runs the change against origin's fresher state when a push loses a race, and restores the checkout when a cycle fails whole. ## Business logic — TL;DR -- **The routine lock is a file on the shared logs branch** - written and read through the same write funnel as every other write to the branch, so it reaches every machine that shares the branch instead of living in one daemon's memory. +- **The routine lock is a file on the shared data branch** - written and read through the same write funnel as every other write to the branch, so it reaches every machine that shares the branch instead of living in one daemon's memory. - **An alive lock stands the caller down, naming its holder** - the refusal says which machine holds the routine and since when, so the reason can be read rather than guessed. - **The taking is decided before anything is started** - the answer is a lock file, not something an agent discovers once it is already running. - **A commit that could not be pushed still counts as taken** - it guards this machine; the cross-machine gap is logged rather than treated as a failure. A cycle that could not commit at all took nothing and says so. @@ -23,7 +23,7 @@ The routine lock: holding one of Auto PM's routines for one machine while it run ## Business logic -### The routine lock is a file on the logs branch +### The routine lock is a file on the data branch #### User story @@ -31,7 +31,7 @@ See `## User story`: a routine taken on one machine has to be visible on the oth #### Business logic -A routine's lock is `routines/.lock.md` on the logs branch, holding the machine that claimed the routine and the timestamp it claimed it at. It is written, read and removed inside the logs branch's write funnel like every other write to the branch, which is what carries the claim to every machine sharing the branch. Each commit says what it did — locking a named routine, releasing a named routine, or releasing the count of locks a previous daemon left behind — so the branch history reads as what happened. +A routine's lock is `routines/.lock.md` on the data branch, holding the machine that claimed the routine and the timestamp it claimed it at. It is written, read and removed inside the data branch's write funnel like every other write to the branch, which is what carries the claim to every machine sharing the branch. Each commit says what it did — locking a named routine, releasing a named routine, or releasing the count of locks a previous daemon left behind — so the branch history reads as what happened. Nothing here ever throws: it runs on a background job with nothing to catch it. @@ -91,7 +91,7 @@ A lock is dead four hours after it was taken, and whoever finds a dead lock take #### Rationale -The expiry is fixed and there is no heartbeat: a triage over hundreds of tickets can legitimately run for hours, so the window has to be generous, and a heartbeat would add code and a stream of commits to the logs branch for a case the boot-time cleanup already covers on the machine that matters. +The expiry is fixed and there is no heartbeat: a triage over hundreds of tickets can legitimately run for hours, so the window has to be generous, and a heartbeat would add code and a stream of commits to the data branch for a case the boot-time cleanup already covers on the machine that matters. ## Before modifying/creating SPEC.md files diff --git a/packages/framework/src/routine-locks.test.SPEC.md b/packages/framework/src/routine-locks.test.SPEC.md index a51bc2f54..57deb868e 100644 --- a/packages/framework/src/routine-locks.test.SPEC.md +++ b/packages/framework/src/routine-locks.test.SPEC.md @@ -1,11 +1,11 @@ -What the tests cover: the routine lock — the file that reserves one of Auto PM's routines on the logs branch. +What the tests cover: the routine lock — the file that reserves one of Auto PM's routines on the data branch. - **The lock file.** It lives at `routines/.lock.md`, names the machine that holds the routine and the moment it took it, and reads back to exactly that pair; content that is not a lock names no holder. - **Taking one.** A free routine is locked in one funneled write. A cycle re-run after a lost push race finds this machine's own claim and still counts it as taken, committing once. An alive lock stands the caller down with a reason naming the holding machine and the time it took it, and writes nothing — whether the holder is another machine or this one's own earlier run. - **The expiry.** A lock four hours old or older is dead and is taken over in the same commit; a minute younger and it still stands. - **Releasing one.** This machine's lock is removed; a lock naming another machine is left standing; a routine with no lock is a no-op that writes nothing. Each of the three reports the lock as dealt with. - **A dead daemon's leftovers.** Sweeping this machine's locks frees those whose run is gone and names them, keeps the one whose run is still going, never touches another machine's, and ignores files under the routines directory that are not locks — all in one commit whose message counts what it freed. -- **Two machines, one shared logs branch, against real git.** The first machine's lock reaches the shared branch; the second machine reads it there and stands down; the second cannot release what the first holds; and once the first machine's release reaches the branch, the second's next attempt takes the routine. +- **Two machines, one shared data branch, against real git.** The first machine's lock reaches the shared branch; the second machine reads it there and stands down; the second cannot release what the first holds; and once the first machine's release reaches the branch, the second's next attempt takes the routine. ## Before modifying/creating SPEC.md files diff --git a/packages/framework/src/routine-locks.test.ts b/packages/framework/src/routine-locks.test.ts index 342ebd792..744b953db 100644 --- a/packages/framework/src/routine-locks.test.ts +++ b/packages/framework/src/routine-locks.test.ts @@ -13,10 +13,9 @@ import { routineLockPath, type RoutineLockDeps, } from './routine-locks.js' -import { LOGS_BRANCH, LOGS_CHECKOUT_DIR } from './framework-dir.js' -import { fileBranchPath, nodeGitRunner } from '@gemstack/agent-data' +import { DATA_BRANCH, fileBranchPath, nodeGitRunner } from '@gemstack/agent-data' const CWD = '/repo' -const DATA = join(CWD, LOGS_CHECKOUT_DIR) +const DATA = fileBranchPath(CWD, DATA_BRANCH) const T0 = Date.parse('2026-08-23T10:00:00.000Z') /** An in-memory data checkout behind a fake funnel, like ticket-locks.test.ts's. */ @@ -120,7 +119,7 @@ test("on boot, this machine's locks whose run is gone are released; a run still // Against real git: two clones of one bare origin, each with its own logs checkout — the // cross-machine race the lock exists for. No funnel fake: what the other machine sees is what -// origin's `agents-logs` holds after the push. +// origin's `agent-data` holds after the push. const git = nodeGitRunner() const RETRIED_RM = { recursive: true, force: true, maxRetries: 10 } as const @@ -136,7 +135,7 @@ async function initRepo(prefix: string, email: string): Promise { return repo } -test('two machines sharing agents-logs: the second finds the first machine\'s lock on origin, and its release frees it (#1659, real git)', async () => { +test('two machines sharing agent-data: the second finds the first machine\'s lock on origin, and its release frees it (#1659, real git)', async () => { const laptop = await initRepo('framework-routine-lock-a-', 'a@a') const bare = await realpath(await mkdtemp(join(tmpdir(), 'framework-routine-lock-bare-'))) await git(['init', '--bare', bare], bare) @@ -149,10 +148,10 @@ test('two machines sharing agents-logs: the second finds the first machine\'s lo await git(['config', 'user.name', 'b'], desktop) try { assert.deepEqual(await acquireRoutineLock(laptop, 'triage-quick', { host: 'laptop' }), { ok: true }) - const onLaptop = await readFile(join(fileBranchPath(laptop, LOGS_BRANCH), 'routines', 'triage-quick.lock.md'), 'utf8') + const onLaptop = await readFile(join(fileBranchPath(laptop, DATA_BRANCH), 'routines', 'triage-quick.lock.md'), 'utf8') assert.match(onLaptop, /^CLAIMED: laptop\nSINCE: \d{4}-/) - // Pushed: origin's agents-logs carries it. - assert.equal((await git(['show', `${LOGS_BRANCH}:routines/triage-quick.lock.md`], bare)).trim(), onLaptop.trim()) + // Pushed: origin's agent-data carries it. + assert.equal((await git(['show', `${DATA_BRANCH}:routines/triage-quick.lock.md`], bare)).trim(), onLaptop.trim()) const onDesktop = await acquireRoutineLock(desktop, 'triage-quick', { host: 'desktop' }) assert.equal(onDesktop.ok, false) @@ -160,13 +159,13 @@ test('two machines sharing agents-logs: the second finds the first machine\'s lo // The desktop cannot release what the laptop holds. assert.equal(await releaseRoutineLock(desktop, 'triage-quick', { host: 'desktop' }), true) - assert.equal(await git(['show', `${LOGS_BRANCH}:routines/triage-quick.lock.md`], bare).then(() => true, () => false), true) + assert.equal(await git(['show', `${DATA_BRANCH}:routines/triage-quick.lock.md`], bare).then(() => true, () => false), true) // The laptop's run ends: its release reaches origin, and the desktop's next try takes the lock. assert.equal(await releaseRoutineLock(laptop, 'triage-quick', { host: 'laptop' }), true) - assert.equal(await git(['show', `${LOGS_BRANCH}:routines/triage-quick.lock.md`], bare).then(() => true, () => false), false) + assert.equal(await git(['show', `${DATA_BRANCH}:routines/triage-quick.lock.md`], bare).then(() => true, () => false), false) assert.deepEqual(await acquireRoutineLock(desktop, 'triage-quick', { host: 'desktop' }), { ok: true }) - assert.match((await git(['show', `${LOGS_BRANCH}:routines/triage-quick.lock.md`], bare)).trim(), /^CLAIMED: desktop/) + assert.match((await git(['show', `${DATA_BRANCH}:routines/triage-quick.lock.md`], bare)).trim(), /^CLAIMED: desktop/) } finally { for (const dir of [laptop, bare, desktopParent]) await rm(dir, RETRIED_RM) } diff --git a/packages/framework/src/routine-locks.ts b/packages/framework/src/routine-locks.ts index 141d05fa8..a520ff706 100644 --- a/packages/framework/src/routine-locks.ts +++ b/packages/framework/src/routine-locks.ts @@ -1,30 +1,29 @@ import { hostname } from 'node:os' import { join } from 'node:path' -import { nodeBranchFileFs, withFileBranch, type BranchFileFs, type CommitMessage, type FileBranchWrite } from '@gemstack/agent-data' -import { LOGS_BRANCH } from './framework-dir.js' +import { DATA_BRANCH, nodeBranchFileFs, withFileBranch, type BranchFileFs, type CommitMessage, type FileBranchWrite } from '@gemstack/agent-data' -// The "one triage at a time" guard (#1659): a `routines/.lock.md` on the logs branch. +// The "one triage at a time" guard (#1659): a `routines/.lock.md` on the data branch. // // The pinned branch (#1293) was this guard before, and stopped being one: since #1644 a triage // commits nothing on its branch, so the branch never reaches origin and guards nothing across // machines, and locally it was a name the *agent* checked from its prompt — every false abort // spent a started agent. The lock is the ticket claim (#1420) applied to a routine: minted by the -// sweep through the logs branch's write funnel before the run starts, so the daemon decides and no -// agent is started to find out, and read by every machine that shares `agents-logs`. +// sweep through the data branch's write funnel before the run starts, so the daemon decides and no +// agent is started to find out, and read by every machine that shares `agent-data`. // // Release is the daemon's, not a PR's: a triage never opens one. The daemon that minted a lock // drops it when the run ends, whatever the ending, and on boot for any it holds whose run is // gone — so a crash on this machine frees the routine at once. A lock another machine left // behind counts as dead after {@link ROUTINE_LOCK_TTL_MS}, fixed: a triage over hundreds of -// tickets can take hours, and a heartbeat would cost code and `agents-logs` churn. +// tickets can take hours, and a heartbeat would cost code and `agent-data` churn. -/** Where routine state lives on the logs branch; only the lock files, for now (#1660). */ +/** Where routine state lives on the data branch, beside the skills' files; only the lock files, for now (#1660). */ const ROUTINES_DIR = 'routines' /** How long a lock stands before whoever finds it may take it over: four hours, no heartbeat. */ export const ROUTINE_LOCK_TTL_MS = 4 * 60 * 60 * 1000 -/** The lock file for a routine, relative to the logs branch root: `routines/triage-quick.lock.md`. */ +/** The lock file for a routine, relative to the data branch root: `routines/triage-quick.lock.md`. */ export function routineLockPath(name: string): string { return `${ROUTINES_DIR}/${name}.lock.md` } @@ -50,7 +49,7 @@ export function routineLockHolder(md: string): RoutineLockHolder | undefined { /** Injectable seams so every operation is unit-testable off disk and git. */ export interface RoutineLockDeps extends Partial { - /** The logs branch's write funnel (default the persistent checkout's cycle); a test's fake stands in. */ + /** The data branch's write funnel (default the persistent checkout's cycle); a test's fake stands in. */ funnel?: (root: string, message: CommitMessage, op: (dir: string) => Promise) => Promise log?: (message: string) => void /** The machine claiming the lock (default this host's name). */ @@ -65,7 +64,7 @@ function resolve(deps: RoutineLockDeps) { write: deps.write ?? fs.write, remove: deps.remove ?? fs.remove, list: deps.list ?? fs.list, - funnel: deps.funnel ?? ((root: string, message: CommitMessage, op: (dir: string) => Promise) => withFileBranch(root, LOGS_BRANCH, message, op)), + funnel: deps.funnel ?? ((root: string, message: CommitMessage, op: (dir: string) => Promise) => withFileBranch(root, DATA_BRANCH, message, op)), log: deps.log ?? (() => {}), host: deps.host ?? hostname(), now: deps.now ?? (() => Date.now()), @@ -84,7 +83,7 @@ export type AcquireRoutineLockResult = | { ok: false; reason: string } /** - * Take the lock for `name` on the logs branch, in one funneled cycle. An alive lock stands the + * Take the lock for `name` on the data branch, in one funneled cycle. An alive lock stands the * caller down naming its holder; an expired one is taken over in the same commit. The funnel * re-runs the op when a push loses a race, so a lock another machine minted meanwhile is found, * not overwritten — and this machine's own lock seen again on the re-run is still ours. diff --git a/packages/framework/src/store/SPEC.md b/packages/framework/src/store/SPEC.md index fa1346728..58666c581 100644 --- a/packages/framework/src/store/SPEC.md +++ b/packages/framework/src/store/SPEC.md @@ -4,7 +4,7 @@ The record is the event log: each agent appends its orchestration events, one li Agents run concurrently because each gets its own git worktree under `.branches/`, in a directory named as its agent branch (born `agent-`). The git side of that — creating the checkout, sharing the parent's dependencies into it, and the retention rule under which it is reclaimed once its branch tip is on the remote — is the `skill-branches` package's; the store reads the checkouts it finds there and the agent records inside them. Teardown never loses work: nothing is committed on an agent's behalf, a checkout holding uncommitted work is kept, and one retention question, "is this recoverable?", stands in place of asking how the agent ended. -A finished agent's history is archived as `.jsonl` + `.json`: lastingly under `agents//` at the root of the logs branch's checkout — on the branch so it survives cleans and stays out of code history, per user so two machines write side by side — and transiently under the untracked `.the-framework/agents/` for agents without a worktree and for crash rescue. The history a project shows is every user's archive plus the transient one, with live agents prepended and winning over their own archived copies (a continued agent is both at once), newest first by agent id — the id is the start time made filesystem-safe, so id order is time order and old records are skipped by filename alone. +A finished agent's lasting record is the `logs` skill's run on the data branch — its card (the agent meta, the skill's fields on top and the rest under the one key the skill keeps for its caller) and its diary (the event log, four kinds of line the skill knows and every other event as it is), which the store maps both ways (`run-record`) — and transiently `.jsonl` + `.json` under the untracked `.the-framework/agents/` for agents without a worktree and for crash rescue. The history a project shows is the branch's runs plus the transient archive, with live agents prepended and winning over their own archived copies (a continued agent is both at once), newest first by agent id — the id is the start time made filesystem-safe, so id order is time order and old records are skipped by filename alone. The store also owns finding things again — an agent id resolves to the checkout to act on (a live agent's own checkout, else its worktree directory, else the project root) and to the events journal to tail (an ended agent's archived log wins over the root journal), one shared resolution so surfaces cannot drift — and healing: an agent whose meta says `running` while its owning process is provably gone is flipped to `stopped`, handed the `end` event its process never wrote so parked questions expire, and archived. diff --git a/packages/framework/src/store/agent-checkout.test.SPEC.md b/packages/framework/src/store/agent-checkout.test.SPEC.md index a687ce3b5..decc4e03c 100644 --- a/packages/framework/src/store/agent-checkout.test.SPEC.md +++ b/packages/framework/src/store/agent-checkout.test.SPEC.md @@ -2,7 +2,7 @@ What the tests cover: which event log a view scoped to one agent follows. - A missing or unsafe agent id falls back to the project root's event log. - An agent whose worktree still exists follows that worktree's own event log. -- An agent whose worktree is gone but whose archive exists follows the archived event log rather than the project root's — including when the archive is filed under a user directory on the logs branch. +- An agent whose worktree is gone but whose archive exists follows the archived event log rather than the project root's — including when the run is filed under a person's directory on the data branch. - A live worktree wins over an archive left from an earlier stint, so a resumed agent streams its current output. - An agent id with neither worktree nor archive still falls back to the project root's event log. diff --git a/packages/framework/src/store/agent-checkout.test.ts b/packages/framework/src/store/agent-checkout.test.ts index 10a692a8e..dfbba5a18 100644 --- a/packages/framework/src/store/agent-checkout.test.ts +++ b/packages/framework/src/store/agent-checkout.test.ts @@ -6,7 +6,9 @@ import { mkdtemp, mkdir, writeFile, rm } from 'node:fs/promises' import { resolveAgentEventsPath } from './agent-checkout.js' import { EVENTS_FILE, ARCHIVE_DIR } from './agent-store.js' import { worktreePath } from '@gemstack/skill-branches' -import { LOGS_CHECKOUT_DIR, THE_FRAMEWORK_DIR } from '../framework-dir.js' +import { THE_FRAMEWORK_DIR } from '../framework-dir.js' +import { DATA_BRANCH, fileBranchPath } from '@gemstack/agent-data' +import { RUNS_DIR } from '@gemstack/skill-logs' // resolveAgentEventsPath probes the real filesystem (same as resolveAgentCheckout), so these // tests build a throwaway project directory rather than a memory fs. @@ -59,10 +61,10 @@ test('resolveAgentEventsPath: an ended run (worktree gone) resolves to its archi } }) -test('resolveAgentEventsPath: finds an archive filed under a user dir on the data branch (#1179/#1582)', async () => { +test('resolveAgentEventsPath: finds a run filed under a person on the data branch (#1179/#1582/#1769)', async () => { const cwd = await makeProject() try { - const events = await seedArchive(cwd, join(cwd, LOGS_CHECKOUT_DIR, ARCHIVE_DIR, 'someone')) + const events = await seedArchive(cwd, join(fileBranchPath(cwd, DATA_BRANCH), RUNS_DIR, 'someone')) assert.equal(await resolveAgentEventsPath(cwd, RUN_ID), events) } finally { await rm(cwd, { recursive: true, force: true }) diff --git a/packages/framework/src/store/agent-store.SPEC.md b/packages/framework/src/store/agent-store.SPEC.md index c763fd815..ec1a856a8 100644 --- a/packages/framework/src/store/agent-store.SPEC.md +++ b/packages/framework/src/store/agent-store.SPEC.md @@ -15,12 +15,12 @@ The durable record of every agent: its event log, the agent meta derived from th - **The event log is the record; the agent meta is its summary** - every fact about an agent is an event, and the agent meta is those events folded together. - **An agent id is its start time** - ids sort chronologically as plain text, so "newest first" needs no timestamps parsed. - **Continuing an agent reopens its record** - a follow-up writes into the same agent rather than creating a second one, and keeps the original label. -- **A finished agent is archived under its user on the logs branch** - the lasting record is per user and off the code history; a second, throwaway copy covers agents with no worktree. +- **A finished agent's lasting record is the `logs` skill's run** - the agent meta and event log are mapped onto the skill's card and diary on the data branch, and read back the same way; a second, throwaway copy covers agents with no worktree. - **A crash never loses history** - an agent that never closed cleanly is archived by whoever notices next. - **A dead agent is forced to an end** - an agent whose owning process is gone is credited with the ending it never wrote, so nothing shows as live or awaiting an answer forever. - **The agent meta survives being read mid-write** - a reader never sees a half-written record and so never reports a live agent as gone. - **A project's agents are the live ones plus the archived ones** - composed once, with the live copy always winning. -- **Facts learned after the agent is gone are patched onto its archive** - the branch and pull request that only became knowable later. +- **Facts learned after the agent is gone are patched onto its card** - the branch and pull request that only became knowable later, through the `logs` skill. ## Business logic @@ -70,7 +70,7 @@ An id is required to be path-safe — letters, digits, dashes and underscores on The daemon allocates the id before it spawns the agent, because the worktree directory is named with it; the agent adopts that id rather than deriving a second, slightly later one. -A history read that only wants recent agents states a cutoff, and archived records older than that are rejected by their filename alone — most of a long history is skipped without being read at all. Only a name that parses as one of these ids can be rejected this way; any other name is read normally. +A history read that only wants recent agents states a cutoff: a run on the data branch is kept by the start time on its card, and a record in the throwaway archive older than the cutoff is rejected by its filename alone. Only a name that parses as one of these ids can be rejected that way; any other name is read normally. ### Continuing an agent reopens its record @@ -86,23 +86,28 @@ A reopened agent keeps its original label. The continuation's own request would The flow an agent started under is recorded so a continuation can re-enter it: without that record, continuing a build agent would drop it into the plain prompt path, losing the framing and the backlog loop that belong to a build. -Continuing also needs the agent's history back in the checkout it reads from. Teardown moved that history into the repo, so restoring puts the archived event log and agent meta back into the worktree — unless the worktree already holds a live agent, whose own log is newer and must not be overwritten. +Continuing also needs the agent's history back in the checkout it reads from. Teardown recorded that history on the data branch, so restoring reads the run back through the `logs` skill — its diary as the framework's events, its card as the agent meta — and puts them into the worktree; a run the branch does not have is looked for in the throwaway archive. A worktree that already holds a live agent is left alone: its own log is newer and must not be overwritten. -### A finished agent is archived under its user on the logs branch +### A finished agent's lasting record is the `logs` skill's run #### User story -The user, and their teammates, can read months later what agents did to the repo — including agents whose throwaway checkouts are long gone. +The user, and their teammates, can read months later what agents did to the repo — including agents whose throwaway checkouts are long gone — and an agent starting on a ticket can read how the last run on it went. #### Business logic -When an agent closes, its event log and agent meta are copied out as a pair named after its id. The lasting home is the user's own directory on the logs branch's checkout: on the logs branch so the record survives cleaning the repo and never touches the code history, and per user so two people's machines write side by side instead of colliding. A second, throwaway home inside the agent's own framework directory covers an agent with no worktree of its own, and the crash rescue. +When an agent's process is gone, the daemon reads its record out of its worktree — the agent meta, ended if the process died running, with the branch actually holding its commits stamped on, since this is the last moment it can be observed — and records it on the data branch as the `logs` skill's run, under the identity the repo commits as, through the branch's write cycle. The mapping is the store's and works both ways: -Archiving is what makes teardown safe. An agent writes its record inside its own worktree, so deleting the worktree would delete the record with it; the copy into the repo happens first. The copy inside the worktree is deliberately left untracked, or it would be committed onto the agent's own branch and collide with the lasting copy at merge time. +- The card is the agent meta with the skill's eleven fields on top — id, start and end time, status, intent, driver, model, branch, pull request, ticket, cost — and every other field of the meta under the one key the skill stores for its caller and never reads. A meta unfolds back out of a card the same way. +- The diary is the event log with four events written as the four kinds of line the skill knows — what the agent said, its result, the run's cost and its ending — and every other event written as it is, under its own kind. A diary reads back as the same events, so a run page replays it, a continuation restores it and a tail of an ended run follows it exactly as they did the live log. -At the moment of teardown, the branch actually holding the agent's commits is read from the checkout and stamped onto the archived record — the last moment it can be observed at all. +The end time and the cost are two facts of the agent meta that exist for the card: the time the ending was folded, and the cost summed over the run's usage events. -Reading a project's history means reading every user's archive directory plus the throwaway one, because the history is a team-visible record of what has been done to the repo. The same agent can appear in more than one place — rescued into the throwaway copy and archived into the committed one — so records are de-duplicated by id, with the committed copy winning. Unreadable or half-written records are skipped rather than failing the whole read. Archiving the same agent twice is harmless. +A second, throwaway home inside the agent's own framework directory covers an agent with no worktree of its own, and the crash rescue: there the event log and agent meta are copied as a pair named after the id, unmapped. + +Recording is what makes teardown safe. An agent writes its record inside its own worktree, so deleting the worktree would delete the record with it; the record on the branch comes first. The copy inside the worktree is deliberately left untracked, or it would be committed onto the agent's own branch and collide with the lasting copy at merge time. + +Reading a project's history means reading the branch's runs — every person's — plus the throwaway archive, because the history is a team-visible record of what has been done to the repo. The same agent can appear in both places — rescued into the throwaway copy and recorded on the branch — so records are de-duplicated by id, with the branch's copy winning. Unreadable or half-written records are skipped rather than failing the whole read. Recording the same agent twice is harmless. An archived agent's event log can be replayed in full, applying the same tolerance for a torn last line as the live log. @@ -132,7 +137,7 @@ The two places that check treat that third state differently on purpose. The che Healing an orphan does not merely change its outcome: an ending is appended to its event log on its behalf, recording that the process died without reporting one, and that ending is then folded in like any other. Every surface keys "this agent is over" off that one ending, so a death that skipped it would leave the agent's last question rendering as answerable forever. Folding it also closes the gate the agent died holding. The healed agent is then archived, so its history is kept. -The boot-time reconciliation covers all three places an orphan can hide: archived records still marked running, the live record at the project root, and an agent inside a worktree. A worktree agent is healed in place and copied into the repo's history, but its worktree is left on disk — an agent that ended this way did not end cleanly, and those are kept for inspection; removing one is an explicit action. The count of agents reconciled is reported back. Every step is best-effort: a failure skips that agent rather than failing the sweep. +The boot-time reconciliation covers all four places an orphan can hide: a run on the data branch still marked running, which is ended through the `logs` skill's write so the ending is a commit rather than an edit the next sync would reset; a record in the throwaway archive; the live record at the project root; and an agent inside a worktree. A worktree agent is healed in place and copied into the throwaway archive, but its worktree is left on disk — an agent that ended this way did not end cleanly, and those are kept for inspection; removing one is an explicit action. The count of agents reconciled is reported back. Every step is best-effort: a failure skips that agent rather than failing the sweep. #### Rationale @@ -174,7 +179,7 @@ The full list is the live agents followed by the archived ones that are not alre This composition — not either half on its own — is what callers actually want, which is why it is defined once here; three separate parts of the product had each grown their own copy of it. -### Facts learned after the agent is gone are patched onto its archive +### Facts learned after the agent is gone are patched onto its card #### User story @@ -182,7 +187,7 @@ An agent finishes, and only afterwards does the pull request get opened, or does #### Business logic -An archived agent's record can be amended with the branch its work landed on and the pull request its work is on. There is no event stream left to carry these facts, and every surface reads the record, so this single amendment is what turns an empty-looking row into its real outcome. Because the archive lives on the logs branch's checkout, an amendment is only durable once committed, so callers outside tests go through the logs branch's writer rather than writing directly. +A run's card can be amended with the branch its work landed on and the pull request its work is on — the `logs` skill's patch, one commit on the data branch. There is no event stream left to carry these facts, and every surface reads the card, so this single amendment is what turns an empty-looking row into its real outcome. The callers are the cloud-work adoption and the dashboard's Open PR action; the store itself no longer patches anything. ## Before modifying/creating SPEC.md files diff --git a/packages/framework/src/store/agent-store.test.SPEC.md b/packages/framework/src/store/agent-store.test.SPEC.md index b1294945c..56a787993 100644 --- a/packages/framework/src/store/agent-store.test.SPEC.md +++ b/packages/framework/src/store/agent-store.test.SPEC.md @@ -33,16 +33,17 @@ What the tests cover: how an agent's record is written, summarized, archived and - Closing copies the agent's log and agent meta into the archive, and the history lists them newest first with what each was asked for and how each ended. - Starting a fresh agent first rescues a prior agent that crashed without closing, so its history survives. -- A worktree agent's record is copied into the repo before its checkout can be removed, and an agent still marked running at that moment is recorded as stopped. -- A named user files the archive under their own directory on the logs branch instead of the throwaway one. -- The history lists every user's archive plus the throwaway one, under their one current name only; an agent filed in two places is listed once; an archived log replays wherever it is filed. -- A history read given a cutoff skips older records by filename without ever opening them, while a record whose id is not one of the framework's timestamps is still read. +- A worktree agent's record is read out for the daemon to record on the data branch before its checkout can be removed — its missing ending written, the branch stamped, nothing landing in the throwaway archive — and the boot rescue copies such a record into the throwaway archive, an agent still marked running recorded as stopped. +- The history lists the branch's runs, every person's, plus the throwaway archive, under their one current name only; an agent filed in two places is listed once; an archived log replays wherever it is filed. +- A history read given a cutoff keeps a run on the branch by the start time on its card, whatever its id, and skips an older record in the throwaway archive by its filename without opening it. +- A run on the branch replays as the framework's events — the skill's four kinds of line mapped back, every other line as it is — is listed with its card's caller fields unfolded into the meta, and restores into a worktree as that event log and that meta. +- A run on the branch stuck at running is ended through the skill's write funnel: the card ends and dates it, keeps the framework's fields under caller, and the diary gains the ending. **Liveness and self-healing** - A fresh start records the process and host that own the agent. - Reading a live agent whose owning process is gone flips it to stopped and archives it; one whose process is alive is left alone; one with no owner recorded, or an owner on another machine, is left alone for the boot reconciliation to handle. -- The boot reconciliation flips archived agents stuck at running, flips and archives the live one counting it once, rescues an agent a crashed daemon left inside a worktree, and covers an agent archived under a user directory. It leaves alone any agent whose process is provably alive on this host, and does nothing on a clean or empty workspace. +- The boot reconciliation flips throwaway-archived agents stuck at running, flips and archives the live one counting it once, and rescues an agent a crashed daemon left inside a worktree. It leaves alone any agent whose process is provably alive on this host, and does nothing on a clean or empty workspace. - Healing a dead agent writes the ending it never wrote: the ending lands in the live log, in the worktree's own log and in the archived copy, and the gate the agent died holding is cleared everywhere. An agent that wrote its own ending gets no second one. **Surviving concurrent reads and writes** @@ -66,11 +67,6 @@ What the tests cover: how an agent's record is written, summarized, archived and - Continuing with nothing to reopen starts a fresh agent. - An archived agent's history is restored into its worktree so the continuation reads its own past; a checkout that already holds a live agent is left untouched, and a missing archive is a no-op. -**Patching an archive afterwards** - -- A pull request opened after the agent's process is gone, and the branch a cloud session's work landed on, are patched onto the archived record and read back by the history. -- Patching an unknown or unsafe agent id changes nothing and reports that it did nothing. - ## Before modifying/creating SPEC.md files You must always read and respect https://raw.githubusercontent.com/brillout/sdd/refs/heads/main/sdd.md diff --git a/packages/framework/src/store/agent-store.test.ts b/packages/framework/src/store/agent-store.test.ts index 6ee43decc..48ded7c07 100644 --- a/packages/framework/src/store/agent-store.test.ts +++ b/packages/framework/src/store/agent-store.test.ts @@ -9,7 +9,7 @@ import { readLiveMeta, readLiveMetas, archiveWorktreeAgent, - patchArchivedAgent, + readWorktreeAgent, restoreArchivedAgent, reconcileOrphanedAgents, loadAgentEvents, @@ -18,7 +18,8 @@ import { type StoreFs, type AgentMeta, } from './agent-store.js' -import { LOGS_CHECKOUT_DIR } from '../framework-dir.js' +import { DATA_BRANCH, fileBranchPath } from '@gemstack/agent-data' +import type { LogsFunnel } from '@gemstack/skill-logs' import type { FrameworkEvent } from '../events.js' /** An in-memory {@link StoreFs} so the store logic is tested without touching disk. */ @@ -483,7 +484,7 @@ test('listAgents reads every user archive and the transient one, under their one // archives that, with no users, nobody has. const meta = (id: string): string => JSON.stringify({ status: 'done', id, startedAt: AT, updatedAt: AT, intent: id }) - const user = join(CWD, LOGS_CHECKOUT_DIR, 'agents', 'dev@example.com') + const user = join(fileBranchPath(CWD, DATA_BRANCH), 'agents', 'dev@example.com') const fs = memFs({ [join(user, '2026-new.json')]: meta('2026-new'), [join(CWD, '.the-framework', 'agents', '2026-transient.json')]: meta('2026-transient'), @@ -674,47 +675,29 @@ test('archiveWorktreeAgent records a run that died mid-flight as stopped, not ru assert.equal((JSON.parse(fs.files.get(join(CWD, '.the-framework', 'agents', 'r1.json'))!) as AgentMeta).status, 'stopped') }) -test('patchArchivedAgent records a PR opened after the run on its archived meta (E6)', async () => { - // The dashboard's Open PR button runs after the session's process is gone, so there is no event - // stream left to carry the fact — but it is the same fact, and every surface reads it from the - // same place either way. - const fs = memFs(worktreeFiles('r1', { version: 1, status: 'done', id: 'r1', startedAt: AT, updatedAt: AT })) - await archiveWorktreeAgent(worktreeAt('r1'), CWD, fs) - assert.equal(await patchArchivedAgent(CWD, 'r1', { pr: { number: 42, url: 'https://x/pull/42' } }, fs), true) - assert.deepEqual((await listAgents(CWD, fs)).find(r => r.id === 'r1')?.pr, { number: 42, url: 'https://x/pull/42' }) -}) - -test('patchArchivedAgent records the branch the cloud work landed on (#1601)', async () => { - // The cloud VM pushes its `claude/*` branch after the wrapper's process is gone, so the fact - // arrives the same way a late PR does: patched onto the archive, read by every surface. - const fs = memFs(worktreeFiles('r1', { version: 1, status: 'done', id: 'r1', startedAt: AT, updatedAt: AT, branch: 'agent-r1' })) - await archiveWorktreeAgent(worktreeAt('r1'), CWD, fs) - assert.equal(await patchArchivedAgent(CWD, 'r1', { branch: 'claude/fix-the-thing' }, fs), true) - assert.equal((await listAgents(CWD, fs)).find(r => r.id === 'r1')?.branch, 'claude/fix-the-thing') -}) - -test('patchArchivedAgent leaves the record as it was when there is nothing to patch (E6/#1601)', async () => { - // Best-effort: the cost of missing it is one surface having to ask gh, which is what all of them - // used to do anyway. - const fs = memFs() - assert.equal(await patchArchivedAgent(CWD, 'nope', { pr: { number: 1, url: 'u' } }, fs), false) - assert.equal(await patchArchivedAgent(CWD, '../escape', { branch: 'claude/x' }, fs), false, 'and an unsafe id is refused') -}) - test('archiveWorktreeAgent is forgiving of a worktree with no run', async () => { assert.equal(await archiveWorktreeAgent(worktreeAt('nope'), CWD, memFs()), undefined) }) const USER = 'git@brillout.com' const archiveAt = (id: string, ext: string) => - join(CWD, LOGS_CHECKOUT_DIR, 'agents', USER, `${id}.${ext}`) + join(fileBranchPath(CWD, DATA_BRANCH), 'agents', USER, `${id}.${ext}`) -test('a named user files the archive under their own sessions, not runs/ (#1179)', async () => { - // The whole point: `agents/` is gitignored, so a `git clean -fdx` took every session with it. - const fs = memFs(worktreeFiles('r1', { version: 1, status: 'done', id: 'r1', startedAt: AT, updatedAt: AT }, '{"kind":"log","message":"hi"}\n')) - await archiveWorktreeAgent(worktreeAt('r1'), CWD, fs, undefined, USER) - assert.equal(fs.files.get(archiveAt('r1', 'jsonl')), '{"kind":"log","message":"hi"}\n') - assert.equal(fs.files.has(join(CWD, '.the-framework', 'agents', 'r1.json')), false, 'and not in the transient dir') +/** The `logs` skill's funnel over the in-memory checkout: runs the op against it, reports a landed push. */ +const memFunnel: LogsFunnel = async (_root, _message, op) => { + await op(fileBranchPath(CWD, DATA_BRANCH)) + return { ok: true, changed: true, pushed: true } +} + +test('readWorktreeAgent hands the daemon what it records on the data branch: the meta, ended and stamped, and the events (#1769)', async () => { + // The lasting record is the `logs` skill's run on the branch; the daemon writes it from this read. + const fs = memFs(worktreeFiles('r1', { version: 1, status: 'running', id: 'r1', startedAt: AT, updatedAt: AT }, '{"kind":"log","message":"hi"}\n')) + const run = await readWorktreeAgent(worktreeAt('r1'), fs, 'agent-fixed') + assert.equal(run?.meta.status, 'stopped', 'a run whose process is gone did not close') + assert.equal(run?.meta.branch, 'agent-fixed', 'the branch is stamped: the checkout is about to go') + assert.deepEqual(run?.events.map(e => e.kind), ['log', 'end'], 'the missing end is written before the copy') + assert.equal(fs.files.has(join(CWD, '.the-framework', 'agents', 'r1.json')), false, 'nothing lands in the transient archive') + assert.equal(await readWorktreeAgent(worktreeAt('nope'), memFs()), undefined) }) test('the history lists every user, and the runs archived before this shipped (#1179)', async () => { @@ -724,36 +707,34 @@ test('the history lists every user, and the runs archived before this shipped (# const fs = memFs({ [join(CWD, '.the-framework', 'agents', 'r1.json')]: done('r1'), [archiveAt('r2', 'json')]: done('r2'), - [join(CWD, LOGS_CHECKOUT_DIR, 'agents', 'someone@else.com', 'r3.json')]: done('r3'), + [join(fileBranchPath(CWD, DATA_BRANCH), 'agents', 'someone@else.com', 'r3.json')]: done('r3'), }) assert.deepEqual((await listAgents(CWD, fs)).map(agent => agent.id), ['r3', 'r2', 'r1']) }) -test('a `since` skips the older runs by filename, without reading them (#1607)', async () => { +test('a `since` keeps the runs started at or after it, on the branch by their card and in the transient archive by their name (#1607)', async () => { // The cost this exists to remove: the adoption poll wants the last 48h and an archive holds - // years, so the read loop spent every pass parsing records its caller was about to discard. + // years, so the read loop used to parse records its caller was about to discard. const at = (iso: string) => JSON.stringify({ version: 1, status: 'done', id: agentIdFromStartedAt(iso), startedAt: iso, updatedAt: iso }) const old = agentIdFromStartedAt('2026-07-04T00:00:00.000Z') const recent = agentIdFromStartedAt('2026-07-06T00:00:00.000Z') const fs = memFs({ [archiveAt(old, 'json')]: at('2026-07-04T00:00:00.000Z'), [archiveAt(recent, 'json')]: at('2026-07-06T00:00:00.000Z'), - [archiveAt('hand-picked', 'json')]: JSON.stringify({ version: 1, status: 'done', id: 'hand-picked', startedAt: AT, updatedAt: AT }), + // An id that is not a timestamp is dated by its card, like every run on the branch. + [archiveAt('hand-picked', 'json')]: JSON.stringify({ version: 1, status: 'done', id: 'hand-picked', startedAt: '2026-07-05T12:00:00.000Z', updatedAt: AT }), + [archiveAt('hand-picked-old', 'json')]: JSON.stringify({ version: 1, status: 'done', id: 'hand-picked-old', startedAt: '2026-07-01T12:00:00.000Z', updatedAt: AT }), + [join(RUNS, `${old}.json`)]: at('2026-07-04T00:00:00.000Z'), + [join(RUNS, 'r-transient.json')]: JSON.stringify({ status: 'done', id: 'r-transient', startedAt: '2026-07-07T00:00:00.000Z', updatedAt: AT }), }) const read: string[] = [] const spy = { ...fs, read: async (path: string) => (read.push(path), fs.read(path)) } const listed = await listAgents(CWD, spy, Date.parse('2026-07-05T00:00:00.000Z')) - assert.deepEqual(listed.map(agent => agent.id).sort(), ['hand-picked', recent].sort()) - assert.equal( - read.some(path => path.includes(old)), - false, - 'the run outside the window is never opened', - ) - // An id that is not one of our timestamps cannot be dated from its name, so it is still read: - // rejecting it unread would hide it from every caller that passes a window. - assert.equal(read.some(path => path.includes('hand-picked')), true) + assert.deepEqual(listed.map(agent => agent.id).sort(), ['hand-picked', 'r-transient', recent].sort()) + // The transient archive is dated by the file name, so the run outside the window is never opened. + assert.equal(read.some(path => path === join(RUNS, `${old}.json`)), false) }) test('a run archived under both schemes is listed once (#1179)', async () => { @@ -773,14 +754,51 @@ test('an archived log replays wherever it is filed (#1179)', async () => { assert.deepEqual(await loadAgentEvents(CWD, 'r1', fs), [{ kind: 'log', message: 'replayed' }]) }) -test('a committed session stuck at running is reconciled too (#1179)', async () => { +test('a run on the branch stuck at running is ended through the skill\'s write, as a commit (#1179/#1769)', async () => { // The boot reconcile used to sweep only `agents/`, so a crashed agent archived under a user would - // have shown as live forever, with a Stop that does nothing. + // have shown as live forever, with a Stop that does nothing. And it used to edit the card in + // place on the checkout, where the next sync's reset lost it: now it goes through the funnel. const fs = memFs({ - [archiveAt('r1', 'json')]: JSON.stringify({ version: 1, status: 'running', id: 'r1', startedAt: AT, updatedAt: AT }), + [archiveAt('r1', 'json')]: JSON.stringify({ status: 'running', id: 'r1', startedAt: AT, caller: { updatedAt: AT, pid: 7 } }), + [archiveAt('r1', 'jsonl')]: '{"kind":"said","text":"hi"}\n', }) - assert.equal(await reconcileOrphanedAgents(CWD, fs, () => false), 1) - assert.equal((JSON.parse(fs.files.get(archiveAt('r1', 'json'))!) as AgentMeta).status, 'stopped') + assert.equal(await reconcileOrphanedAgents(CWD, fs, () => false, memFunnel), 1) + const card = JSON.parse(fs.files.get(archiveAt('r1', 'json'))!) as { status: string; endedAt?: string; caller: { pid: number } } + assert.equal(card.status, 'stopped') + assert.ok(card.endedAt, 'the ending is dated') + assert.equal(card.caller.pid, 7, 'the framework\'s own fields stay under caller') + assert.equal(fs.files.get(archiveAt('r1', 'jsonl')), '{"kind":"said","text":"hi"}\n{"kind":"ended","status":"stopped","detail":"its process died without reporting an end"}\n') + assert.equal((await listAgents(CWD, fs)).find(a => a.id === 'r1')?.status, 'stopped') +}) + +test('a run on the branch replays and restores as the framework\'s events, its card as the meta (#1769)', async () => { + const fs = memFs({ + [archiveAt('r1', 'json')]: JSON.stringify({ status: 'failed', id: 'r1', startedAt: AT, endedAt: AT, cost: 0.5, caller: { updatedAt: AT, kind: 'build', pid: 9 } }), + [archiveAt('r1', 'jsonl')]: [ + '{"kind":"session","driver":"claude-code","workspace":"/w","fake":false}', + '{"kind":"said","text":"Reading."}', + '{"kind":"result","text":"Done.","sessionId":"s1"}', + '{"kind":"cost","usd":0.5,"inputTokens":1,"outputTokens":2,"cacheReadTokens":0,"cacheCreationTokens":0,"turns":1}', + '{"kind":"ended","status":"failed","detail":"API 500"}', + '', + ].join('\n'), + }) + const events = await loadAgentEvents(CWD, 'r1', fs) + assert.deepEqual(events, [ + { kind: 'session', driver: 'claude-code', workspace: '/w', fake: false }, + { kind: 'driver', event: { type: 'text', text: 'Reading.' } }, + { kind: 'driver', event: { type: 'result', text: 'Done.', sessionId: 's1' } }, + { kind: 'usage', costUsd: 0.5, inputTokens: 1, outputTokens: 2, cacheReadTokens: 0, cacheCreationTokens: 0, turns: 1 }, + { kind: 'end', ok: false, detail: 'API 500' }, + ]) + const listed = (await listAgents(CWD, fs)).find(a => a.id === 'r1')! + assert.equal(listed.kind, 'build', 'caller unfolds into the meta') + assert.equal(listed.cost, 0.5) + assert.equal(listed.status, 'failed') + const wt = join(CWD, '.branches', 'agent-r1') + assert.equal(await restoreArchivedAgent(CWD, wt, 'r1', fs), true) + assert.equal(fs.files.get(join(wt, '.the-framework', 'events.jsonl')), events!.map(e => JSON.stringify(e) + '\n').join('')) + assert.equal((JSON.parse(fs.files.get(join(wt, '.the-framework', 'agent.json'))!) as AgentMeta).pid, 9) }) test('reconcileOrphanedAgents rescues a run a crashed daemon left in a worktree (#737)', async () => { diff --git a/packages/framework/src/store/agent-store.ts b/packages/framework/src/store/agent-store.ts index 9c50748b5..50eff2039 100644 --- a/packages/framework/src/store/agent-store.ts +++ b/packages/framework/src/store/agent-store.ts @@ -4,7 +4,9 @@ import { hostname } from 'node:os' import type { AutoHandoffSkip, FrameworkEvent } from '../events.js' import { nodeFs } from '../node-fs.js' import { isSafeAgentId, worktreeDirEntries } from '@gemstack/skill-branches' -import { THE_FRAMEWORK_DIR, LOGS_CHECKOUT_DIR } from '../framework-dir.js' +import { THE_FRAMEWORK_DIR } from '../framework-dir.js' +import { findRun, listRuns, readDiary, runFiles, writeRun, type LogsDeps, type LogsFunnel } from '@gemstack/skill-logs' +import { eventsOf, fromRunCard, toDiaryLine, toRunCard } from './run-record.js' import { agentIdFromStartedAt, startedAtFromAgentId } from '../agent-id.js' /** @@ -24,18 +26,14 @@ export const EVENTS_FILE = 'events.jsonl' export const META_FILE = 'agent.json' /** - * Where finished agents are archived, under both placements that name has: the lasting - * `agents//` on the data branch's checkout (#1179/#1582), and the transient - * `.the-framework/agents/` that an agent with no worktree of its own — or one archiving inside - * its own throwaway checkout — writes into. {@link archiveDir}/{@link committedArchiveDir} are - * where a caller picks; both are read when a project's history is listed. + * The transient archive, `.the-framework/agents/`: where an agent with no worktree of its own — + * a project that is not a git repository — archives on close, where the crash rescue puts a run + * whose process died, and where an agent's own worktree keeps its copy. Untracked by design. The + * lasting record of a run is on the data branch, as the `logs` skill's card and diary (#1769): + * the daemon writes it there at teardown, and every history read lists both places. * * The live agent stays at `events.jsonl`/`agent.json` (the daemon tails it); on - * {@link AgentStore.close} a copy lands here as `.jsonl` + `.json` (#303), giving the - * history sidebar a per-agent log to replay. - * - * The name lives here rather than in `sessions.ts`, which owns the per-user naming: that module - * reads the store, so the constant travelling the other way would be a cycle. + * {@link AgentStore.close} a copy lands here as `.jsonl` + `.json` (#303). */ export const ARCHIVE_DIR = 'agents' @@ -56,6 +54,10 @@ export interface AgentMeta { startedAt: string /** ISO timestamp of the last event written. */ updatedAt: string + /** ISO timestamp the run ended (the `end` event), absent while it is going. The skill's card field. */ + endedAt?: string + /** What the run cost so far in US dollars, summed over its `usage` events; absent until one says. The skill's card field. */ + cost?: number /** * The OS pid of the process that owns this agent (the one tailing `control.jsonl`), on {@link host}. * Persisted so a reader can tell a live agent from one whose process died without writing `end` @@ -353,8 +355,12 @@ export function applyEventToMeta(meta: AgentMeta, event: FrameworkEvent, at: str // Any new turn means the agent is working again, so the agent is no longer parked (#785). if (event.event.type === 'start') delete next.settledAt break + case 'usage': + if (event.costUsd !== undefined) next.cost = (next.cost ?? 0) + event.costUsd + break case 'end': next.status = event.ok ? 'done' : event.stopped ? 'stopped' : 'failed' + next.endedAt = at delete next.pendingChoice // a finished run is not awaiting anything delete next.settledAt // nor is it waiting on you // The bridge dies with the agent, so a kept port would send the pane at whatever else @@ -631,61 +637,38 @@ export class AgentStore { } } -/** - * The transient archive inside a `.the-framework` dir: what an agent's own worktree copy and the - * crash rescue use. Untracked by design — the lasting copy lives on the data branch (#1582). - */ +/** The transient archive inside a `.the-framework` dir. See {@link ARCHIVE_DIR}. */ function archiveDir(dir: string): string { return join(dir, ARCHIVE_DIR) } -/** - * A user's lasting archive (#1179/#1582): `agents//` at the root of the data branch's - * checkout. On the branch, so the history survives `git clean -fdx` AND never touches the code - * history; per user, so two people's machines write side by side instead of conflicting. - */ -function committedArchiveDir(cwd: string, user: string): string { - return join(cwd, LOGS_CHECKOUT_DIR, ARCHIVE_DIR, user) -} - /** Paths of an agent's archived log + meta inside one archive directory. */ function archivePaths(agentsDir: string, id: string): { events: string; meta: string } { return { events: join(agentsDir, `${id}.jsonl`), meta: join(agentsDir, `${id}.json`) } } -/** - * Where one agent's archive actually sits, searched across {@link archiveDirs}, or `undefined` when - * it is nowhere. An agent id alone no longer names a path: which user archived it decides that, and a - * reader (the continue (#762), a removal) only has the id. - */ -async function findArchive(fs: StoreFs, cwd: string, agentId: string): Promise<{ events: string; meta: string } | undefined> { - for (const agentsDir of await archiveDirs(fs, cwd)) { - const paths = archivePaths(agentsDir, agentId) - if (await fs.exists(paths.meta)) return paths - } - return undefined +/** The transient archive of one agent under the project, or `undefined` when it has none. */ +async function findTransientArchive(fs: StoreFs, cwd: string, agentId: string): Promise<{ events: string; meta: string } | undefined> { + const paths = archivePaths(archiveDir(join(cwd, THE_FRAMEWORK_DIR)), agentId) + return (await fs.exists(paths.meta)) ? paths : undefined } /** - * Every directory a project's archived agents may sit in, committed first: each user's - * `agents//` on the data branch's checkout (#1582), then the transient - * `.the-framework/agents/` an agent with no worktree archives into. - * - * Every user's archive is listed, not just the reader's — the history is a team-visible record of - * what the agent has done to the repo, which is the point of committing it. - * - * A directory is recognized by having a readable archive child, so a stray file is simply not one - * (readdir yields `[]` for anything that is not a directory). + * The `logs` skill's seams over a {@link StoreFs}: the store reaches the runs on the data + * branch's checkout through the same injectable filesystem as its own files, so a test with an + * in-memory fs reads and writes the committed side as it does the transient one. The funnel is + * the skill's own (the persistent checkout's write cycle) unless a caller hands one in. */ -async function archiveDirs(fs: StoreFs, cwd: string): Promise { - const dirs: string[] = [] - const committed = join(cwd, LOGS_CHECKOUT_DIR, ARCHIVE_DIR) - for (const name of await fs.readdir(committed)) { - const candidate = join(committed, name) - if ((await fs.readdir(candidate)).length > 0) dirs.push(candidate) +function runDeps(fs: StoreFs, funnel?: LogsFunnel): LogsDeps { + return { + read: path => fs.read(path), + list: path => fs.readdir(path), + write: async (path, content) => { + await fs.mkdir(join(path, '..')) + await fs.write(path, content) + }, + ...(funnel ? { funnel } : {}), } - dirs.push(join(cwd, THE_FRAMEWORK_DIR, ARCHIVE_DIR)) - return dirs } /** @@ -716,8 +699,10 @@ async function archivePriorAgent(fs: StoreFs, dir: string): Promise { /** * Put an archived agent's history back where an agent reads it (#762), so a continued agent picks up its - * own log rather than starting empty. The inverse of {@link archiveWorktreeAgent}: teardown moved the - * history to the repo, and continuing needs it in the checkout again. + * own log rather than starting empty. The inverse of the teardown's recording: teardown moved the + * history onto the data branch, and continuing needs it in the checkout again — the run's diary + * as the framework's events, its card as the framework's meta. A run the branch does not have is + * looked for in the transient archive. * * A no-op when the worktree already holds a live agent (nothing to restore, and its log is newer), * or when there is no archive. Never throws. @@ -732,7 +717,15 @@ export async function restoreArchivedAgent( if (!isSafeAgentId(agentId)) return false const dir = join(worktree, THE_FRAMEWORK_DIR) if (await fs.exists(join(dir, META_FILE))) return false - const archive = await findArchive(fs, repo, agentId) + const card = await findRun(repo, agentId, runDeps(fs)) + if (card) { + const diary = (await readDiary(repo, agentId, runDeps(fs))) ?? [] + await fs.mkdir(dir) + await fs.write(join(dir, EVENTS_FILE), eventsOf(diary).map(event => JSON.stringify(event) + '\n').join('')) + await writeMetaFile(fs, join(dir, META_FILE), fromRunCard(card)) + return true + } + const archive = await findTransientArchive(fs, repo, agentId) if (!archive) return false await fs.mkdir(dir) await fs.write(join(dir, EVENTS_FILE), (await fs.exists(archive.events)) ? await fs.read(archive.events) : '') @@ -744,59 +737,63 @@ export async function restoreArchivedAgent( } /** - * Archive a worktree agent's history into the *main repo* (#737), returning the meta it archived. - * - * An agent writes its `agent.json` / `events.jsonl` inside its own worktree (#736), so deleting that - * worktree would delete the agent's history with it. This copies it into the repo, which is the one - * place the dashboard's history reads from, so teardown becomes safe. - * - * `user` files the copy under that user's lasting `agents//` on the data branch's checkout - * (#1179/#1582) instead of the transient `agents/`. It is this copy, not the one the agent left in - * its own worktree, that is meant to last: every agent in a git repo gets a worktree, so this is - * the only archive of it that outlives the checkout. The caller owns getting it committed — the - * daemon funnels this through the data branch's writer. The worktree's own copy deliberately - * stays untracked — it would otherwise be committed onto the agent's branch as well and collide - * with this one on merge. + * A worktree agent's history, read for recording once its process is gone (#737): its meta and + * its events, out of its own checkout. The worktree is about to go, so this is the last moment + * the run can be read. * - * A meta still marked `running` is flipped to `stopped` first: this runs when the process is - * already gone, so `running` means it died without closing (crash, kill -9), exactly the case - * {@link reconcileOrphanedAgents} handles for the project path. Idempotent per id, and forgiving: - * a worktree with no run, or an unreadable one, yields `undefined` rather than throwing. + * A meta still marked `running` is flipped to `stopped` first, in the worktree's own files too + * (#1359): this runs when the process is already gone, so `running` means it died without + * closing (crash, kill -9), and the log gains the `end` event the dead process never wrote before + * anything copies it. `branch` is stamped on the meta because the caller reads it off the checkout + * and this is the last moment it can be observed (#799). Forgiving: a worktree with no run, or + * an unreadable one, yields `undefined` rather than throwing. */ -export async function archiveWorktreeAgent( +export async function readWorktreeAgent( worktree: string, - repo: string, fs: StoreFs = nodeStoreFs(), branch?: string, - user?: string, -): Promise { +): Promise<{ meta: AgentMeta; events: FrameworkEvent[] } | undefined> { try { const worktreeDir = join(worktree, THE_FRAMEWORK_DIR) const live = await readMetaFile(fs, join(worktreeDir, META_FILE)) if (!live?.id || !isSafeAgentId(live.id)) return undefined - // The flip writes the worktree's own log + meta too (#1359): the death gains its `end` - // event before the archive copies the log, so no reader — live tail or archived replay — - // is left holding an open gate for a dead agent. const stopped: AgentMeta = live.status === 'running' ? await recordOrphanEnd(fs, worktreeDir, live) : live - // The branch is read from the checkout by the caller and stamped here, because this is the - // last moment it can be observed: the worktree is about to go (#799). const meta: AgentMeta = branch ? { ...stopped, branch } : stopped - const dest = user ? committedArchiveDir(repo, user) : archiveDir(join(repo, THE_FRAMEWORK_DIR)) - await archiveAgent(fs, dest, meta, join(worktreeDir, EVENTS_FILE)) - return meta + const eventsPath = join(worktreeDir, EVENTS_FILE) + const events = (await fs.exists(eventsPath)) ? parseEventLog(await fs.read(eventsPath)) : [] + return { meta, events } + } catch { + return undefined + } +} + +/** + * Archive a worktree agent's history into the project's transient archive (#737), returning the + * meta it archived: the boot reconcile's rescue of a run a dead daemon left in a worktree, where + * nothing reads it. The daemon's own teardown records the run on the data branch instead, through + * the `logs` skill, from what {@link readWorktreeAgent} reads. + */ +export async function archiveWorktreeAgent(worktree: string, repo: string, fs: StoreFs = nodeStoreFs(), branch?: string): Promise { + const run = await readWorktreeAgent(worktree, fs, branch) + if (!run) return undefined + try { + await archiveAgent(fs, archiveDir(join(repo, THE_FRAMEWORK_DIR)), run.meta, join(worktree, THE_FRAMEWORK_DIR, EVENTS_FILE)) + return run.meta } catch { return undefined } } /** - * The archived log + meta paths of one agent, wherever it is filed, or `[]` when it is nowhere. - * Exported so a caller that deletes a session (the dashboard's Remove) does not have to know which - * user archived it — before #1179 the path was derivable from the id alone, and now it is not. + * The archived log + meta paths of one agent, wherever it is filed — its card and diary on the + * data branch's checkout, else its transient copy — or `[]` when it is nowhere. For a caller that + * needs the file itself: the tail of an ended run, the retry's read of its failure detail. */ export async function archivedAgentPaths(cwd: string, agentId: string, fs: StoreFs = nodeStoreFs()): Promise { if (!isSafeAgentId(agentId)) return [] - const archive = await findArchive(fs, cwd, agentId).catch(() => undefined) + const files = await runFiles(cwd, agentId, runDeps(fs)).catch(() => undefined) + if (files) return [files.card, files.diary] + const archive = await findTransientArchive(fs, cwd, agentId).catch(() => undefined) return archive ? [archive.meta, archive.events] : [] } @@ -850,36 +847,27 @@ function isDeadRunningAgent(meta: AgentMeta | undefined, isAlive: (pid: number) return meta?.status === 'running' && ownerLiveness(meta, isAlive) !== 'live' } -/** - * Every archived meta a project has, across all of {@link archiveDirs}, with the path it came from. - * De-duplicated by agent id, first directory winning: the crash rescue archives into the transient - * `agents/` and the close into the user's committed one, so an agent can sit in both places and the - * history must show it once. The user directories are searched first, so the committed copy wins. - */ -async function readAllArchivedMetaEntries(fs: StoreFs, cwd: string, since?: number): Promise> { - const seen = new Set() - const entries: Array<{ path: string; meta: AgentMeta }> = [] - for (const agentsDir of await archiveDirs(fs, cwd)) { - for (const entry of await readArchivedMetaEntries(fs, agentsDir, since).catch(() => [])) { - if (seen.has(entry.meta.id)) continue - seen.add(entry.meta.id) - entries.push(entry) - } - } - return entries +/** The transient archive's metas with their paths, `[]` when there is none; `since` as in {@link listAgents}. */ +async function transientArchivedMetaEntries(fs: StoreFs, cwd: string, since?: number): Promise> { + return readArchivedMetaEntries(fs, archiveDir(join(cwd, THE_FRAMEWORK_DIR)), since).catch(() => []) } /** - * List a project's archived agents, most-recent first: every user's committed archive plus the - * transient `agents/`. The id sorts chronologically so no timestamp parse is needed. Missing or - * unreadable dir/entries are skipped, never thrown. + * List a project's archived agents, most-recent first: the runs on the data branch (every + * person's, read through the `logs` skill and unfolded into the framework's meta) plus the + * transient archive, de-duplicated by id with the branch's copy winning — the crash rescue + * archives transiently and the teardown records on the branch, so a run can sit in both and the + * history must show it once. The id sorts chronologically so no timestamp parse is needed. + * Missing or unreadable dir/entries are skipped, never thrown. * * `since` (epoch ms) is for a caller that only wants recent runs — a poll on a cadence, not the - * history list. It is answered from the filenames, so the records it excludes cost no read at all. + * history list. */ export async function listAgents(cwd: string, fs: StoreFs = nodeStoreFs(), since?: number): Promise { - const entries = await readAllArchivedMetaEntries(fs, cwd, since) - return entries.map(entry => entry.meta).sort(byIdDesc) + const committed = (await listRuns(cwd, since === undefined ? {} : { since }, runDeps(fs)).catch((): never[] => [])).map(fromRunCard) + const seen = new Set(committed.map(meta => meta.id)) + const transient = (await transientArchivedMetaEntries(fs, cwd, since)).map(entry => entry.meta).filter(meta => !seen.has(meta.id)) + return [...committed, ...transient].sort(byIdDesc) } /** @@ -897,11 +885,16 @@ function ownerLiveness(meta: AgentMeta, isAlive: (pid: number) => boolean): 'liv } /** - * Reconcile runs a dead process left marked `running` — the live `agent.json`, an archived - * `agents/*.json`, or an agent inside a worktree. Such an agent shows as active while nothing is left - * to read its `control.jsonl`, so its Stop is a no-op. Each is flipped to `stopped`; the live - * run is archived first (idempotent) so its history is kept. Returns how many were reconciled. - * Best-effort: a read/write error skips that agent, never throws. + * Reconcile runs a dead process left marked `running` — a run on the data branch, the live + * `agent.json`, a transient `agents/*.json`, or an agent inside a worktree. Such an agent shows as + * active while nothing is left to read its `control.jsonl`, so its Stop is a no-op. Each is + * flipped to `stopped`; the live run is archived first (idempotent) so its history is kept. + * Returns how many were reconciled. Best-effort: a read/write error skips that agent, never throws. + * + * A run on the data branch is ended through the `logs` skill's write, so the ending is a commit: + * a card edited in place on the branch's checkout is not a fact yet — the next sync's rebase + * refuses the dirty tree and the funnel resets it. `funnel` is the skill's own unless a test + * hands one in. * * An agent whose pid is alive on this host is left alone (#926). This used to flip every `running` * meta on the assumption that a fresh dashboard drives no in-flight run, which holds only while @@ -913,17 +906,26 @@ export async function reconcileOrphanedAgents( cwd: string, fs: StoreFs = nodeStoreFs(), isAlive: (pid: number) => boolean = isPidAlive, + funnel?: LogsFunnel, ): Promise { const dir = join(cwd, THE_FRAMEWORK_DIR) let fixed = 0 - // Archived agents stuck at `running` (e.g. a prior live agent the next agent never rescued), wherever - // they are archived. Done before the live agent so its fresh archive isn't re-counted here. - for (const { path, meta } of await readAllArchivedMetaEntries(fs, cwd)) { + const deps = runDeps(fs, funnel) + for (const card of await listRuns(cwd, {}, deps).catch((): never[] => [])) { + const meta = fromRunCard(card) + if (!isDeadRunningAgent(meta, isAlive)) continue + // The surrogate end (#1359) lands in both files: the replayed diary sees the run finish, and + // the meta fold drops the pendingChoice the agent died holding. + const event = orphanEndEvent() + const diary = (await readDiary(cwd, card.id, deps).catch(() => undefined)) ?? [] + const ended = await writeRun(cwd, toRunCard(applyEventToMeta(meta, event, new Date().toISOString())), [...diary, toDiaryLine(event)], deps) + if (ended.ok || ended.committed) fixed++ + } + // The transient archive's runs stuck at `running`. Done before the live agent so its fresh + // archive isn't re-counted here. + for (const { path, meta } of await transientArchivedMetaEntries(fs, cwd)) { if (!isDeadRunningAgent(meta, isAlive)) continue try { - // The archived pair sits side by side (`.json` + `.jsonl`), so the surrogate end - // (#1359) lands in both: the replayed log sees the agent finish, and the meta fold drops - // the pendingChoice the agent died holding. const event = orphanEndEvent() await fs.append(path.replace(/\.json$/, '.jsonl'), JSON.stringify(event) + '\n').catch(() => {}) await writeMetaFile(fs, path, applyEventToMeta(meta, event, new Date().toISOString())) @@ -943,7 +945,7 @@ export async function reconcileOrphanedAgents( // Runs living in worktrees (#736/#737). A daemon that died mid-run never ran its teardown, so // each of those agents is orphaned the same way — except its history sits inside the worktree, // where nothing reads it. Flip it in place (so the dashboard stops showing it as live) and copy - // it into the repo's history. The worktree itself is left on disk: an agent that ended this way did + // it into the transient archive. The worktree itself is left on disk: an agent that ended this way did // not end cleanly, and those are kept for inspection. Removing one is an explicit action. for (const entry of await worktreeDirEntries(cwd, path => fs.subdirs(path))) { const worktreeDir = join(entry.path, THE_FRAMEWORK_DIR) @@ -1039,9 +1041,10 @@ export async function readLiveMetas( } /** - * Read one archived agent's event log for replay. Returns `undefined` for an - * unknown or unsafe id; a torn trailing line is dropped (same rule as the live - * {@link AgentStore.loadEvents}). + * Read one archived agent's event log for replay: the run's diary on the data branch, read + * through the `logs` skill and turned back into the framework's events, else the transient + * archive's log. Returns `undefined` for an unknown or unsafe id; a torn trailing line is dropped + * (same rule as the live {@link AgentStore.loadEvents}). */ export async function loadAgentEvents( cwd: string, @@ -1049,7 +1052,9 @@ export async function loadAgentEvents( fs: StoreFs = nodeStoreFs(), ): Promise { if (!isSafeAgentId(id)) return undefined - const archive = await findArchive(fs, cwd, id) + const diary = await readDiary(cwd, id, runDeps(fs)).catch(() => undefined) + if (diary) return eventsOf(diary) + const archive = await findTransientArchive(fs, cwd, id) if (!archive || !(await fs.exists(archive.events))) return undefined return parseEventLog(await fs.read(archive.events)) } @@ -1106,35 +1111,3 @@ export async function readEventLog(cwd: string, fs: StoreFs = nodeStoreFs()): Pr return [] } } - -/** The facts a settled run learns after its process is gone: the PR its work is on, the branch it landed on. */ -export type ArchivePatch = Partial> - -/** - * Patch an archived run's record with a fact discovered once the agent's process is gone (E6, - * #1601): the pull request opened for its work, or the branch a cloud session's work landed - * on. There is no event stream left to carry it, and every surface reads the record, so this - * one write is what turns a "nothing committed" row into its real branch and PR. - * - * The archive lives on the data branch's checkout, and a fact written there is only durable once - * committed — {@link patchArchivedAgentOnDataBranch} is the funneled form every caller outside a - * test uses. - */ -export async function patchArchivedAgent( - cwd: string, - agentId: string, - patch: ArchivePatch, - fs: StoreFs = nodeStoreFs(), -): Promise { - if (!isSafeAgentId(agentId)) return false - try { - const archive = await findArchive(fs, cwd, agentId) - if (!archive) return false - const meta = await readMetaFile(fs, archive.meta) - if (!meta) return false - await writeMetaFile(fs, archive.meta, { ...meta, ...patch }) - return true - } catch { - return false - } -} diff --git a/packages/framework/src/store/index.ts b/packages/framework/src/store/index.ts index 010117507..561769dfd 100644 --- a/packages/framework/src/store/index.ts +++ b/packages/framework/src/store/index.ts @@ -12,9 +12,8 @@ export { loadAgentEvents, readLiveMetas as readLiveMetas, archiveWorktreeAgent, + readWorktreeAgent, archivedAgentPaths, - patchArchivedAgent, - type ArchivePatch, restoreArchivedAgent, agentIdFromStartedAt, startedAtFromAgentId as startedAtFromAgentId, @@ -28,3 +27,4 @@ export { type OpenStoreOptions, } from './agent-store.js' export { resolveAgentCheckout, resolveAgentEventsPath } from './agent-checkout.js' +export { toRunCard, fromRunCard, toDiaryLine, fromDiaryLine, diaryOf, eventsOf } from './run-record.js' diff --git a/packages/framework/src/store/run-record.SPEC.md b/packages/framework/src/store/run-record.SPEC.md new file mode 100644 index 000000000..1b33c7898 --- /dev/null +++ b/packages/framework/src/store/run-record.SPEC.md @@ -0,0 +1,14 @@ +The framework's run in the `logs` skill's two shapes, both ways: the agent meta as the run's card and back, the event log as the run's diary and back. + +## Context + +The `logs` skill owns what a run says to an agent: a card of eleven plain fields and four kinds of diary line. The framework records far more about a run than that, and none of it is the skill's to know. So the framework keeps its own fields inside the skill's files, in the places the skill stores and never reads, and every reader of a recorded run comes back through this mapping — the run page's replay, the tail of an ended run, a continuation's restore, the boot-time healing. + +## Business logic — TL;DR + +- **The card** - the agent meta's id, start and end time, status, intent, driver, model, branch, pull request, ticket and cost are the card's own fields; every other field of the meta sits under the one key the skill keeps for its caller. A card unfolds back into the meta with the skill's fields winning; a card with no caller at all gets a last-updated time from its end or its start. +- **The diary** - what the agent said, its result, the run's cost and its ending are written as the skill's four kinds of line, their other fields carried along; every other event is written as it is, under its own kind. A diary reads back as the same events, a line of any other kind passing through untouched. + +## Before modifying/creating SPEC.md files + +You must always read and respect https://raw.githubusercontent.com/brillout/sdd/refs/heads/main/sdd.md diff --git a/packages/framework/src/store/run-record.test.SPEC.md b/packages/framework/src/store/run-record.test.SPEC.md new file mode 100644 index 000000000..5333a89de --- /dev/null +++ b/packages/framework/src/store/run-record.test.SPEC.md @@ -0,0 +1,8 @@ +What the tests cover: the framework's run in the `logs` skill's two shapes, both ways. + +- **The card** - the framework's agent meta becomes a card with the skill's eleven fields on top and every other field under `caller`; a meta with none of those gets no `caller` key at all; the card unfolds back into the same meta, and a card without a last-updated time gets one from its end or its start. +- **The diary** - what the agent said, its result, the run's cost and its ending become the skill's four kinds of line, their extra fields carried along; every other event is written as it is; the diary reads back as the same events, an ending's `done`, `stopped` and `failed` included. + +## Before modifying/creating SPEC.md files + +You must always read and respect https://raw.githubusercontent.com/brillout/sdd/refs/heads/main/sdd.md diff --git a/packages/framework/src/store/run-record.test.ts b/packages/framework/src/store/run-record.test.ts new file mode 100644 index 000000000..814cd8102 --- /dev/null +++ b/packages/framework/src/store/run-record.test.ts @@ -0,0 +1,62 @@ +import { strict as assert } from 'node:assert' +import { test } from 'node:test' +import type { FrameworkEvent } from '../events.js' +import type { AgentMeta } from './agent-store.js' +import { diaryOf, eventsOf, fromRunCard, toRunCard } from './run-record.js' + +test('the meta round-trips through the skill\'s card: eleven fields on top, the rest under caller (#1769)', () => { + const meta: AgentMeta = { + status: 'done', + id: 'r1', + startedAt: '2026-09-08T18:14:30.651Z', + updatedAt: '2026-09-08T18:15:40.433Z', + endedAt: '2026-09-08T18:15:40.433Z', + pid: 923, + host: 'laptop', + intent: 'fix it', + kind: 'prompt', + handoff: { push: true, pr: true, merge: true }, + branch: 'agent-r1', + driver: 'claude-code', + workspace: '/w', + model: 'opus', + readyForMerge: true, + mergeOutcome: 'auto-armed', + pr: { number: 1765, url: 'https://x/pull/1765' }, + ticket: 'tickets/2026-09-01_a.md', + cost: 0.62, + } + const card = toRunCard(meta) + assert.deepEqual(Object.keys(card), ['id', 'startedAt', 'status', 'endedAt', 'intent', 'driver', 'model', 'branch', 'pr', 'ticket', 'cost', 'caller']) + assert.deepEqual(card.caller, { updatedAt: meta.updatedAt, pid: 923, host: 'laptop', kind: 'prompt', handoff: meta.handoff, workspace: '/w', readyForMerge: true, mergeOutcome: 'auto-armed' }) + assert.deepEqual(fromRunCard(card), meta) + // The framework's last-updated time is its own, so even a bare meta carries a caller; a card + // with no caller at all (another writer's) unfolds with an updatedAt from its end or its start. + const bare = toRunCard({ status: 'running', id: 'r2', startedAt: 't', updatedAt: 't' }) + assert.deepEqual(bare, { id: 'r2', startedAt: 't', status: 'running', caller: { updatedAt: 't' } }) + assert.deepEqual(fromRunCard({ id: 'r2', startedAt: 't', status: 'done', endedAt: 'e' }), { id: 'r2', startedAt: 't', status: 'done', endedAt: 'e', updatedAt: 'e' }) +}) + +test('the events round-trip through the diary: four kinds mapped, every other line as it is (#1769)', () => { + const events: FrameworkEvent[] = [ + { kind: 'session', driver: 'claude-code', workspace: '/w', fake: false, model: 'opus' }, + { kind: 'driver', event: { type: 'start', prompt: 'go' } }, + { kind: 'driver', event: { type: 'text', text: 'Reading.' } }, + { kind: 'driver', event: { type: 'action', label: 'Bash' } }, + { kind: 'driver', event: { type: 'result', text: 'Done.', sessionId: 's1', usage: { inputTokens: 1, outputTokens: 2, cacheReadTokens: 0, cacheCreationTokens: 0 } } }, + { kind: 'usage', costUsd: 0.5, inputTokens: 1, outputTokens: 2, cacheReadTokens: 0, cacheCreationTokens: 0, turns: 1 }, + { kind: 'usage', inputTokens: 1, outputTokens: 2, cacheReadTokens: 0, cacheCreationTokens: 0, turns: 1 }, + { kind: 'end', ok: true }, + { kind: 'end', ok: false, stopped: true, detail: 'its process died' }, + { kind: 'end', ok: false, detail: 'API 500' }, + ] + const diary = diaryOf(events) + assert.deepEqual(diary.map(line => line.kind), ['session', 'driver', 'said', 'driver', 'result', 'cost', 'cost', 'ended', 'ended', 'ended']) + assert.deepEqual(diary[2], { kind: 'said', text: 'Reading.' }) + assert.deepEqual(diary[4], { kind: 'result', text: 'Done.', sessionId: 's1', usage: { inputTokens: 1, outputTokens: 2, cacheReadTokens: 0, cacheCreationTokens: 0 } }) + assert.deepEqual(diary[5], { kind: 'cost', usd: 0.5, inputTokens: 1, outputTokens: 2, cacheReadTokens: 0, cacheCreationTokens: 0, turns: 1 }) + assert.deepEqual(diary[7], { kind: 'ended', status: 'done' }) + assert.deepEqual(diary[8], { kind: 'ended', status: 'stopped', detail: 'its process died' }) + assert.deepEqual(diary[9], { kind: 'ended', status: 'failed', detail: 'API 500' }) + assert.deepEqual(eventsOf(diary), events) +}) diff --git a/packages/framework/src/store/run-record.ts b/packages/framework/src/store/run-record.ts new file mode 100644 index 000000000..4bb116cfd --- /dev/null +++ b/packages/framework/src/store/run-record.ts @@ -0,0 +1,100 @@ +import type { DriverEvent } from 'agent-driver' +import type { AnyDiaryLine, RunCard } from '@gemstack/skill-logs' +import type { FrameworkEvent } from '../events.js' +import type { AgentMeta } from './agent-store.js' + +/** + * The framework's run, in the `logs` skill's two shapes (#1769). The skill owns what a run's card + * says to an agent — eleven plain fields — and four kinds of diary line; everything else the + * framework records is its own. So the card carries the framework's remaining meta under the one + * key the skill stores and never reads, `caller`, and the diary carries the framework's other + * events as they are, among the four the skill knows. This module is the whole mapping, both + * ways: the daemon writes through it at teardown, and every reader of an archived run — the run + * page's replay, the tail of an ended run, a continuation's restore — comes back through it. + */ + +/** The framework's meta as the skill's card: the skill's fields on top, the rest under `caller`. */ +export function toRunCard(meta: AgentMeta): RunCard { + const { id, startedAt, endedAt, status, intent, driver, model, branch, pr, ticket, cost, ...caller } = meta + const card: RunCard = { id, startedAt, status } + if (endedAt !== undefined) card.endedAt = endedAt + if (intent !== undefined) card.intent = intent + if (driver !== undefined) card.driver = driver + if (model !== undefined) card.model = model + if (branch !== undefined) card.branch = branch + if (pr !== undefined) card.pr = pr + if (ticket !== undefined) card.ticket = ticket + if (cost !== undefined) card.cost = cost + if (Object.keys(caller).length > 0) card.caller = caller + return card +} + +/** The skill's card as the framework's meta: `caller` unfolded, the skill's fields winning. */ +export function fromRunCard(card: RunCard): AgentMeta { + const { caller, ...own } = card + return { updatedAt: card.endedAt ?? card.startedAt, ...(caller as Partial), ...own } as AgentMeta +} + +/** + * One framework event as one diary line: what the agent said, its result, the run's end and its + * cost become the skill's four kinds; every other event is written as it is, under its own kind. + */ +export function toDiaryLine(event: FrameworkEvent): AnyDiaryLine { + switch (event.kind) { + case 'driver': { + if (event.event.type === 'text') return { kind: 'said', text: event.event.text } + if (event.event.type === 'result') { + const { type: _type, ...rest } = event.event + return { kind: 'result', ...rest } + } + return event as unknown as AnyDiaryLine + } + case 'end': + return { + kind: 'ended', + status: event.ok ? 'done' : event.stopped ? 'stopped' : 'failed', + ...(event.detail !== undefined ? { detail: event.detail } : {}), + } + case 'usage': { + const { kind: _kind, costUsd, ...rest } = event + return { kind: 'cost', ...(costUsd !== undefined ? { usd: costUsd } : {}), ...rest } + } + default: + return event as unknown as AnyDiaryLine + } +} + +/** The inverse of {@link toDiaryLine}: a line of any other kind is a framework event as written. */ +export function fromDiaryLine(line: AnyDiaryLine): FrameworkEvent { + switch (line.kind) { + case 'said': + return { kind: 'driver', event: { type: 'text', text: String(line['text']) } } + case 'result': { + const { kind: _kind, ...rest } = line + return { kind: 'driver', event: { type: 'result', ...rest } as DriverEvent } + } + case 'ended': + return { + kind: 'end', + ok: line['status'] === 'done', + ...(line['status'] === 'stopped' ? { stopped: true } : {}), + ...(typeof line['detail'] === 'string' ? { detail: line['detail'] } : {}), + } + case 'cost': { + const { kind: _kind, usd, ...rest } = line + return { kind: 'usage', ...(usd !== undefined ? { costUsd: usd } : {}), ...rest } as FrameworkEvent + } + default: + return line as unknown as FrameworkEvent + } +} + +/** A run's events as its diary. */ +export function diaryOf(events: readonly FrameworkEvent[]): AnyDiaryLine[] { + return events.map(toDiaryLine) +} + +/** A diary as the run's events, for a reader that replays them. */ +export function eventsOf(lines: readonly AnyDiaryLine[]): FrameworkEvent[] { + return lines.map(fromDiaryLine) +} diff --git a/packages/framework/src/system-prompt.SPEC.md b/packages/framework/src/system-prompt.SPEC.md index 34a9e87ff..10a8ab4c3 100644 --- a/packages/framework/src/system-prompt.SPEC.md +++ b/packages/framework/src/system-prompt.SPEC.md @@ -47,7 +47,7 @@ The context docs are the files the agent keeps in context, rendered as commented Two of the docs are not the agent's to edit by hand — `tickets/**.md` and `TODO_AGENTS.md` — because they live on the `agent-data` branch rather than in the agent's checkout. Their bullets say so and send the agent to the `tickets` skill and the `queue` skill, which is where their format, and every way of reading and changing them, is written down. No format text of their own rides in the channel. The user's picked in-context directories, when any, come first on the `Context:` line, framing whatever follows. -In a checkout The Framework created, those skills are the checkout's: the packages link them where the agent's harness looks for skills, and the `tickets` and `queue` commands are on the agent's PATH. Anywhere else — a terminal run in the user's own checkout, a GitHub Actions runner, a cloud session — neither is true, so the channel carries a temporary bridge instead, right after the built-in prompt and after the branch section that is its counterpart: how to read and write the branch, and claim a ticket, with git alone, followed by the two skills' own text so the formats exist in exactly one place. Temporary is the word the code uses for it: it exists only until skills are committed into the repository itself, and then it goes. +In a checkout The Framework created, those skills are the checkout's: the packages link them where the agent's harness looks for skills, and the `tickets`, `queue` and `logs` commands are on the agent's PATH. Anywhere else — a terminal run in the user's own checkout, a GitHub Actions runner, a cloud session — neither is true, so the channel carries a temporary bridge instead, right after the built-in prompt and after the branch section that is its counterpart: how to read and write the branch, and claim a ticket, with git alone, followed by the three skills' own text so the formats exist in exactly one place. Temporary is the word the code uses for it: it exists only until skills are committed into the repository itself, and then it goes. #### Rationale diff --git a/packages/framework/src/system-prompt.test.SPEC.md b/packages/framework/src/system-prompt.test.SPEC.md index 8e66242d0..28e8e8cd1 100644 --- a/packages/framework/src/system-prompt.test.SPEC.md +++ b/packages/framework/src/system-prompt.test.SPEC.md @@ -4,7 +4,7 @@ What the tests cover: the composition of an agent's system channel and the readi - `SYSTEM.md` is read and trimmed; absent or whitespace-only reads as no user prompt. - The built-in prompt template carries its section headings verbatim, names `TODO_AGENTS.md`, defers the branch to the "Branch management" section rather than creating one itself (the workspace rules are the skill's, not the template's), no longer asks for the retired analysis artifact or carries pre-rewrite headings, and has exactly one variable slot (the user's prompt). - Rendering splits the system and user halves, fully renders the system half, and is not confused by a user prompt that itself contains the boundary heading. -- The ticket and queue formats ride in the channel only for an agent outside a checkout The Framework created: the section that has it read and write the `agent-data` branch with git, followed by the `tickets` skill's own text and then the `queue` skill's, each with its front matter dropped, all after the branch section that is their counterpart. An agent in its own checkout gets neither, and no inline format section of the channel's own is left. Vanilla drops them along with the docs and the built-in prompt, keeps the user's own dirs and prompt, and an empty vanilla block is truly empty. +- The ticket and queue formats ride in the channel only for an agent outside a checkout The Framework created: the section that has it read and write the `agent-data` branch with git, followed by the `tickets` skill's own text, then the `queue` skill's, then the `logs` skill's, each with its front matter dropped, all after the branch section that is their counterpart. An agent in its own checkout gets neither, and no inline format section of the channel's own is left. Vanilla drops them along with the docs and the built-in prompt, keeps the user's own dirs and prompt, and an empty vanilla block is truly empty. - Composition order and exactness: context (user dirs first, docs after), built-in prompt, the branch and tickets sections, user prompt, then the await and signal protocols — and nothing else, whatever the options; the signal protocol is always last. - An agent in a checkout The Framework created is sent to the `branches` skill by the prompt's own session-name step, with none of the skill's text in the channel; every other agent, hands-off included, gets the "Branch management" section that has it branch with git instead; vanilla drops both along with the built-in prompt. - The browser protocol appears only when the agent has a browser; the hands-off protocol (land everything) appears only for a hands-off agent, whose await protocol is the same one every agent gets. A local agent gets neither. Both survive vanilla and none survives transparent. diff --git a/packages/framework/src/system-prompt.test.ts b/packages/framework/src/system-prompt.test.ts index d8cfa1092..8c2459f1f 100644 --- a/packages/framework/src/system-prompt.test.ts +++ b/packages/framework/src/system-prompt.test.ts @@ -11,7 +11,7 @@ import { systemPromptBlock, SYSTEM_PROMPT_TEMPLATE, } from './system-prompt.js' -import { BRANCH_YOURSELF, QUEUE_SKILL, TICKETS_SKILL, TICKETS_YOURSELF } from './prompts.generated.js' +import { BRANCH_YOURSELF, LOGS_SKILL, QUEUE_SKILL, TICKETS_SKILL, TICKETS_YOURSELF } from './prompts.generated.js' import { loadUserSystemPrompt, SYSTEM_PROMPT_FILE } from './system-prompt-file.js' import { THE_FRAMEWORK_DIR } from './framework-dir.js' @@ -22,7 +22,7 @@ const KNOWLEDGE_CONTEXT = `Context:\n${KNOWLEDGE_LINES}` /** The context block: the bullets alone — the formats are the `tickets` skill's, not the channel's (#1748). */ const CONTEXT_BLOCK = KNOWLEDGE_CONTEXT /** What an agent outside a daemon-made checkout gets after the prompt: branch, then tickets and queue, with git (temporary, #1748). */ -const ELSEWHERE = [BRANCH_YOURSELF, `${TICKETS_YOURSELF}\n\n${TICKETS_SKILL}\n\n${QUEUE_SKILL}`] +const ELSEWHERE = [BRANCH_YOURSELF, `${TICKETS_YOURSELF}\n\n${TICKETS_SKILL}\n\n${QUEUE_SKILL}\n\n${LOGS_SKILL}`] test('CONTEXT_DOCS is the repo-context fragment (#683): business knowledge plus the roadmap/queue pointers', () => { const paths = CONTEXT_DOCS.map(d => d.path) @@ -149,10 +149,12 @@ test('the ticket and queue formats are the `tickets` and `queue` skills\': in th assert.ok(TICKETS_SKILL.includes('tickets/_.md') && !TICKETS_SKILL.startsWith('---'), 'the tickets skill text, front matter dropped') assert.ok(QUEUE_SKILL.includes('## Priority 9') && !QUEUE_SKILL.startsWith('---'), 'the queue skill text, front matter dropped') assert.ok(elsewhere.indexOf(QUEUE_SKILL) > elsewhere.indexOf(TICKETS_SKILL), 'the queue skill after the tickets skill') + assert.ok(LOGS_SKILL.includes('npx logs --ticket ') && !LOGS_SKILL.startsWith('---'), 'the logs skill text, front matter dropped') + assert.ok(elsewhere.indexOf(LOGS_SKILL) > elsewhere.indexOf(QUEUE_SKILL), 'the logs skill after the queue skill') assert.ok(elsewhere.indexOf(TICKETS_YOURSELF) > elsewhere.indexOf(BRANCH_YOURSELF), 'after the branch counterpart') // Framework-authored content, so `--vanilla` drops it with the docs and the built-in prompt. const vanilla = systemPromptBlock({ vanilla: true, user: 'Only mine.' }) - assert.ok(!vanilla.includes(TICKETS_SKILL) && !vanilla.includes(QUEUE_SKILL) && !vanilla.includes(TICKETS_YOURSELF)) + assert.ok(!vanilla.includes(TICKETS_SKILL) && !vanilla.includes(QUEUE_SKILL) && !vanilla.includes(LOGS_SKILL) && !vanilla.includes(TICKETS_YOURSELF)) }) test('systemPromptBlock defaults to the knowledge-doc context line + the built-in #326 prompt', () => { diff --git a/packages/framework/src/system-prompt.ts b/packages/framework/src/system-prompt.ts index 02cfa7512..a06a680fe 100644 --- a/packages/framework/src/system-prompt.ts +++ b/packages/framework/src/system-prompt.ts @@ -1,5 +1,5 @@ import { renderTemplate } from './prompt-template.js' -import { BRANCH_YOURSELF, QUEUE_SKILL, SYSTEM_PROMPT, TICKETS_SKILL, TICKETS_YOURSELF } from './prompts.generated.js' +import { BRANCH_YOURSELF, LOGS_SKILL, QUEUE_SKILL, SYSTEM_PROMPT, TICKETS_SKILL, TICKETS_YOURSELF } from './prompts.generated.js' import { AWAIT_PROTOCOL, BROWSER_PROTOCOL, HANDS_OFF_PROTOCOL, SIGNAL_PROTOCOL } from './turn-gate.js' // No Node imports here, deliberately. This module composes the prompt and the @@ -72,12 +72,13 @@ export const BUSINESS_KNOWLEDGE_DOCS: readonly ContextDoc[] = [DECISIONS_DOC, FA /** * TEMPORARY (#1748): what an agent outside a checkout the daemon created is told about the - * tickets and the queue, since nothing links the `tickets` and `queue` skills into its checkout - * and their commands are not on its PATH — the counterpart of {@link BRANCH_YOURSELF}: how to read - * and write the branch with git, followed by the two skills' own formats so they exist in one - * place. Dies when use-npm-skills commits the skills into the repository. + * tickets, the queue and the logs, since nothing links the `tickets`, `queue` and `logs` skills + * into its checkout and their commands are not on its PATH — the counterpart of + * {@link BRANCH_YOURSELF}: how to read and write the branch with git, followed by the three + * skills' own texts so they exist in one place. Dies when use-npm-skills commits the skills into + * the repository. */ -const TICKETS_BRIDGE = `${TICKETS_YOURSELF}\n\n${TICKETS_SKILL}\n\n${QUEUE_SKILL}` +const TICKETS_BRIDGE = `${TICKETS_YOURSELF}\n\n${TICKETS_SKILL}\n\n${QUEUE_SKILL}\n\n${LOGS_SKILL}` /** * Everything the agent keeps in context when it starts (#683), which diff --git a/packages/framework/src/worktrees.SPEC.md b/packages/framework/src/worktrees.SPEC.md index 8fcfca8a5..fb8736a3b 100644 --- a/packages/framework/src/worktrees.SPEC.md +++ b/packages/framework/src/worktrees.SPEC.md @@ -94,7 +94,7 @@ The user wants an agent gone from the dashboard for good — not just its checko #### Business logic -Deleting an agent is the sibling of reclaiming its checkout, and the difference is the point. Reclaiming takes the checkout back and keeps the agent — its row and its replayable log — because the history was already archived. Deleting removes that archive as well: the agent meta that puts the row in the list and the event log that replays it, wherever they are filed. Records are found by looking them up rather than by deriving a path from the agent id, because an agent is archived under whichever user ran it. A record on the logs branch is deleted inside the logs branch's write cycle, so the deletion is itself a committed and pushed change; any transient copy is simply unlinked. A file that is already gone is not an error, so a half-deleted agent still finishes cleanly. +Deleting an agent is the sibling of reclaiming its checkout, and the difference is the point. Reclaiming takes the checkout back and keeps the agent — its row and its replayable log — because the history was already archived. Deleting removes that archive as well: the agent meta that puts the row in the list and the event log that replays it, wherever they are filed. A run on the data branch is deleted by the `logs` skill, card and diary, inside the branch's write cycle, so the deletion is itself a committed and pushed change; a transient copy is simply unlinked, looked up rather than derived from the agent id. A file that is already gone is not an error, so a half-deleted agent still finishes cleanly. The checkout goes first, force-removed — any uncommitted work in it is discarded along with the agent, which is the intent here, unlike reclaiming, which keeps a checkout that holds any. diff --git a/packages/framework/src/worktrees.ts b/packages/framework/src/worktrees.ts index da5e8a16d..6ca587594 100644 --- a/packages/framework/src/worktrees.ts +++ b/packages/framework/src/worktrees.ts @@ -1,10 +1,10 @@ import { readFile } from 'node:fs/promises' -import { join, sep } from 'node:path' +import { join } from 'node:path' import { errorMessage } from './error-message.js' import { listAgents, readLiveMetas, archivedAgentPaths, META_FILE, type AgentMeta, type AgentStatus } from './store/index.js' -import { fileBranchPath, withFileBranch } from '@gemstack/agent-data' +import { deleteRun, runFiles } from '@gemstack/skill-logs' import { agentBranchName, listWorktreeDirs, isSafeAgentId, reclaimWorktree, removeWorktree, pruneWorktrees, worktreePath, worktreeSize, type ReclaimOutcome } from '@gemstack/skill-branches' -import { LOGS_BRANCH, THE_FRAMEWORK_DIR } from './framework-dir.js' +import { THE_FRAMEWORK_DIR } from './framework-dir.js' /** A retained worktree and the agent that left it behind (#752). */ export interface WorktreeRow { @@ -247,19 +247,15 @@ export async function deleteProjectAgent(cwd: string, agentId: string, opts: Del await removeWorktree(cwd, worktreePath(cwd, agentId)) await pruneWorktrees(cwd) } - // Then the records that put the row in the list. Looked up rather than derived from the id: a - // session is archived under whichever user ran it (#1179), so the id alone no longer names its - // path. Tolerant of an absent file, so a half-deleted session (its worktree already gone) - // still finishes cleanly. A record on the logs branch is removed inside its write funnel - // (#1582) — the deletion is a committed, pushed change — while a transient copy is an unlink. - const paths = await archivedAgentPaths(cwd, agentId) - const dataRoot = fileBranchPath(cwd, LOGS_BRANCH) + sep - for (const path of paths.filter(p => !p.startsWith(dataRoot))) await removeFile(path) - if (paths.some(p => p.startsWith(dataRoot))) { - const removed = await withFileBranch(cwd, LOGS_BRANCH, `[The Framework] delete session ${agentId}`, async () => { - for (const path of paths.filter(p => p.startsWith(dataRoot))) await removeFile(path) - }) + // Then the records that put the row in the list. The run on the data branch is deleted by the + // `logs` skill as one committed, pushed change (#1582/#1769); a transient copy is an unlink. + // Tolerant of an absent file, so a half-deleted session (its worktree already gone) still + // finishes cleanly. + if (await runFiles(cwd, agentId)) { + const removed = await deleteRun(cwd, agentId) if (!removed.ok && !removed.committed) return { ok: false, error: removed.error } + } else { + for (const path of await archivedAgentPaths(cwd, agentId)) await removeFile(path) } return { ok: true } } catch (err) { diff --git a/packages/skill-logs/DECISIONS.md b/packages/skill-logs/DECISIONS.md new file mode 100644 index 000000000..cb0611cb2 --- /dev/null +++ b/packages/skill-logs/DECISIONS.md @@ -0,0 +1,71 @@ +Non-obvious decisions only, grouped by business-logic flow. Anything not listed is left +to the implementer's judgment. Flag conflicts instead of silently deviating. Keep +outdated decisions (no history). + +A bullet is a person's pick, and says what it was picked over. What the code does belongs +in SPEC.md; a choice made while implementing is the implementer's judgment, not a +decision. An AI proposes a bullet and asks; it never adds or rewrites one. + +## The logs +- Two callers: the command an agent runs, which only reads, and a long-lived program that + keeps the branch checked out, records every run it drives through this package's + functions when the run is over, and shows the runs in pages of its own. The executable + is `logs`. The package ships `SKILL.md`, the agent's instructions. +- A run is two files on `agent-data`, the branch `@gemstack/agent-data` names, never on a + code branch: `agents//.json`, the card, and `agents//.jsonl`, the + diary. Both are pushed. Keeping the diary on the machine that ran it was considered and + dropped: a run's page on another machine, a continued run on another machine and a + wiped laptop all read the diary from the branch, and a few megabytes a year is the + price. +- `` is the git email the writer's repository commits as, lowercased, unsafe + characters replaced, `anonymous` when unusable. The folder groups the runs by person and + protects nothing: ids never collide, and the same email authors the commits. A flat + `agents/.json` with the person inside the card was the alternative and was not + taken: the grouping stays visible in a file listing. +- The card is small and the package's: `id`, `startedAt`, `endedAt`, `status`, `intent`, + `driver`, `model`, `branch`, `pr`, `ticket`, `cost`. Everything else a writer records + sits under one key of its own, `caller`, which the package stores and never reads. + Owning the writer's whole record, renamed field by field, was rejected: a program's + private fields would become a standalone package's public API. Under semver these + fields are the branch's public API: a change to them is a major. +- The diary is JSON lines. The package knows four kinds — `said`, `result`, `ended`, + `cost` — and skips every other line a writer adds. A writer maps its own events onto + the four; the package never learns them. A line may carry more fields than its kind + needs; they pass through. +- Agents only read. A note an agent leaves on its own run was considered and deferred: the + record lands on the branch only when the run is over, so a note would need a file of its + own or a record written at the start. +- The skill says when to look back: before planning or working a ticket, read its runs. + Leaving that to the caller's prompts was the alternative and was not taken: outside the + caller nobody would ever read the logs. + +## Flow: the command +- `logs` lists newest first and prints the package's fields only, never `caller`: the + writer's bookkeeping is not the agent's business. +- `logs` prints the newest 20 runs unless `--limit` says otherwise: a list of every run, + each with its prompt, is more than an agent should read for a look back. +- `--ticket ` matches a card whose ticket is that path or ends with `/`, so a + ticket's filename and the path a queue entry links to both find it. The package does + not know where tickets live. +- `show` prints the card with the four kinds of diary line, never the whole diary: an + agent cannot read a hundred kilobytes of a writer's bookkeeping, and the writer's own + pages replay the whole file themselves. +- A read fetches origin once and reads everything from that copy: only origin has every + writer's pushes. With no origin the local branch is read. Outside a repository a + command refuses `not-a-repo`; only git's own "not a git repository" reads as that. An + id no run has refuses `no-run`. +- Every command that runs prints one JSON document. A refusal also puts one line on + stderr and exits 1. A malformed command line (an unknown flag, the wrong argument + count, a `--limit` that is not a whole number above 0, an id that is not one) is + rejected first: the usage on stderr, nothing on stdout, exit 2. Anything a command + throws refuses with `git-failed`. + +## Flow: what the writer records +- A run is recorded once, when it is over, as one commit carrying both files. Two late + facts, the branch the work landed on and the pull request, are patched onto the card + afterwards, one commit each. A run is deleted as one commit, both files. A run a dead + writer left marked running is recorded again by the next writer that notices, ended, + where it already sits. Nothing else on a card changes after it lands. +- The writer's persistent checkout is `.branches/agent-data`, and its writes go through + that checkout's serialized cycle, the same as the other skills'. The command never + touches it. diff --git a/packages/skill-logs/SKILL.SPEC.md b/packages/skill-logs/SKILL.SPEC.md new file mode 100644 index 000000000..e27ef3f3b --- /dev/null +++ b/packages/skill-logs/SKILL.SPEC.md @@ -0,0 +1,17 @@ +The skill's instructions: what an agent is told about the project's agent logs — where they live, how to read them, when to look back, and the two formats. + +## User story + +- The user expects an agent starting on a ticket to know how the last attempt went — what that agent said before it stopped or failed, or that a done run already has a pull request — without the user putting any of that in a prompt. + +## Business logic — TL;DR + +- **The runs are on a branch, not in your checkout** - every run's card and diary live under `agents//` on the `agent-data` branch, never on a code branch; the agent's own checkout does not contain them. +- **The `logs` command is how to read them, and reading is all it does** - it ships with the `@gemstack/skill-logs` package the repository depends on: the agent installs the repository's dependencies once, when there is no `node_modules` yet, and runs `npx logs`, so every command the skill names runs as written on a fresh clone. The program that ran an agent records its run; a refusal exits 1 with a line on stderr, a wrong command line exits 2 with the usage. +- **Read: the bare command and `show`** - the runs newest first as cards, the newest 20 unless `--limit` says otherwise, narrowed by `--ticket` (a ticket's filename or the path a queue entry links to) or `--branch`; `show ` one run's card with what the agent said, its result, how it ended and what it cost. +- **Before you plan or work a ticket, read its runs** - a stopped or failed run says what to avoid, and `show` says what that agent said before it ended; a done run with a pull request means the work may already be there, so read the pull request before doing it again; a run with no ticket is found by branch or in the list. +- **The formats** - the card's fields, one example, with `status` one of four words, `cost` in US dollars, every field but three optional, and the writing program's own bookkeeping under `caller`, never printed; the diary's four kinds of line, one example each, any other kind the writing program's and left out of `show`. + +## Before modifying/creating SPEC.md files + +You must always read and respect https://raw.githubusercontent.com/brillout/sdd/refs/heads/main/sdd.md diff --git a/packages/skill-logs/SKILL.md b/packages/skill-logs/SKILL.md new file mode 100644 index 000000000..2dc94756b --- /dev/null +++ b/packages/skill-logs/SKILL.md @@ -0,0 +1,63 @@ +--- +name: logs +description: The record of every run agents made on this project — what was asked, on which ticket, where the work went, how it ended, what the agent said — where it lives, how to read it, and when to look back before you work. +--- + +# Agent logs + +Every run an agent made on this project leaves a record on the branch `agent-data`, never on a code branch; your checkout does not contain it. A run is two files under `agents//`: the card, `.json` — what was asked, the ticket, the branch, the pull request, how it ended, what it cost — and the diary, `.jsonl` — what the agent said along the way, its result. + +Read them with the `logs` command, a dependency of this repository (`@gemstack/skill-logs`). With no `node_modules`, install first with the lockfile's package manager (`npm install` for `package-lock.json`). Then run it as `npx logs`. The command only reads: the program that ran an agent records its run, at its end. A refusal exits 1 with a line on stderr; a wrong command line exits 2 with the usage. + +## Read + +``` +npx logs [--ticket ] [--branch ] [--limit N] + the runs, newest first, as one JSON array of cards — the newest 20 + unless --limit says otherwise; --ticket keeps the runs that worked + one ticket (its filename, or the path a queue entry links to); + --branch the runs on one branch +npx logs show one run: its card, plus `diary`, the lines of what the agent + said, its result, how it ended, and what it cost +``` + +## Before you plan or work a ticket, read its runs + +``` +npx logs --ticket +``` + +A ticket, or a queue entry that links one, may have been worked before. A `stopped` or `failed` run tells you what to avoid: `npx logs show ` for what that agent said before it ended. A `done` run with a `pr` means the work may already be there: read the pull request before doing it again. A run with no ticket is found by `--branch`, or in the list. + +## The card + +```json +{ + "id": "2026-09-08T18-14-30-111Z", + "startedAt": "2026-09-08T18:14:30.651Z", + "endedAt": "2026-09-08T18:15:40.433Z", + "status": "done", + "intent": "what the agent was asked to do", + "driver": "claude-code", + "model": "opus", + "branch": "agent-2026-09-08T18-14-30-111Z", + "pr": { "number": 1765, "url": "https://github.com/org/repo/pull/1765" }, + "ticket": "tickets/2026-09-01_some-ticket.md", + "cost": 0.62 +} +``` + +`status` is `running`, `done`, `stopped` or `failed`. `cost` is in US dollars. Every field but `id`, `startedAt` and `status` is absent when unknown. The program that wrote the card may keep its own bookkeeping under one more key, `caller`; the command never prints it. + +## The diary + +One JSON object per line. Four kinds are the agent's, in the order they happened: + +``` +{"kind":"said","text":"…"} something the agent said +{"kind":"result","text":"…"} the agent's final answer for a turn +{"kind":"cost","usd":0.62} what a stretch of the run cost, in US dollars +{"kind":"ended","status":"failed","detail":"…"} how the run ended; detail when it did not end well +``` + +Any other kind of line is the writing program's own; `show` leaves it out. diff --git a/packages/skill-logs/SPEC.md b/packages/skill-logs/SPEC.md new file mode 100644 index 000000000..ac1b5ce97 --- /dev/null +++ b/packages/skill-logs/SPEC.md @@ -0,0 +1,23 @@ +The agent logs for coding agents, as an npm package: the record of every run agents made on a project — for each run a card (what was asked, the ticket, the branch, the pull request, how it ended, what it cost) and a diary (what the agent said, its result) — under `agents//` on the `agent-data` branch of the project's own repository, never on a code branch; the `logs` command that reads it from any clone; and the instructions an agent follows to use it (`SKILL.md`): before planning or working a ticket, read its runs. + +The package knows git, the filesystem, the card's fields and four kinds of diary line, nothing else. The card carries eleven plain fields of the package's own; whatever else the program that ran the agent recorded sits under one key, `caller`, stored and never read. The diary is JSON lines, of which the package reads four kinds — what the agent said, its result, how the run ended, what it cost — and skips the rest. The same functions serve every caller: a long-lived process (a daemon that records the runs it drives and shows them) that keeps a checkout of the branch, and an agent's own shell, through the `logs` command a caller puts on the PATH of every agent it starts on its machine. Agents only read; the process that ran an agent records its run, at its end. + +The branch is a file store, the primitive `@gemstack/agent-data` provides: a branch nobody edits in a working tree, safe to push and pull eagerly. A long-lived process writes through its persistent checkout, `.branches/agent-data`, in one serialized cycle per branch — sync with origin, apply, commit, push — and reads the checkout; a push that loses a race is re-applied against the fresher state rather than forced. The `logs` command reads origin's copy of the branch, fetched once, and holds no checkout. + +## Glossary + +- **the card** - a run's `.json`: the package's fields, and the writer's own under `caller`. +- **the diary** - a run's `.jsonl`: one JSON object per line, four kinds the package knows among whatever the writer adds. +- **the funnel** - a caller's write cycle over the branch: apply a change to a checkout of it, commit, push. A long-lived process passes its own; the package's default is the persistent checkout's cycle. + +## Business logic — TL;DR + +- **The name** (`names`) - the runs directory, `agents`, at the root of the shared data branch `agent-data` (`@gemstack/agent-data` names the branch). +- **A run** (`run`) - the card and the diary as text and as values: which fields a card has and how one reads back, which lines are the agent's, how a person's directory is named from a git email, how a run matches a ticket, and newest first. +- **Where the runs live** (`store`) - the branch bound to a project, for a long-lived process: list, find, read a diary, record a run, patch two late facts onto its card, delete it, each write one commit through the funnel. +- **The command line** (`cli`, `bin/`) - the reads as commands for a shell: JSON on stdout, a reason on stderr, an exit code that tells a refusal from a usage error; the executable's directory and the skill's own directory are exported (`bin-dir`) for a caller that spawns agents. +- **The skill** (`SKILL.md`) - what the agent is told: the runs are on a branch and not in its checkout, the `logs` command is how it reads them, when to look back, and the two formats. + +## Before modifying/creating SPEC.md files + +You must always read and respect https://raw.githubusercontent.com/brillout/sdd/refs/heads/main/sdd.md diff --git a/packages/skill-logs/bin/SPEC.md b/packages/skill-logs/bin/SPEC.md new file mode 100644 index 000000000..dfdbf55b0 --- /dev/null +++ b/packages/skill-logs/bin/SPEC.md @@ -0,0 +1,5 @@ +The `logs` executable: it hands the shell's arguments, working directory and streams to the command line (`src/cli`) and exits with the code it returns. No logic of its own. + +## Before modifying/creating SPEC.md files + +You must always read and respect https://raw.githubusercontent.com/brillout/sdd/refs/heads/main/sdd.md diff --git a/packages/skill-logs/bin/logs b/packages/skill-logs/bin/logs new file mode 100755 index 000000000..8b9e2ba32 --- /dev/null +++ b/packages/skill-logs/bin/logs @@ -0,0 +1,11 @@ +#!/usr/bin/env node +import { runCli } from '../dist/cli.js' + +runCli(process.argv.slice(2), { cwd: process.cwd(), stdout: line => console.log(line), stderr: line => console.error(line) }) + .then(code => { + process.exitCode = code + }) + .catch(err => { + console.error(err) + process.exitCode = 1 + }) diff --git a/packages/skill-logs/package.json b/packages/skill-logs/package.json new file mode 100644 index 000000000..a3bf7a645 --- /dev/null +++ b/packages/skill-logs/package.json @@ -0,0 +1,50 @@ +{ + "name": "@gemstack/skill-logs", + "version": "0.0.1", + "description": "The agent logs for coding agents: the record of every run made on a project, its card and its diary, kept on a branch of the project's repository and read with the logs command.", + "license": "MIT", + "repository": { + "type": "git", + "url": "git+https://github.com/framework/the-framework.git", + "directory": "packages/skill-logs" + }, + "type": "module", + "engines": { + "node": ">=22.12.0" + }, + "publishConfig": { + "access": "public" + }, + "files": [ + "dist", + "bin/logs", + "SKILL.md" + ], + "bin": { + "logs": "bin/logs" + }, + "exports": { + ".": { + "types": "./dist/index.d.ts", + "default": "./dist/index.js" + }, + "./names": { + "types": "./dist/names.d.ts", + "default": "./dist/names.js" + }, + "./SKILL.md": "./SKILL.md" + }, + "scripts": { + "build": "pnpm -C ../agent-data build && tsc -p tsconfig.build.json", + "typecheck": "pnpm -C ../agent-data build && tsc --noEmit", + "test": "pnpm build && tsc -p tsconfig.test.json && node --test --test-timeout=60000 'dist-test/**/*.test.js'", + "clean": "rm -rf dist dist-test" + }, + "dependencies": { + "@gemstack/agent-data": "workspace:*" + }, + "devDependencies": { + "@types/node": "^20.0.0", + "typescript": "^7.0.2" + } +} diff --git a/packages/skill-logs/src/SPEC.md b/packages/skill-logs/src/SPEC.md new file mode 100644 index 000000000..5b4c94105 --- /dev/null +++ b/packages/skill-logs/src/SPEC.md @@ -0,0 +1,5 @@ +The package's source: the runs directory's name, what a run's two files hold, the reads and the writes behind the agent logs on the `agent-data` branch, and the command line. See the package `SPEC.md` for how the modules relate; each module's `*.SPEC.md` holds its business logic. + +## Before modifying/creating SPEC.md files + +You must always read and respect https://raw.githubusercontent.com/brillout/sdd/refs/heads/main/sdd.md diff --git a/packages/skill-logs/src/bin-dir.SPEC.md b/packages/skill-logs/src/bin-dir.SPEC.md new file mode 100644 index 000000000..517a3ed19 --- /dev/null +++ b/packages/skill-logs/src/bin-dir.SPEC.md @@ -0,0 +1,5 @@ +Where the `logs` executable lives, for a caller that puts it on a spawned process's PATH: a daemon does so for every agent it starts on its machine, so the agent's shell finds the command by name. Beside it, the package's own directory — the one holding `SKILL.md` — and the skill's name, `logs`, which is what `SKILL.md` declares itself and the name a harness lists the skill under, for a caller that links the skill into an agent's checkout. All three sit beside the compiled code, so the same paths hold from a workspace checkout and from an installed package. + +## Before modifying/creating SPEC.md files + +You must always read and respect https://raw.githubusercontent.com/brillout/sdd/refs/heads/main/sdd.md diff --git a/packages/skill-logs/src/bin-dir.ts b/packages/skill-logs/src/bin-dir.ts new file mode 100644 index 000000000..2b40b7f7d --- /dev/null +++ b/packages/skill-logs/src/bin-dir.ts @@ -0,0 +1,14 @@ +import { fileURLToPath } from 'node:url' + +/** + * Where the `logs` executable lives, for a caller that puts it on a spawned process's PATH — a + * daemon, for every agent it starts. Beside `dist/`, so it is the same path from a workspace + * checkout and from an installed package. + */ +export const CLI_BIN_DIR = fileURLToPath(new URL('../bin/', import.meta.url)) + +/** The skill's name: the `name` in `SKILL.md`'s front matter, and the directory a harness lists it under. */ +export const SKILL_NAME = 'logs' + +/** This package's directory — the one holding `SKILL.md`. Beside `dist/`, like the executable. */ +export const SKILL_DIR = fileURLToPath(new URL('..', import.meta.url)) diff --git a/packages/skill-logs/src/cli.SPEC.md b/packages/skill-logs/src/cli.SPEC.md new file mode 100644 index 000000000..a114f9caa --- /dev/null +++ b/packages/skill-logs/src/cli.SPEC.md @@ -0,0 +1,71 @@ +The package's command line: the reads a caller's pages do, for an agent (or a person) in a shell, in any clone of the repository — so an agent reads the runs through the one implementation a caller uses, and a second surface is never a second behaviour. Read-only: an agent's run is recorded by the process that ran it. + +## User story + +- An agent, started in a clone that holds no copy of the runs, reads the ones that worked its ticket before it plans or works it, and what those agents said. +- The user, in a terminal, does the same without any other tool. + +## Glossary + +- **refusal** - a rule saying no to a command: a run nobody has. Not a failure: the command ran, and the answer is "not this one". + +## Business logic — TL;DR + +- **Two commands over the package** - the bare command lists, `show` opens one run; each is the corresponding read and nothing more. +- **The list is narrowed and capped** - by ticket, by branch, and to the newest 20 unless told otherwise; the cards past the cap are never read. +- **The agent sees the package's fields only** - never the writer's bookkeeping under `caller`, and of the diary only the four kinds the package knows. +- **JSON out, a reason for a person, an exit code that says which** - every result is one JSON document on stdout; a refusal or a git failure also puts one line on stderr and exits 1; a command that cannot be read gets the usage on stderr and exits 2. +- **Reads come off origin, fetched once** - so a command sees every run every machine pushed. + +## Business logic + +### Two commands over the package + +#### User story + +See `## User story`. + +#### Business logic + +- the bare command, `[--ticket ] [--branch ] [--limit N]` - the runs, newest first, as a JSON array of cards. +- `show ` - one run: its card, plus `diary`, the agent's lines in order. An id that is not one is a usage error; an id no run has is refused. + +### The list is narrowed and capped + +#### User story + +An agent asks for one ticket's runs and gets those, not a history of hundreds. + +#### Business logic + +The runs are walked newest first from the branch's directory listings alone. `--ticket ` keeps a run whose card worked that ticket, by its exact path or by its file name; `--branch ` a run whose card names that branch; both together, both. The list stops at the cap — 20 unless `--limit` gives a whole number above 0 — counting the runs kept, and the cards past it are never read. + +### The agent sees the package's fields only + +#### Business logic + +A card is printed without `caller`. `show` prints the diary's four kinds — what the agent said, its result, how the run ended, what it cost — in order, and no other line; a run with no diary file has none. + +### JSON out, a reason for a person, an exit code that says which + +#### User story + +An agent parses what it is told; a person reads it; a script branches on the exit code. + +#### Business logic + +Every command writes exactly one JSON document to stdout. A refusal is `{ ok: false, reason }` — the reason a short fixed word plus what identifies the case (the id) — with one sentence on stderr saying the same for a person, and exit code 1. The reasons: `no-run`, `not-a-repo`. A git failure past the decision is reported the same way, reason `git-failed`, with git's own line. A command that cannot be read — an unknown command, an argument missing or extra, an unknown option, a limit that is not a whole number above 0, an id that is not one — gets the usage on stderr, no JSON, and exit code 2. + +### Reads come off origin, fetched once + +#### User story + +An agent on one machine reads the run the daemon on another machine recorded a minute ago. + +#### Business logic + +A read fetches the branch once, up front, and reads origin's copy — so the command sees every run every writer pushed. A repository with no remote reads its local copy of the branch instead. Outside any repository the command is refused as `not-a-repo` rather than failing on git — only git's own "not a repository" reads as that; a timeout, a missing git or a corrupt repository stays the failure it is. Nothing lands in the agent's own clone: no ref moves, no checkout is made. + +## Before modifying/creating SPEC.md files + +You must always read and respect https://raw.githubusercontent.com/brillout/sdd/refs/heads/main/sdd.md diff --git a/packages/skill-logs/src/cli.test.SPEC.md b/packages/skill-logs/src/cli.test.SPEC.md new file mode 100644 index 000000000..31c6a17fa --- /dev/null +++ b/packages/skill-logs/src/cli.test.SPEC.md @@ -0,0 +1,11 @@ +What the tests cover: every command of the command line against real git, from clones acting as agents that share one origin, and the contract around them. + +- **The bare command** - every person's runs off origin, newest first, the package's fields only (no `caller`), from a clone holding no checkout of the branch; `--ticket` by the ticket's file name or its path, `--branch`, `--limit`, and the cap counting the runs kept; the default cap; nothing lands in the agent's clone. +- **`show`** - the card with the agent's four kinds of diary line and none of the writer's; a run with no diary file has no lines; an id no run has is refused as no such run, on stdout and in one line on stderr. +- **Usage** - an unknown command, an unknown option, a limit that is not a whole number above 0, an argument missing or extra and an id that is not one all get the usage on stderr, no JSON, and exit code 2. +- **A repository with no remote** - reads come off its local copy of the branch. +- **Outside a repository** - refused as such. + +## Before modifying/creating SPEC.md files + +You must always read and respect https://raw.githubusercontent.com/brillout/sdd/refs/heads/main/sdd.md diff --git a/packages/skill-logs/src/cli.test.ts b/packages/skill-logs/src/cli.test.ts new file mode 100644 index 000000000..ccc8d8f04 --- /dev/null +++ b/packages/skill-logs/src/cli.test.ts @@ -0,0 +1,190 @@ +import { strict as assert } from 'node:assert' +import { test } from 'node:test' +import { mkdir, mkdtemp, realpath, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { nodeGitRunner, DATA_BRANCH } from '@gemstack/agent-data' +import { runCli, USAGE, DEFAULT_LIMIT } from './cli.js' +import { RUNS_DIR } from './names.js' + +const git = nodeGitRunner() +const RETRIED_RM = { recursive: true, force: true, maxRetries: 10 } as const + +const R1 = '2026-07-04T00-00-00-000Z' +const R2 = '2026-07-05T00-00-00-000Z' +const R3 = '2026-07-06T00-00-00-000Z' +const card = (id: string, over: Record = {}) => ({ id, startedAt: id.replace(/-(\d\d)-(\d\d)-(\d\d\d)Z$/, ':$1:$2.$3Z'), status: 'done', ...over }) + +/** The runs on the branch: two people, three runs, one diary with the writer's lines among the agent's. */ +const RUNS: Record = { + [`a@a/${R1}.json`]: JSON.stringify(card(R1, { intent: 'first try', ticket: 'tickets/2026-07-01_fix.md', branch: 'agent-r1', status: 'failed', caller: { pid: 1, host: 'laptop' } })), + [`a@a/${R1}.jsonl`]: [ + '{"kind":"session","driver":"claude-code"}', + '{"kind":"said","text":"Reading the ticket."}', + '{"kind":"action","label":"Bash"}', + '{"kind":"result","text":"Could not finish."}', + '{"kind":"cost","usd":0.5,"inputTokens":9}', + '{"kind":"ended","status":"failed","detail":"API 500"}', + '', + ].join('\n'), + [`b@b/${R2}.json`]: JSON.stringify(card(R2, { intent: 'second try', ticket: 'tickets/2026-07-01_fix.md', branch: 'agent-r2', pr: { number: 3, url: 'https://x/pull/3' }, cost: 1.25 })), + [`b@b/${R2}.jsonl`]: '{"kind":"ended","status":"done"}\n', + [`b@b/${R3}.json`]: JSON.stringify(card(R3, { intent: 'unrelated', branch: 'agent-r3' })), + 'b@b/notes.md': 'not a run\n', +} + +/** A bare origin with an `agent-data` branch holding `runs`, plus N clones acting as agents. */ +async function rig(clones: number, runs: Record = RUNS) { + const bare = await realpath(await mkdtemp(join(tmpdir(), 'logs-cli-bare-'))) + await git(['init', '--bare', '-b', 'main', bare], bare) + const seed = await realpath(await mkdtemp(join(tmpdir(), 'logs-cli-seed-'))) + await git(['clone', bare, seed], seed) + await git(['config', 'user.email', 's@s'], seed) + await git(['config', 'user.name', 's'], seed) + await writeFile(join(seed, 'README.md'), '# t\n') + await git(['add', '-A'], seed) + await git(['commit', '-m', 'init'], seed) + await git(['push', 'origin', 'main'], seed) + const commit = (await git(['commit-tree', '4b825dc642cb6eb9a060e54bf8d69288fbee4904', '-m', 'create the agent-data branch'], seed)).trim() + await git(['checkout', '-B', DATA_BRANCH, commit], seed) + for (const [rel, content] of Object.entries(runs)) { + await mkdir(join(seed, RUNS_DIR, rel, '..'), { recursive: true }) + await writeFile(join(seed, RUNS_DIR, rel), content) + } + if (Object.keys(runs).length) { + await git(['add', '-A'], seed) + await git(['commit', '-m', 'seed'], seed) + } + await git(['push', 'origin', DATA_BRANCH], seed) + const agents: string[] = [] + for (let i = 0; i < clones; i++) { + const parent = await realpath(await mkdtemp(join(tmpdir(), `logs-cli-agent${i}-`))) + const clone = join(parent, 'clone') + await git(['clone', bare, clone], parent) + await git(['checkout', '-b', `agent-a${i}`], clone) + agents.push(clone) + } + const cleanup = async () => { + for (const dir of [bare, seed, ...agents.map(a => join(a, '..'))]) await rm(dir, RETRIED_RM) + } + return { bare, seed, agents, cleanup } +} + +/** Run one command, capturing the contract: parsed stdout, stderr lines, exit code. */ +async function run(cwd: string, argv: string[]) { + const out: string[] = [] + const err: string[] = [] + const code = await runCli(argv, { cwd, stdout: line => out.push(line), stderr: line => err.push(line) }) + return { code, json: out.length ? JSON.parse(out.join('\n')) : undefined, stderr: err.join('\n') } +} + +test('the bare command lists every person\'s runs off origin, newest first, the package\'s fields only; --ticket, --branch and --limit narrow it', async () => { + const { agents, cleanup } = await rig(1) + const [a] = agents + try { + const all = await run(a!, []) + assert.equal(all.code, 0) + assert.deepEqual( + all.json.map((c: { id: string }) => c.id), + [R3, R2, R1], + ) + assert.deepEqual(all.json[2], { ...card(R1), intent: 'first try', ticket: 'tickets/2026-07-01_fix.md', branch: 'agent-r1', status: 'failed' }, 'caller is the writer\'s, not printed') + // By the ticket's file name, or the path a queue entry links to. + const byFile = await run(a!, ['--ticket', '2026-07-01_fix.md']) + assert.deepEqual(byFile.json.map((c: { id: string; status: string }) => [c.id, c.status]), [[R2, 'done'], [R1, 'failed']]) + assert.deepEqual((await run(a!, ['--ticket', 'tickets/2026-07-01_fix.md'])).json.map((c: { id: string }) => c.id), [R2, R1]) + assert.deepEqual((await run(a!, ['--ticket', 'other.md'])).json, []) + assert.deepEqual((await run(a!, ['--branch', 'agent-r3'])).json.map((c: { id: string }) => c.id), [R3]) + assert.deepEqual((await run(a!, ['--limit', '1'])).json.map((c: { id: string }) => c.id), [R3]) + assert.deepEqual((await run(a!, ['--ticket', '2026-07-01_fix.md', '--limit', '1'])).json.map((c: { id: string }) => c.id), [R2], 'the cap counts matches') + // Nothing lands locally: the agent's clone holds no copy of the branch. + await assert.rejects(git(['rev-parse', '--verify', DATA_BRANCH], a!)) + assert.equal((await git(['status', '--porcelain'], a!)).trim(), '') + } finally { + await cleanup() + } +}) + +test('the bare command caps at the default limit', async () => { + const many: Record = {} + for (let i = 0; i < DEFAULT_LIMIT + 5; i++) { + const id = `2026-01-01T00-00-${String(i).padStart(2, '0')}-000Z` + many[`p@p/${id}.json`] = JSON.stringify(card(id)) + } + const { agents, cleanup } = await rig(1, many) + try { + assert.equal((await run(agents[0]!, [])).json.length, DEFAULT_LIMIT) + assert.equal((await run(agents[0]!, ['--limit', '100'])).json.length, DEFAULT_LIMIT + 5) + } finally { + await cleanup() + } +}) + +test('show prints the card with the agent\'s lines of the diary, never the writer\'s; an unknown run is refused', async () => { + const { agents, cleanup } = await rig(1) + const [a] = agents + try { + const shown = await run(a!, ['show', R1]) + assert.equal(shown.code, 0) + assert.deepEqual(shown.json, { + ...card(R1), + intent: 'first try', + ticket: 'tickets/2026-07-01_fix.md', + branch: 'agent-r1', + status: 'failed', + diary: [ + { kind: 'said', text: 'Reading the ticket.' }, + { kind: 'result', text: 'Could not finish.' }, + { kind: 'cost', usd: 0.5, inputTokens: 9 }, + { kind: 'ended', status: 'failed', detail: 'API 500' }, + ], + }) + assert.deepEqual((await run(a!, ['show', R3])).json.diary, [], 'no diary file: no lines') + const missing = await run(a!, ['show', '2026-01-01T00-00-00-000Z']) + assert.equal(missing.code, 1) + assert.deepEqual(missing.json, { ok: false, reason: 'no-run', id: '2026-01-01T00-00-00-000Z' }) + assert.match(missing.stderr, /no run is named/) + } finally { + await cleanup() + } +}) + +test('usage: an unknown command, a stray argument, a bad limit and an unsafe id', async () => { + const { agents, cleanup } = await rig(1, {}) + const [a] = agents + try { + const unknown = await run(a!, ['dance']) + assert.equal(unknown.code, 2) + assert.equal(unknown.json, undefined) + assert.equal(unknown.stderr, USAGE) + assert.equal((await run(a!, ['--limit', '0'])).code, 2) + assert.equal((await run(a!, ['--limit', 'many'])).code, 2) + assert.equal((await run(a!, ['--nope'])).code, 2) + assert.equal((await run(a!, ['show'])).code, 2) + assert.equal((await run(a!, ['show', 'a', 'b'])).code, 2) + assert.equal((await run(a!, ['show', '../escape'])).code, 2) + assert.deepEqual((await run(a!, [])).json, [], 'no runs yet reads as none') + } finally { + await cleanup() + } +}) + +test('a repository with no remote reads its local branch; outside a repository is a refusal', async () => { + const solo = await realpath(await mkdtemp(join(tmpdir(), 'logs-cli-solo-'))) + try { + await git(['init', '-b', 'main'], solo) + await git(['config', 'user.email', 's@s'], solo) + await git(['config', 'user.name', 's'], solo) + await writeFile(join(solo, 'README.md'), '# t\n') + await git(['add', '-A'], solo) + await git(['commit', '-m', 'init'], solo) + const commit = (await git(['commit-tree', '4b825dc642cb6eb9a060e54bf8d69288fbee4904', '-m', 'create the agent-data branch'], solo)).trim() + await git(['branch', DATA_BRANCH, commit], solo) + assert.deepEqual((await run(solo, [])).json, []) + const outside = await run(tmpdir(), []) + assert.equal(outside.code, 1) + assert.deepEqual(outside.json, { ok: false, reason: 'not-a-repo' }) + } finally { + await rm(solo, RETRIED_RM) + } +}) diff --git a/packages/skill-logs/src/cli.ts b/packages/skill-logs/src/cli.ts new file mode 100644 index 000000000..0fddcdc9c --- /dev/null +++ b/packages/skill-logs/src/cli.ts @@ -0,0 +1,168 @@ +import { parseArgs } from 'node:util' +import { checkoutRoot, gitReason, nodeGitRunner, openBranchReader, type BranchReader, type GitRunner, DATA_BRANCH } from '@gemstack/agent-data' +import { RUNS_DIR } from './names.js' +import { agentLines, isRunId, newestFirst, parseDiary, parseRunCard, publicCard, runCardFile, runDiaryFile, runIdOfFile, workedTicket, type RunCard } from './run.js' + +/** + * The command line over the package: the reads a daemon's pages do, for an agent (and a person) + * in a shell, in any clone of the repository. Read-only: an agent's run is recorded by the + * process that ran it, not by the agent. + * + * The contract: JSON on stdout, one line for a person on stderr, and the exit code says how it + * went — 0 for a result, 1 for a refusal or a git failure, 2 for a command that could not be + * read. A refusal is a rule saying no (there is no such run); it is reported on stdout as + * `{ ok: false, reason }` so a caller parsing the output learns why, and on stderr so a person does. + * + * Reads go to origin's copy of the branch, fetched once, so a command sees every run every + * machine pushed. The persistent checkout a daemon keeps is never touched. + */ + +/** How many runs the bare command prints unless told otherwise: enough to see what happened lately, not the whole history. */ +export const DEFAULT_LIMIT = 20 + +export const USAGE = `usage: logs [command] + + (no command) [--ticket ] [--branch ] [--limit N] + the runs, newest first (the newest ${DEFAULT_LIMIT} unless --limit says + otherwise); --ticket keeps the runs that worked one ticket, + --branch the runs on one branch + show one run: its card, and what the agent said, its result, + how it ended, what it cost + +JSON on stdout. Exit code 1 for a refusal or a git failure (the reason on stderr), 2 for a usage error.` + +/** The streams and the working directory a run of the CLI sees. */ +export interface CliIo { + cwd: string + stdout: (line: string) => void + stderr: (line: string) => void +} + +/** A refusal: a rule said no, and the caller learns which. */ +export type CliRefusal = { ok: false; reason: string; [key: string]: unknown } + +/** Thrown inside a command to end it with a refusal. */ +class Refused extends Error { + constructor( + readonly outcome: CliRefusal, + readonly line: string, + ) { + super(line) + } +} + +/** Thrown inside a command for an argument that cannot be read: usage on stderr, exit 2. */ +class Usage extends Error {} + +/** Run the CLI: `argv` is everything after the program name. Resolves to the exit code. */ +export async function runCli(argv: string[], io: CliIo, git: GitRunner = nodeGitRunner()): Promise { + const [command, ...rest] = argv + const run = command === undefined || command.startsWith('--') ? list : Object.hasOwn(COMMANDS, command) ? COMMANDS[command] : undefined + const args = command === undefined || command.startsWith('--') ? argv : rest + if (!run) { + io.stderr(USAGE) + return 2 + } + try { + io.stdout(JSON.stringify(await run(args, io, git))) + return 0 + } catch (err) { + if (err instanceof Usage) { + io.stderr(`${err.message}\n\n${USAGE}`) + return 2 + } + if (err instanceof Refused) { + io.stdout(JSON.stringify(err.outcome)) + io.stderr(err.line) + return 1 + } + const detail = gitReason(err) + io.stdout(JSON.stringify({ ok: false, reason: 'git-failed', detail })) + io.stderr(detail) + return 1 + } +} + +type Command = (args: string[], io: CliIo, git: GitRunner) => Promise + +/** The bare command: the runs, newest first, filtered and capped. */ +const list: Command = async (args, io, git) => { + const { values } = parse(args, { ticket: { type: 'string' }, branch: { type: 'string' }, limit: { type: 'string' } }, 0) + const limit = values.limit === undefined ? DEFAULT_LIMIT : limitArg(values.limit) + const reader = await open(io.cwd, git) + const cards: RunCard[] = [] + // Newest first, and no further than the cap: the cards past it are never read. + for (const { person, id } of await runEntries(reader)) { + if (cards.length >= limit) break + const card = parseRunCard((await reader.read(`${RUNS_DIR}/${person}/${runCardFile(id)}`)) ?? '') + if (!card) continue + if (values.ticket !== undefined && !workedTicket(card, values.ticket)) continue + if (values.branch !== undefined && card.branch !== values.branch) continue + cards.push(card) + } + return cards.map(publicCard) +} + +const COMMANDS: Record = { + async show(args, io, git) { + const { positionals } = parse(args, {}, 1) + const id = positionals[0]! + if (!isRunId(id)) throw new Usage(`not a run id: ${id}`) + const reader = await open(io.cwd, git) + const entry = (await runEntries(reader)).find(e => e.id === id) + const card = entry ? parseRunCard((await reader.read(`${RUNS_DIR}/${entry.person}/${runCardFile(id)}`)) ?? '') : undefined + if (!entry || !card) throw new Refused({ ok: false, reason: 'no-run', id }, `no run is named ${id}`) + const diary = parseDiary((await reader.read(`${RUNS_DIR}/${entry.person}/${runDiaryFile(id)}`)) ?? '') + return { ...publicCard(card), diary: agentLines(diary) } + }, +} + +/** Every run on the branch by person and id, newest first, from the directory listings alone. */ +async function runEntries(reader: BranchReader): Promise> { + const entries: Array<{ person: string; id: string }> = [] + for (const person of await reader.list(RUNS_DIR)) { + for (const name of await reader.list(`${RUNS_DIR}/${person}`)) { + const id = runIdOfFile(name) + if (id !== undefined) entries.push({ person, id }) + } + } + return newestFirst(entries) +} + +/** The branch opened for reading, from wherever the command runs; outside a repo, a refusal. */ +async function open(cwd: string, git: GitRunner): Promise { + await inRepo(() => checkoutRoot(cwd, git)) + return openBranchReader(cwd, DATA_BRANCH, { git }) +} + +function limitArg(value: string): number { + if (!/^\d+$/.test(value) || Number(value) < 1) throw new Usage(`--limit takes a whole number above 0, got ${value}`) + return Number(value) +} + +/** + * Outside a repo, the commands have nothing to act on: said as a refusal, not a git failure. + * Only git's own "not a git repository" reads as that; a timeout, a missing git, or a corrupt + * repo stays the failure it is. + */ +async function inRepo(read: () => Promise): Promise { + try { + return await read() + } catch (err) { + if (!/not a git repository/i.test(err instanceof Error ? err.message : String(err))) throw err + throw new Refused({ ok: false, reason: 'not-a-repo' }, 'not inside a git repository') + } +} + +type Options = Record + +/** `parseArgs` with the positional count checked: too few or too many is a usage error. */ +function parse(args: string[], options: O, min: number, max: number = min) { + try { + const parsed = parseArgs({ args, options, allowPositionals: true, strict: true }) + if (parsed.positionals.length < min || parsed.positionals.length > max) throw new Usage(`expected ${max === min ? min : `${min} to ${max}`} argument(s), got ${parsed.positionals.length}`) + return parsed + } catch (err) { + throw err instanceof Usage ? err : new Usage(err instanceof Error ? err.message : String(err)) + } +} diff --git a/packages/skill-logs/src/index.SPEC.md b/packages/skill-logs/src/index.SPEC.md new file mode 100644 index 000000000..7da6668bd --- /dev/null +++ b/packages/skill-logs/src/index.SPEC.md @@ -0,0 +1,5 @@ +The package's main entry point: it gathers the runs directory's name, a run's card and diary (their fields, parsing and formatting), the branch's reads and writes with their seams, the command line and the executable's directory into one place for a caller to import. No business logic of its own. The name is also reachable on its own (`names`), for code that runs in a browser and must not pull in git. + +## Before modifying/creating SPEC.md files + +You must always read and respect https://raw.githubusercontent.com/brillout/sdd/refs/heads/main/sdd.md diff --git a/packages/skill-logs/src/index.ts b/packages/skill-logs/src/index.ts new file mode 100644 index 000000000..2a4424615 --- /dev/null +++ b/packages/skill-logs/src/index.ts @@ -0,0 +1,26 @@ +export { RUNS_DIR } from './names.js' +export { + isRunId, + runCardFile, + runDiaryFile, + runIdOfFile, + personDir, + ANONYMOUS_DIR, + parseRunCard, + formatRunCard, + parseDiary, + formatDiary, + isDiaryLine, + agentLines, + workedTicket, + newestFirst, + publicCard, + type RunCard, + type RunPatch, + type RunStatus, + type DiaryLine, + type AnyDiaryLine, +} from './run.js' +export { listRuns, findRun, runFiles, readDiary, writeRun, patchRun, deleteRun, runsPath, logsFunnel, resolveLogsDeps, type LogsDeps, type LogsFiles, type LogsFunnel } from './store.js' +export { runCli, USAGE, DEFAULT_LIMIT, type CliIo, type CliRefusal } from './cli.js' +export { CLI_BIN_DIR, SKILL_DIR, SKILL_NAME } from './bin-dir.js' diff --git a/packages/skill-logs/src/names.SPEC.md b/packages/skill-logs/src/names.SPEC.md new file mode 100644 index 000000000..93be91b6a --- /dev/null +++ b/packages/skill-logs/src/names.SPEC.md @@ -0,0 +1,5 @@ +The one name the package hangs off: the runs directory, `agents`, at the root of the shared data branch `agent-data` (named by `@gemstack/agent-data`, not here), under which each person's directory holds their runs. A convention, not a setting: `SKILL.md` names the same directory to every agent. Reachable on its own by code that runs in a browser and must not pull in git. + +## Before modifying/creating SPEC.md files + +You must always read and respect https://raw.githubusercontent.com/brillout/sdd/refs/heads/main/sdd.md diff --git a/packages/skill-logs/src/names.ts b/packages/skill-logs/src/names.ts new file mode 100644 index 000000000..4e61af428 --- /dev/null +++ b/packages/skill-logs/src/names.ts @@ -0,0 +1,10 @@ +/** + * The one name the package hangs off. No node imports, so browser-side code can name it too. + * + * The branch itself is not named here: the runs live on the shared data branch, `agent-data`, + * whose name `@gemstack/agent-data` exports as `DATA_BRANCH`. A convention, not a setting: + * `SKILL.md` names the same branch to every agent. + */ + +/** Where the runs live: `agents//.json` and `.jsonl` under the branch root. */ +export const RUNS_DIR = 'agents' diff --git a/packages/skill-logs/src/run.SPEC.md b/packages/skill-logs/src/run.SPEC.md new file mode 100644 index 000000000..a46b3f9a1 --- /dev/null +++ b/packages/skill-logs/src/run.SPEC.md @@ -0,0 +1,71 @@ +A run: one agent's work on the project from its start to its end, recorded as two files — the card and the diary. This is the pure half: what the two files hold, how they read back, how a run is matched. Nothing here touches git or a disk. + +## User story + +- An agent about to work a ticket reads the runs that worked it before, and what those agents said, without learning anything about the program that ran them. +- The user opens a run's card in a file listing and reads it as a person: the plain fields first, the program's own bookkeeping last. + +## Glossary + +- **the card** - a run's `.json`: eleven fields of the package's own, and the writer's under `caller`. +- **the diary** - a run's `.jsonl`: one JSON object per line, each with a `kind`. + +## Business logic — TL;DR + +- **The card's fields** - id, start and end time, status, intent, driver, model, branch, pull request, ticket, cost; the writer's own under one key. +- **A card reads back strictly** - each field kept only in the right shape, unknown top-level fields dropped, and no card at all without an id, a start and a status. +- **The diary's four kinds** - what the agent said, its result, how the run ended, what it cost; every other line is the writer's and is skipped. +- **A run id is a file name** - letters, digits, dashes and underscores only, so no id can climb out of a directory. +- **A person's directory is their git email, made safe** - lowercased, unsafe characters replaced, `anonymous` when nothing usable is left. +- **A run worked a ticket by its path or its file name** - the exact path recorded, or one ending in `/`. +- **Newest first is the id order reversed** - the writer's ids sort by time. + +## Business logic + +### The card's fields + +#### Business logic + +A card holds: `id`; `startedAt` and `endedAt` (ISO timestamps, the end absent while the run is going); `status`, one of `running`, `done`, `stopped`, `failed`; `intent`, what the agent was asked; `driver`, the agent program that ran, as the writer names it; `model`; `branch`, where the work is; `pr`, the pull request's number and URL; `ticket`, the ticket the run worked, as the writer names tickets; `cost`, in US dollars. Every field but the id, the start and the status may be absent. Under `caller` the writer keeps whatever else it records, as one object: the package stores it as given and never reads it. Two late facts, the branch and the pull request, are what a writer may patch onto a card after it landed. + +A card is written with the package's fields first in a fixed order, `caller` last, pretty-printed, so a person reads the plain part before the bookkeeping. + +### A card reads back strictly + +#### Business logic + +A card's text reads back as a card only when it is a JSON object with a string id that is a run id, a string start and a known status; anything else is not a card. Each other field is kept only in the shape the card defines — a string, a number, a pull request with a number and a URL, an object under `caller` — and a field of another shape is dropped rather than kept. A top-level field the card does not define is dropped too: the card is the package's contract, and the writer's place is `caller`. + +### The diary's four kinds + +#### Business logic + +A diary's lines are JSON objects with a string `kind`, in order; a line that is not one is skipped, and a line that does not parse ends the read, keeping what came before — a writer torn mid-line wrote nothing after it that can be trusted. Four kinds are the agent's: `said` with the text the agent said; `result` with the text of its final answer for a turn; `ended` with the status the run ended in — `done`, `stopped` or `failed` — and a detail when it did not end well; `cost` with what a stretch of the run cost in US dollars, when known. A line of one of these kinds without the fields it needs is not the agent's line. Any other kind is the writer's, and a line of any kind may carry more fields than named here; they pass through untouched. + +### A run id is a file name + +#### Business logic + +An id is a file name on the branch and a path segment in every read, so it is letters, digits, dashes and underscores only; anything else — a dot, a slash, a space, nothing at all — is not an id, and a file not named `.json` after one is not a card. + +### A person's directory is their git email, made safe + +#### Business logic + +The directory a person's runs go under is the git email their repository commits as: trimmed, lowercased, every character outside letters, digits, `@`, `.`, `_`, `+` and `-` replaced by `-`. The result must start with a letter or digit and be at most 64 characters, which rules out `.`, `..`, a dotfile name and a separator — the value comes from repository configuration and is joined onto a path. An email that cannot be made to fit, or none, files under `anonymous`, so the run is kept rather than dropped. + +### A run worked a ticket by its path or its file name + +#### Business logic + +A run worked a ticket when its card's ticket is exactly that text, or ends in `/` followed by it — so a ticket's file name and the path a queue entry links to both find it. Where tickets live is the writer's business, not the package's. + +### Newest first is the id order reversed + +#### Business logic + +The writer's ids sort by time, so sorting ids as text in reverse is newest first; no timestamp is parsed. + +## Before modifying/creating SPEC.md files + +You must always read and respect https://raw.githubusercontent.com/brillout/sdd/refs/heads/main/sdd.md diff --git a/packages/skill-logs/src/run.test.SPEC.md b/packages/skill-logs/src/run.test.SPEC.md new file mode 100644 index 000000000..4d83f2fd0 --- /dev/null +++ b/packages/skill-logs/src/run.test.SPEC.md @@ -0,0 +1,11 @@ +What the tests cover: a run's card and diary as pure text and values. + +- **Ids and file names** - letters, digits, dashes and underscores make an id; a dot, a slash, a space or nothing does not; a card file is named after one, a diary or any other file is not a card. +- **A person's directory** - an email trimmed, lowercased and made safe; a value that could climb out of a directory never starts with a dot or holds a separator; no identity, a blank one or an absurd length files as `anonymous`. +- **A card** - reads back with the package's fields and the writer's under `caller`, unknown top-level fields dropped, and a field of the wrong shape dropped; not JSON, not an object, no status, an unknown status or an id that is not one is no card; the command's view of a card has no `caller`; a card is written with the package's fields first and `caller` last, absent fields left out, and reads back the same. +- **A diary** - JSON lines each with a kind, a line without one or a bare value skipped; the agent's lines are the four kinds with their fields, extra fields carried along, and a known kind missing its fields is not the agent's; a torn line ends the read and keeps what came before; a diary formats back to one object per line. +- **Matching and ordering** - a run worked a ticket by its exact path or its file name, not by a shorter suffix, and a run with no ticket worked none; newest first is the id order reversed. + +## Before modifying/creating SPEC.md files + +You must always read and respect https://raw.githubusercontent.com/brillout/sdd/refs/heads/main/sdd.md diff --git a/packages/skill-logs/src/run.test.ts b/packages/skill-logs/src/run.test.ts new file mode 100644 index 000000000..9ffe04cf2 --- /dev/null +++ b/packages/skill-logs/src/run.test.ts @@ -0,0 +1,128 @@ +import { strict as assert } from 'node:assert' +import { test } from 'node:test' +import { ANONYMOUS_DIR, agentLines, formatDiary, formatRunCard, isRunId, newestFirst, parseDiary, parseRunCard, personDir, publicCard, runIdOfFile, workedTicket } from './run.js' + +test('a run id is letters, digits, dashes and underscores; a card file is named after one', () => { + assert.equal(isRunId('2026-09-08T18-14-30-111Z'), true) + assert.equal(isRunId('hand_picked'), true) + for (const bad of ['', '../escape', 'a/b', '.', '..', 'with space', 'dot.json']) assert.equal(isRunId(bad), false, bad) + assert.equal(runIdOfFile('2026-09-08T18-14-30-111Z.json'), '2026-09-08T18-14-30-111Z') + assert.equal(runIdOfFile('2026-09-08T18-14-30-111Z.jsonl'), undefined, 'a diary is not a card') + assert.equal(runIdOfFile('.hidden.json'), undefined) + assert.equal(runIdOfFile('notes.md'), undefined) +}) + +test('a person is filed under their git email, made safe; no identity is still filed', () => { + assert.equal(personDir('git@example.com'), 'git@example.com') + assert.equal(personDir(' Git@Example.COM '), 'git@example.com', 'trimmed and lowercased') + // The value comes from repository configuration and is joined onto a path: never `.`, `..`, + // a dotfile or a separator. + for (const hostile of ['..', '.', '../../etc/passwd', '.hidden', '/absolute', '..@evil.com']) { + const dir = personDir(hostile) + assert.ok(!dir.startsWith('.') && !dir.includes('/'), `${hostile} -> ${dir}`) + } + assert.equal(personDir('..'), ANONYMOUS_DIR) + assert.equal(personDir(undefined), ANONYMOUS_DIR) + assert.equal(personDir(' '), ANONYMOUS_DIR) + assert.equal(personDir('a'.repeat(200)), ANONYMOUS_DIR, 'an absurd length is not a directory name') +}) + +test('a card reads back with the package\'s fields and the writer\'s under caller, nothing else', () => { + const json = JSON.stringify({ + status: 'done', + id: 'r1', + startedAt: '2026-09-08T18:14:30.651Z', + endedAt: '2026-09-08T18:15:40.433Z', + intent: 'fix it', + driver: 'claude-code', + model: 'opus', + branch: 'agent-r1', + pr: { number: 7, url: 'https://x/pull/7' }, + ticket: 'tickets/2026-09-01_fix.md', + cost: 0.62, + caller: { pid: 923, host: 'laptop' }, + stray: 'ignored', + model_extra: 1, + }) + const card = parseRunCard(json)! + assert.deepEqual(card, { + id: 'r1', + startedAt: '2026-09-08T18:14:30.651Z', + status: 'done', + endedAt: '2026-09-08T18:15:40.433Z', + intent: 'fix it', + driver: 'claude-code', + model: 'opus', + branch: 'agent-r1', + pr: { number: 7, url: 'https://x/pull/7' }, + ticket: 'tickets/2026-09-01_fix.md', + cost: 0.62, + caller: { pid: 923, host: 'laptop' }, + }) + assert.deepEqual(Object.keys(publicCard(card)).includes('caller'), false, 'the command prints the package\'s fields only') + assert.ok(!('caller' in publicCard(card))) + // A field of the wrong shape is dropped, not kept: the card is the package's contract. + const odd = parseRunCard(JSON.stringify({ id: 'r2', startedAt: 'now', status: 'failed', pr: { number: '7' }, cost: '1', caller: [] }))! + assert.deepEqual(odd, { id: 'r2', startedAt: 'now', status: 'failed' }) +}) + +test('what is not a card reads as none', () => { + assert.equal(parseRunCard('not json'), undefined) + assert.equal(parseRunCard('[]'), undefined) + assert.equal(parseRunCard(JSON.stringify({ id: 'r1', startedAt: 'now' })), undefined, 'no status') + assert.equal(parseRunCard(JSON.stringify({ id: 'r1', startedAt: 'now', status: 'paused' })), undefined, 'not a status') + assert.equal(parseRunCard(JSON.stringify({ id: '../x', startedAt: 'now', status: 'done' })), undefined, 'not an id') +}) + +test('a card is written with the package\'s fields first, caller last, and reads back the same', () => { + const card = { id: 'r1', startedAt: 't', status: 'running' as const, caller: { pid: 1 }, intent: 'x' } + const text = formatRunCard(card) + assert.ok(text.endsWith('\n')) + assert.deepEqual(Object.keys(JSON.parse(text)), ['id', 'startedAt', 'status', 'intent', 'caller'], 'absent fields are not written') + assert.deepEqual(parseRunCard(text), card) +}) + +test('a diary is JSON lines; the agent\'s are four kinds, everything else is the writer\'s', () => { + const lines = parseDiary( + [ + '{"kind":"session","driver":"claude-code"}', + '{"kind":"said","text":"Reading the ticket."}', + '{"kind":"action","label":"Bash"}', + '"a bare string"', + '{"no":"kind"}', + '{"kind":"result","text":"Done.","sessionId":"s1"}', + '{"kind":"cost","usd":0.5,"inputTokens":28}', + '{"kind":"cost"}', + '{"kind":"ended","status":"failed","detail":"API 500"}', + '', + ].join('\n'), + ) + assert.equal(lines.length, 7, 'a line without a kind, and a non-object, are not lines') + assert.deepEqual(agentLines(lines), [ + { kind: 'said', text: 'Reading the ticket.' }, + { kind: 'result', text: 'Done.', sessionId: 's1' }, + { kind: 'cost', usd: 0.5, inputTokens: 28 }, + { kind: 'cost' }, + { kind: 'ended', status: 'failed', detail: 'API 500' }, + ]) + // A known kind with the wrong fields is not the agent's line. + assert.deepEqual(agentLines(parseDiary('{"kind":"said"}\n{"kind":"ended","status":"running"}\n{"kind":"cost","usd":"1"}\n')), []) + assert.equal(formatDiary(lines), lines.map(l => JSON.stringify(l) + '\n').join('')) +}) + +test('a torn line ends the diary read and keeps what came before', () => { + assert.deepEqual(parseDiary('{"kind":"said","text":"a"}\n{"kind":"sai'), [{ kind: 'said', text: 'a' }]) +}) + +test('a run worked a ticket by its exact path or its file name; newest first is the id order reversed', () => { + const card = { id: 'r1', startedAt: 't', status: 'done' as const, ticket: 'tickets/2026-09-01_fix.md' } + assert.equal(workedTicket(card, 'tickets/2026-09-01_fix.md'), true) + assert.equal(workedTicket(card, '2026-09-01_fix.md'), true) + assert.equal(workedTicket(card, '01_fix.md'), false, 'a suffix of the file name is not the file') + assert.equal(workedTicket({ id: 'r1', startedAt: 't', status: 'done' }, '2026-09-01_fix.md'), false) + assert.deepEqual(newestFirst([{ id: '2026-07-01T00-00-00-000Z' }, { id: '2026-09-01T00-00-00-000Z' }, { id: '2026-08-01T00-00-00-000Z' }]).map(c => c.id), [ + '2026-09-01T00-00-00-000Z', + '2026-08-01T00-00-00-000Z', + '2026-07-01T00-00-00-000Z', + ]) +}) diff --git a/packages/skill-logs/src/run.ts b/packages/skill-logs/src/run.ts new file mode 100644 index 000000000..e72aab72f --- /dev/null +++ b/packages/skill-logs/src/run.ts @@ -0,0 +1,214 @@ +// A run: one agent's work on the project, from its start to its end, recorded as two files on the +// branch — the card, `.json`, and the diary, `.jsonl`. This module is the pure half: what +// the two files hold, how they are read back, and how a run is matched. Nothing here touches git +// or a disk. + +/** How a run stands: still going, or how it ended. */ +export type RunStatus = 'running' | 'done' | 'stopped' | 'failed' + +/** + * The card: what was asked, where the work went, how it ended, what it cost. The package's + * fields, every one plain; `caller` is the writer's own bookkeeping, one key, stored and never read + * here. + */ +export interface RunCard { + /** The run's name, file-safe; the writer's ids sort by time, so the id order is the time order. */ + id: string + /** ISO timestamp. */ + startedAt: string + /** ISO timestamp, absent while the run is going. */ + endedAt?: string + status: RunStatus + /** What the agent was asked to do. */ + intent?: string + /** The agent program that ran: `claude-code`, `codex`, … as the writer names it. */ + driver?: string + model?: string + /** The branch the work is on. */ + branch?: string + /** The pull request the work is on. */ + pr?: { number: number; url: string } + /** The ticket the run worked, as the writer names tickets: a path, or a file name. */ + ticket?: string + /** What the run cost, in US dollars. */ + cost?: number + /** The writer's own record, under one key: stored as given, never read. */ + caller?: Record +} + +/** The two late facts a writer patches onto a card once the run's process is gone. */ +export type RunPatch = Partial> + +/** + * A diary line the package knows. A writer may put more fields on any of them, and any other + * kind of line beside them; both pass through untouched and are ignored here. + */ +export type DiaryLine = + /** Something the agent said. */ + | { kind: 'said'; text: string } + /** The agent's final answer for a turn. */ + | { kind: 'result'; text: string } + /** How the run ended, and why when it did not end well. */ + | { kind: 'ended'; status: Exclude; detail?: string } + /** What a stretch of the run cost, in US dollars. */ + | { kind: 'cost'; usd?: number } + +/** Any line of a diary: a JSON object with a `kind`, the package's or the writer's. */ +export type AnyDiaryLine = { kind: string } & Record + +const STATUSES: readonly RunStatus[] = ['running', 'done', 'stopped', 'failed'] + +function isStatus(value: unknown): value is RunStatus { + return typeof value === 'string' && (STATUSES as readonly string[]).includes(value) +} + +/** + * Whether a string can name a run: letters, digits, `-` and `_` only. An id is a file name on + * the branch and a path segment in every read, so this is the one thing that must hold: nothing + * an id can climb out of a directory with. + */ +export function isRunId(id: string): boolean { + return /^[A-Za-z0-9_-]+$/.test(id) +} + +/** The card's file name under a person's directory. */ +export function runCardFile(id: string): string { + return `${id}.json` +} + +/** The diary's file name under a person's directory. */ +export function runDiaryFile(id: string): string { + return `${id}.jsonl` +} + +/** The id a card file is named after, or `undefined` for a file that is not one. */ +export function runIdOfFile(name: string): string | undefined { + const id = /^(.+)\.json$/.exec(name)?.[1] + return id !== undefined && isRunId(id) ? id : undefined +} + +/** + * Longest directory name made from an email. Well past any real address, and short enough that + * the paths under it stay inside the limits of every platform. + */ +const MAX_PERSON_DIR = 64 + +/** The directory a person with no usable git identity is filed under. */ +export const ANONYMOUS_DIR = 'anonymous' + +/** + * The directory a person's runs are filed under, from the git email their repository commits as: + * lowercased, with anything outside a conservative set replaced by `-`. The result starts with a + * letter or digit, which rules out `.`, `..` and dotfile names — the value comes from repository + * configuration and is joined onto a path. Anything that cannot be made to fit is + * {@link ANONYMOUS_DIR} rather than a guess, so the run is still kept. + */ +export function personDir(email: string | undefined): string { + const cleaned = (email ?? '') + .trim() + .toLowerCase() + .replace(/[^a-z0-9@._+-]/g, '-') + return cleaned.length > 0 && cleaned.length <= MAX_PERSON_DIR && /^[a-z0-9][a-z0-9@._+-]*$/.test(cleaned) ? cleaned : ANONYMOUS_DIR +} + +/** + * A card read back from its file: the package's fields, each kept only when it has the right + * shape, plus `caller` as it is. `undefined` for anything that is not a card — not JSON, not an + * object, or missing an id, a start or a status. + */ +export function parseRunCard(json: string): RunCard | undefined { + let raw: unknown + try { + raw = JSON.parse(json) + } catch { + return undefined + } + if (!raw || typeof raw !== 'object' || Array.isArray(raw)) return undefined + const r = raw as Record + if (typeof r['id'] !== 'string' || !isRunId(r['id']) || typeof r['startedAt'] !== 'string' || !isStatus(r['status'])) return undefined + const card: RunCard = { id: r['id'], startedAt: r['startedAt'], status: r['status'] } + for (const key of ['endedAt', 'intent', 'driver', 'model', 'branch', 'ticket'] as const) { + if (typeof r[key] === 'string') card[key] = r[key] + } + if (typeof r['cost'] === 'number') card.cost = r['cost'] + const pr = r['pr'] + if (pr && typeof pr === 'object' && typeof (pr as Record)['number'] === 'number' && typeof (pr as Record)['url'] === 'string') { + card.pr = { number: (pr as { number: number }).number, url: (pr as { url: string }).url } + } + const caller = r['caller'] + if (caller && typeof caller === 'object' && !Array.isArray(caller)) card.caller = caller as Record + return card +} + +/** A card as it is written: the package's fields first, `caller` last, pretty, one trailing newline. */ +export function formatRunCard(card: RunCard): string { + const { id, startedAt, endedAt, status, intent, driver, model, branch, pr, ticket, cost, caller } = card + const ordered = { id, startedAt, endedAt, status, intent, driver, model, branch, pr, ticket, cost, caller } + return JSON.stringify(ordered, null, 2) + '\n' +} + +/** + * Every line of a diary, in order, each a JSON object with a string `kind`. A line that is not + * one is skipped; a line that does not parse ends the read — a writer torn mid-line wrote nothing + * after it that can be trusted — and everything before it is kept. + */ +export function parseDiary(jsonl: string): AnyDiaryLine[] { + const lines: AnyDiaryLine[] = [] + for (const line of jsonl.split('\n')) { + const trimmed = line.trim() + if (!trimmed) continue + let raw: unknown + try { + raw = JSON.parse(trimmed) + } catch { + break + } + if (raw && typeof raw === 'object' && !Array.isArray(raw) && typeof (raw as Record)['kind'] === 'string') lines.push(raw as AnyDiaryLine) + } + return lines +} + +/** A diary as it is written: one JSON object per line. */ +export function formatDiary(lines: readonly AnyDiaryLine[]): string { + return lines.map(line => JSON.stringify(line) + '\n').join('') +} + +/** Whether a line is one of the four kinds the package knows, with the fields that kind needs. */ +export function isDiaryLine(line: AnyDiaryLine): line is DiaryLine & AnyDiaryLine { + switch (line.kind) { + case 'said': + case 'result': + return typeof line['text'] === 'string' + case 'ended': + return isStatus(line['status']) && line['status'] !== 'running' && (line['detail'] === undefined || typeof line['detail'] === 'string') + case 'cost': + return line['usd'] === undefined || typeof line['usd'] === 'number' + default: + return false + } +} + +/** The agent's own lines of a diary: what it said, its results, how it ended, what it cost. Everything else is the writer's. */ +export function agentLines(lines: readonly AnyDiaryLine[]): Array { + return lines.filter(isDiaryLine) +} + +/** + * Whether a card's run worked `ticket`: the card names that exact path, or a path ending in + * `/` — so a ticket's file name and the path a queue entry links to both find it. Where + * tickets live is the writer's business, not the package's. + */ +export function workedTicket(card: RunCard, ticket: string): boolean { + return card.ticket !== undefined && (card.ticket === ticket || card.ticket.endsWith(`/${ticket}`)) +} + +/** Newest first: the writer's ids sort by time, so the id order is the time order, reversed. */ +export function newestFirst(cards: readonly T[]): T[] { + return [...cards].sort((a, b) => (a.id < b.id ? 1 : a.id > b.id ? -1 : 0)) +} + +/** The card as the command prints it: the package's fields, never the writer's. */ +export function publicCard(card: RunCard): Omit { + const { caller: _caller, ...own } = card + return own +} diff --git a/packages/skill-logs/src/store.SPEC.md b/packages/skill-logs/src/store.SPEC.md new file mode 100644 index 000000000..a7e3bcb8a --- /dev/null +++ b/packages/skill-logs/src/store.SPEC.md @@ -0,0 +1,61 @@ +Where the runs live, bound to the branch: `agents//.json` and `.jsonl` at the root of the `agent-data` branch of the project's repository, checked out at `.branches/agent-data` for a long-lived process. This module is that process's side: it reads the checkout and writes through the funnel. + +## User story + +- The user's daemon records every run it drove the moment the run is over, and the record is on origin before anyone asks for it. +- The user's dashboard lists the project's runs, other machines' included, and opens one to replay it. +- A pull request opened after a run, or the branch a cloud session's work landed on, shows on the run's row. +- The user deletes a run from the dashboard and it is gone for every machine. + +## Glossary + +- **the funnel** - the caller's write cycle over the branch: apply a change to a checkout of it, commit, push. The default is the persistent checkout's serialized cycle. + +## Business logic — TL;DR + +- **The seams every operation takes** - plain file operations and the funnel, injected so every operation is testable off disk and git, with the real filesystem and the persistent checkout's cycle as the defaults. +- **Listing and finding** - every person's runs off the checkout, newest first, a card that does not parse skipped, a cutoff by start time; one run's card, its two files, and its diary by id. +- **Recording a run** - the card and the diary as one commit, under the directory of the person the repository commits as, or where the run already sits. +- **Patching and deleting** - two late facts onto a card as one commit; both files removed as one commit. + +## Business logic + +### The seams every operation takes + +#### Business logic + +Every operation takes two things it does not own: plain file operations — read, write, delete, list — against whatever directory it is handed, and the funnel. A caller that leaves them out gets the defaults: the real filesystem and the persistent checkout's serialized cycle on the `agent-data` branch. A git runner is a third seam, for the one git question the module asks itself: the email the repository commits as. + +### Listing and finding + +#### User story + +See `## User story`: the dashboard's list and its run page. + +#### Business logic + +Listing reads every person's directory under the runs directory of the persistent checkout and every card file in each, newest first; a card that does not parse is skipped, and a missing directory is no runs. A cutoff, when given, keeps only the runs whose start time is at or after it. Finding a run by id searches every person's directory for its card file and answers with the card, or with the two files' paths for a caller that needs the file itself, or with the diary's lines — every line, the writer's included — `[]` for a run with no diary file; an id that is not one, or that no run has, is none of these. Nothing here throws. + +### Recording a run + +#### User story + +See `## User story`: the daemon at the end of a run. + +#### Business logic + +Recording a run writes its card and its diary as one commit through the funnel, under the directory of the person the repository commits as — read from git at the moment of writing, made safe as `run` says. A run that already sits somewhere on the branch, under any person, is written where it sits: a run recorded again — its ending, written by a later process that found it left running — stays in one place. An id that is not one is refused before anything is written. The funnel's outcome says whether the commit landed and whether it was pushed; a push that could not go out rides the next cycle. + +### Patching and deleting + +#### User story + +See `## User story`: the pull request that arrives late, the delete button. + +#### Business logic + +Patching puts the branch the work landed on, the pull request, or both onto a run's card as one commit, the rest of the card unchanged; it reports whether the card now carries the patch, committed — false when there is no such run. Deleting removes a run's card and diary as one commit; a run that is not there is a landed no-op. Both find the run inside the funnel's cycle, so a cycle re-run after a lost push race looks again. + +## Before modifying/creating SPEC.md files + +You must always read and respect https://raw.githubusercontent.com/brillout/sdd/refs/heads/main/sdd.md diff --git a/packages/skill-logs/src/store.test.SPEC.md b/packages/skill-logs/src/store.test.SPEC.md new file mode 100644 index 000000000..35a377c40 --- /dev/null +++ b/packages/skill-logs/src/store.test.SPEC.md @@ -0,0 +1,12 @@ +What the tests cover: the branch-bound reads and writes for a long-lived process, against real git with a bare origin. + +- **Recording** - a run lands as its card and diary under the directory of the person the repository commits as (lowercased), as one pushed commit named after the run, leaving the checkout clean; the diary is one object per line. +- **Listing and finding** - newest first; a cutoff keeps the runs started at or after it; a run is found by id with its card, its two files' paths and its diary — `[]` for a run with no diary file, none for an id no run has or that is not an id. +- **Patching** - the branch and the pull request land on the card as one pushed commit, and survive the next sync of the branch; a run that is not there patches nothing. +- **Deleting** - both files go as one pushed commit; a run already gone is a landed no-op. +- **Recording again** - a run another person recorded, ended by this machine, stays under that person; a card that does not parse is skipped by the listing; an id that is not one is refused unwritten. +- **No remote** - a run is recorded locally and the push is reported as not done; a project with no checkout has no runs. + +## Before modifying/creating SPEC.md files + +You must always read and respect https://raw.githubusercontent.com/brillout/sdd/refs/heads/main/sdd.md diff --git a/packages/skill-logs/src/store.test.ts b/packages/skill-logs/src/store.test.ts new file mode 100644 index 000000000..3c6e6c4d2 --- /dev/null +++ b/packages/skill-logs/src/store.test.ts @@ -0,0 +1,114 @@ +import { strict as assert } from 'node:assert' +import { test } from 'node:test' +import { mkdtemp, readFile, realpath, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { nodeGitRunner, fileBranchPath, withFileBranch, DATA_BRANCH } from '@gemstack/agent-data' +import { deleteRun, findRun, listRuns, patchRun, readDiary, runFiles, writeRun } from './store.js' +import { RUNS_DIR } from './names.js' + +const git = nodeGitRunner() +const RETRIED_RM = { recursive: true, force: true, maxRetries: 10 } as const + +/** A repo committing as `email`, with a bare origin. */ +async function repo(email = 'Dev@Example.com'): Promise<{ root: string; bare: string; cleanup: () => Promise }> { + const root = await realpath(await mkdtemp(join(tmpdir(), 'logs-store-'))) + const bare = await realpath(await mkdtemp(join(tmpdir(), 'logs-store-bare-'))) + await git(['init', '--bare', '-b', 'main', bare], bare) + await git(['init', '-b', 'main'], root) + await git(['config', 'user.email', email], root) + await git(['config', 'user.name', 't'], root) + await git(['config', 'commit.gpgsign', 'false'], root) + await writeFile(join(root, 'README.md'), '# t\n') + await git(['add', '-A'], root) + await git(['commit', '-m', 'init'], root) + await git(['remote', 'add', 'origin', bare], root) + await git(['push', 'origin', 'main'], root) + return { root, bare, cleanup: async () => { for (const dir of [root, bare]) await rm(dir, RETRIED_RM) } } +} + +const card = (id: string, over: Record = {}) => ({ id, startedAt: id.replace(/-(\d\d)-(\d\d)-(\d\d\d)Z$/, ':$1:$2.$3Z'), status: 'done' as const, ...over }) +const R1 = '2026-07-04T00-00-00-000Z' +const R2 = '2026-07-05T00-00-00-000Z' +const R3 = '2026-07-06T00-00-00-000Z' + +test('a run is recorded under the person the repo commits as, one pushed commit; listed newest first; found, read, patched and deleted', async () => { + const { root, bare, cleanup } = await repo() + try { + const wrote = await writeRun(root, card(R2, { intent: 'second', caller: { pid: 1 } }), [{ kind: 'said', text: 'hi' }, { kind: 'ended', status: 'done' }]) + assert.deepEqual(wrote, { ok: true, changed: true, pushed: true }) + assert.deepEqual(await writeRun(root, card(R1, { intent: 'first', ticket: 'tickets/a.md' }), []), { ok: true, changed: true, pushed: true }) + const person = join(fileBranchPath(root, DATA_BRANCH), RUNS_DIR, 'dev@example.com') + assert.deepEqual(JSON.parse(await readFile(join(person, `${R2}.json`), 'utf8')), { ...card(R2), intent: 'second', caller: { pid: 1 } }) + assert.equal(await readFile(join(person, `${R2}.jsonl`), 'utf8'), '{"kind":"said","text":"hi"}\n{"kind":"ended","status":"done"}\n') + assert.equal((await git(['log', '-1', '--format=%s', DATA_BRANCH], bare)).trim(), `logs: record run ${R1}`) + assert.equal((await git(['status', '--porcelain'], person)).trim(), '', 'committed, not merely written') + + assert.deepEqual((await listRuns(root)).map(c => c.id), [R2, R1], 'newest first') + assert.deepEqual((await listRuns(root, { since: Date.parse('2026-07-04T12:00:00.000Z') })).map(c => c.id), [R2], 'since keeps the runs started at or after it') + assert.deepEqual(await findRun(root, R1), { ...card(R1), intent: 'first', ticket: 'tickets/a.md' }) + assert.equal(await findRun(root, 'nope'), undefined) + assert.equal(await findRun(root, '../escape'), undefined) + assert.deepEqual(await readDiary(root, R2), [{ kind: 'said', text: 'hi' }, { kind: 'ended', status: 'done' }]) + assert.deepEqual(await readDiary(root, R1), []) + assert.equal(await readDiary(root, 'nope'), undefined) + assert.deepEqual(await runFiles(root, R1), { card: join(person, `${R1}.json`), diary: join(person, `${R1}.jsonl`) }) + + assert.equal(await patchRun(root, R1, { branch: 'claude/fix', pr: { number: 7, url: 'https://x/pull/7' } }), true) + assert.equal((await git(['log', '-1', '--format=%s', DATA_BRANCH], bare)).trim(), `logs: patch run ${R1}`) + assert.deepEqual((await findRun(root, R1))?.pr, { number: 7, url: 'https://x/pull/7' }) + // The sync the daemon runs a minute later keeps it: the patch was committed, not left dirty. + await withFileBranch(root, DATA_BRANCH, 'sync', async () => {}) + assert.equal((await findRun(root, R1))?.branch, 'claude/fix') + assert.equal(await patchRun(root, 'nope', { branch: 'x' }), false, 'no such run: nothing patched') + + assert.deepEqual(await deleteRun(root, R1), { ok: true, changed: true, pushed: true }) + assert.equal((await git(['log', '-1', '--format=%s', DATA_BRANCH], bare)).trim(), `logs: delete run ${R1}`) + assert.deepEqual((await listRuns(root)).map(c => c.id), [R2]) + assert.equal(await git(['show', `${DATA_BRANCH}:${RUNS_DIR}/dev@example.com/${R1}.jsonl`], bare).then(() => true, () => false), false, 'the diary went with it') + assert.deepEqual(await deleteRun(root, R1), { ok: true, changed: false, pushed: false }, 'gone already: a landed no-op') + } finally { + await cleanup() + } +}) + +test('a run recorded again stays where it sits, even under another person; a card that does not parse is skipped', async () => { + const { root, cleanup } = await repo('me@example.com') + try { + // Another machine's run, filed under its person on the branch. + await withFileBranch(root, DATA_BRANCH, 'seed', async dir => { + const { mkdir } = await import('node:fs/promises') + await mkdir(join(dir, RUNS_DIR, 'them@example.com'), { recursive: true }) + await writeFile(join(dir, RUNS_DIR, 'them@example.com', `${R3}.json`), JSON.stringify(card(R3, { status: 'running' }))) + await writeFile(join(dir, RUNS_DIR, 'them@example.com', `${R3}.jsonl`), '') + await writeFile(join(dir, RUNS_DIR, 'them@example.com', 'broken.json'), '{not json') + }) + assert.deepEqual((await listRuns(root)).map(c => [c.id, c.status]), [[R3, 'running']]) + // This machine ends it: the card stays under them, not under me. + assert.equal((await writeRun(root, card(R3, { status: 'stopped', endedAt: 'later' }), [{ kind: 'ended', status: 'stopped' }])).ok, true) + const files = await runFiles(root, R3) + assert.ok(files?.card.includes('/them@example.com/')) + assert.equal((await findRun(root, R3))?.status, 'stopped') + assert.deepEqual(await readDiary(root, R3), [{ kind: 'ended', status: 'stopped' }]) + assert.equal((await writeRun(root, { ...card('../escape'), id: '../escape' }, [])).ok, false, 'not an id: nothing written') + } finally { + await cleanup() + } +}) + +test('a repo with no remote records locally and says the push did not happen', async () => { + const root = await realpath(await mkdtemp(join(tmpdir(), 'logs-store-solo-'))) + try { + await git(['init', '-b', 'main'], root) + await git(['config', 'user.email', 's@s'], root) + await git(['config', 'user.name', 's'], root) + await writeFile(join(root, 'README.md'), '# t\n') + await git(['add', '-A'], root) + await git(['commit', '-m', 'init'], root) + assert.deepEqual(await writeRun(root, card(R1), []), { ok: true, changed: true, pushed: false }) + assert.deepEqual((await listRuns(root)).map(c => c.id), [R1]) + assert.deepEqual(await listRuns('/nowhere'), [], 'no checkout, no runs') + } finally { + await rm(root, RETRIED_RM) + } +}) diff --git a/packages/skill-logs/src/store.ts b/packages/skill-logs/src/store.ts new file mode 100644 index 000000000..527816f0e --- /dev/null +++ b/packages/skill-logs/src/store.ts @@ -0,0 +1,172 @@ +import { join } from 'node:path' +import { DATA_BRANCH, fileBranchPath, nodeBranchFileFs, nodeGitRunner, withFileBranch, type BranchFileFs, type CommitMessage, type FileBranchWrite, type GitRunner } from '@gemstack/agent-data' +import { RUNS_DIR } from './names.js' +import { + formatDiary, + formatRunCard, + newestFirst, + parseDiary, + parseRunCard, + personDir, + runCardFile, + runDiaryFile, + runIdOfFile, + isRunId, + type AnyDiaryLine, + type RunCard, + type RunPatch, +} from './run.js' + +// Where the runs live, bound to the branch: `agents//.json` and `.jsonl` at the +// root of the `agent-data` branch of the project's repository, checked out under +// `.branches/agent-data` for a long-lived process. This module is that process's side: it reads +// the checkout and writes through the caller's funnel. The command an agent runs reads the +// branch off origin instead (`cli`), holding no checkout. + +/** The plain-file seams an operation on the branch's files needs. */ +export type LogsFiles = BranchFileFs + +/** + * A caller's write funnel: apply `op` to a checkout of the branch, commit, push. The daemon's is + * the persistent checkout's serialized cycle; a test's fake stands in. + */ +export type LogsFunnel = (root: string, message: CommitMessage, op: (dir: string) => Promise) => Promise + +/** Injectable seams so every operation is unit-testable off disk and git; production takes the defaults. */ +export interface LogsDeps extends Partial { + funnel?: LogsFunnel + git?: GitRunner + log?: (message: string) => void +} + +/** The default funnel: the persistent checkout's write cycle, on the `agent-data` branch. */ +export const logsFunnel: LogsFunnel = (root, message, op) => withFileBranch(root, DATA_BRANCH, message, op) + +/** Fill in whatever a caller left out, so an operation reads the same way in tests and out. */ +export function resolveLogsDeps(deps: LogsDeps): LogsFiles & { funnel: LogsFunnel; git: GitRunner; log: (message: string) => void } { + const fs = nodeBranchFileFs() + return { + read: deps.read ?? fs.read, + write: deps.write ?? fs.write, + remove: deps.remove ?? fs.remove, + list: deps.list ?? fs.list, + funnel: deps.funnel ?? logsFunnel, + git: deps.git ?? nodeGitRunner(), + log: deps.log ?? (() => {}), + } +} + +/** The runs directory inside a checkout of the branch. */ +function runsDir(checkout: string): string { + return join(checkout, RUNS_DIR) +} + +/** The persistent checkout's runs directory under a project. */ +export function runsPath(root: string): string { + return runsDir(fileBranchPath(root, DATA_BRANCH)) +} + +/** The card and diary paths of one run inside a checkout, searched across every person's directory; `undefined` when it is nowhere. */ +async function locate(checkout: string, id: string, r: Pick): Promise<{ card: string; diary: string } | undefined> { + if (!isRunId(id)) return undefined + const dir = runsDir(checkout) + for (const person of await r.list(dir)) { + if ((await r.list(join(dir, person))).includes(runCardFile(id))) { + return { card: join(dir, person, runCardFile(id)), diary: join(dir, person, runDiaryFile(id)) } + } + } + return undefined +} + +/** + * Every run of the project, newest first, off the persistent checkout: each person's directory + * is read, a card that does not parse is skipped. `since` (epoch ms) keeps only the runs started + * at or after it. Never throws: a missing directory is no runs. + */ +export async function listRuns(root: string, opts: { since?: number } = {}, deps: LogsDeps = {}): Promise { + const r = resolveLogsDeps(deps) + const dir = runsPath(root) + const cards: RunCard[] = [] + for (const person of await r.list(dir)) { + for (const name of await r.list(join(dir, person))) { + if (runIdOfFile(name) === undefined) continue + const card = parseRunCard(await r.read(join(dir, person, name)).catch(() => '')) + if (!card) continue + if (opts.since !== undefined && !(Date.parse(card.startedAt) >= opts.since)) continue + cards.push(card) + } + } + return newestFirst(cards) +} + +/** Where one run's two files sit on the persistent checkout, or `undefined` when the run is not there. */ +export async function runFiles(root: string, id: string, deps: LogsDeps = {}): Promise<{ card: string; diary: string } | undefined> { + return locate(fileBranchPath(root, DATA_BRANCH), id, resolveLogsDeps(deps)) +} + +/** One run's card, or `undefined` when there is no such run or its card does not parse. */ +export async function findRun(root: string, id: string, deps: LogsDeps = {}): Promise { + const r = resolveLogsDeps(deps) + const files = await runFiles(root, id, deps) + return files ? parseRunCard(await r.read(files.card).catch(() => '')) : undefined +} + +/** One run's diary, every line: `[]` for a run with no diary file, `undefined` for no such run. */ +export async function readDiary(root: string, id: string, deps: LogsDeps = {}): Promise { + const r = resolveLogsDeps(deps) + const files = await runFiles(root, id, deps) + if (!files) return undefined + return parseDiary(await r.read(files.diary).catch(() => '')) +} + +/** + * Record a run: its card and its diary, as one commit, under the directory of the person the + * repository commits as — or where the run already sits, when it does, so a run recorded again + * (its ending, written by a later process) stays in one place. Never throws; the funnel's outcome + * says whether the commit landed and whether it was pushed. + */ +export async function writeRun(root: string, card: RunCard, diary: readonly AnyDiaryLine[], deps: LogsDeps = {}): Promise { + const r = resolveLogsDeps(deps) + if (!isRunId(card.id)) return { ok: false, committed: false, error: `not a run id: ${card.id}` } + const email = await r.git(['config', 'user.email'], root).catch(() => '') + const person = personDir(email) + return r.funnel(root, `logs: record run ${card.id}`, async checkout => { + const existing = await locate(checkout, card.id, r) + const files = existing ?? { + card: join(runsDir(checkout), person, runCardFile(card.id)), + diary: join(runsDir(checkout), person, runDiaryFile(card.id)), + } + await r.write(files.card, formatRunCard(card)) + await r.write(files.diary, formatDiary(diary)) + }) +} + +/** + * Patch a late fact onto a run's card — the branch its work landed on, the pull request — as one + * commit. True when the card now carries it, committed; a push that could not go out rides the + * next cycle. False when there is no such run. + */ +export async function patchRun(root: string, id: string, patch: RunPatch, deps: LogsDeps = {}): Promise { + const r = resolveLogsDeps(deps) + let patched = false + const result = await r.funnel(root, `logs: patch run ${id}`, async checkout => { + patched = false + const files = await locate(checkout, id, r) + const card = files ? parseRunCard(await r.read(files.card).catch(() => '')) : undefined + if (!files || !card) return + await r.write(files.card, formatRunCard({ ...card, ...patch })) + patched = true + }) + return patched && (result.ok || result.committed) +} + +/** Delete a run, card and diary, as one commit. A run that is not there is a landed no-op. */ +export async function deleteRun(root: string, id: string, deps: LogsDeps = {}): Promise { + const r = resolveLogsDeps(deps) + return r.funnel(root, `logs: delete run ${id}`, async checkout => { + const files = await locate(checkout, id, r) + if (!files) return + await r.remove(files.card) + await r.remove(files.diary).catch(() => {}) + }) +} diff --git a/packages/skill-logs/tsconfig.build.json b/packages/skill-logs/tsconfig.build.json new file mode 100644 index 000000000..e57806493 --- /dev/null +++ b/packages/skill-logs/tsconfig.build.json @@ -0,0 +1,6 @@ +{ + "extends": "../../tsconfig.base.json", + "compilerOptions": { "outDir": "dist", "rootDir": "src" }, + "include": ["src"], + "exclude": ["src/**/*.test.ts"] +} diff --git a/packages/skill-logs/tsconfig.json b/packages/skill-logs/tsconfig.json new file mode 100644 index 000000000..404aab404 --- /dev/null +++ b/packages/skill-logs/tsconfig.json @@ -0,0 +1,5 @@ +{ + "extends": "../../tsconfig.base.json", + "compilerOptions": { "noEmit": true, "rootDir": "src" }, + "include": ["src"] +} diff --git a/packages/skill-logs/tsconfig.test.json b/packages/skill-logs/tsconfig.test.json new file mode 100644 index 000000000..eebda2fc6 --- /dev/null +++ b/packages/skill-logs/tsconfig.test.json @@ -0,0 +1,5 @@ +{ + "extends": "../../tsconfig.base.json", + "compilerOptions": { "outDir": "dist-test", "rootDir": "src" }, + "include": ["src"] +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index ad47efbca..fb4df0358 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -11,6 +11,9 @@ importers: '@gemstack/skill-branches': specifier: workspace:* version: link:packages/skill-branches + '@gemstack/skill-logs': + specifier: workspace:* + version: link:packages/skill-logs '@gemstack/skill-queue': specifier: workspace:* version: link:packages/skill-queue @@ -50,6 +53,9 @@ importers: '@gemstack/skill-branches': specifier: workspace:* version: link:../skill-branches + '@gemstack/skill-logs': + specifier: workspace:* + version: link:../skill-logs '@gemstack/skill-queue': specifier: workspace:* version: link:../skill-queue @@ -158,6 +164,19 @@ importers: specifier: ^7.0.2 version: 7.0.2 + packages/skill-logs: + dependencies: + '@gemstack/agent-data': + specifier: workspace:* + version: link:../agent-data + devDependencies: + '@types/node': + specifier: ^20.0.0 + version: 20.19.43 + typescript: + specifier: ^7.0.2 + version: 7.0.2 + packages/skill-queue: dependencies: '@gemstack/agent-data':