diff --git a/CHANGELOG.md b/CHANGELOG.md index 538c388..399b957 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,204 @@ ## Unreleased +- **The bare repo and the mirrors hold what the checkout's branches, + tags and HEAD reach.** They were local clones, which copy the whole + object store, so objects that only remote-tracking branches, the + stash, the reflogs or earlier recovery refs keep went into the bare + repo, the mirrors and every workspace, and each cleanup kept them + again as new recovery refs, more after every engagement in the same + checkout. A checkout of about 900 MB whose branches reach 21 MB gave + every workspace the 900 MB. Both are now cloned through Git's + transport (`--no-local`). A mirror also keeps the commit the + engagement's base records for its submodule (`refs/swarm/gitlink`), + fetched by id when the submodule's checkout has moved on, and + `start` warns when the submodule lacks it. +- **retire keeps the Git checkouts a container has outside its + volume.** It refused any container with a repository in its own + layer, and agents leave some there routinely (a test's fixture in + `/tmp`), so an engagement ended with `docker cp` and `docker rm` by + hand. A checkout there is now scanned in the container's snapshot, + copied into the checkout as recovery refs (`.../l/`, listed in the + snapshot's `sources`), and the checkout is verified with `git fsck` + before any container goes. Work outside Git's history in it stops + retire unless `--discard-uncommitted` (new for retire; cleanup passes + its own on) drops exactly what it lists; other repositories there + (bare ones, git dirs apart from their work trees) still refuse, and a + container's checkouts may take `SWARM_RETIRE_LAYER_LIMIT_MB` megabytes + (default 100). cleanup, like retire, now writes a snapshot's + `sources` ref only after `git fsck` passed. +- **Hardening: what agents write cannot steer the host's Git or fill + its disk.** The host's reads of the bare repo and the mirrors never + fetch and never check signatures; the preservation scans read no + global or system Git config, and stop before they read a work tree + when a repository's config sets a filter driver (which now blocks, + like `lfs.storage`) or includes other files, or when a repository + below it is not one they check; what a retire or cleanup helper + container writes to a file on the host is capped + (`SWARM_HELPER_OUTPUT_LIMIT_MB`, default 16384) and keeps no container + log, and a scan archive is checked before it is extracted. +- **The dashboard shows workspace volumes with no container, and an + unfinished cleanup.** A numbered agent or post-process whose + container retire removed showed `configured`, like one that never + started; it now shows `volume only` (its files are on its workspace + volume until cleanup). Once no container is left, a line counts the + engagement's workspace volumes; a cleanup that has not finished is + shown instead. +- **Fix: harvest merges agent work built on the engagement's base + after HEAD moved on.** It refused any branch that did not descend from + HEAD, so agent work pushed after an interactive branch was merged, or + after commits made in the checkout, could not be harvested. It now + checks the commit the engagement started from (`base_commit`, recorded + on the host): a branch built on it merges while HEAD's history still + has it, and still has what earlier harvests of the engagement merged + (recorded too). A history rewritten after the start, or one that + dropped harvested work or gave up a merge (each merge is recorded + before it starts, and committed whatever the Git config says), still + refuses; `harvest.sh --again` merges that work again on purpose, after + undoing a harvest say, even once the bare repo's branches no longer + bring it back. A merge that stops on a conflict says how to + finish or give it up, a harvest refuses while a merge is in progress, + and one within the same second as the previous one no longer fails on + its restore tag. Harvest no longer fetches tags from the bare repo, + and refuses when a tag or branch of the checkout has the name of a + harvested branch (`_agent-harvest/`). +- **Fix: a run that Bash 3.2 stops on an unset variable no longer ends + with status 0.** macOS's `/bin/bash` runs an EXIT trap with status 0 + when `set -u` stops a script, so `launch.sh`, retire and cleanup, + which have EXIT traps, would report a stop of that kind (a bug) as + success. Such a run now ends with 1. `tests/test.sh --unit` also + fails a test file that never printed its summary, which the same + quirk would have let pass; files that skip on purpose print one. +- **Breaking: workspace volumes, one Docker daemon per engagement, new + engagements only on a clean runtime directory.** Each engine container + gets a `/workspace` volume of its own (`-ws-`, labeled + with the project, checkout, engagement, engagement token, container, + kind, context and a nonce that is read back), so its work outlives it + for retire and cleanup. The workspace mount, the engagement labels, + `--restart no`, `--rm=false` and the container name come after + `docker_args`. The state file records the engagement's Docker daemon + and a random token; `post-process` and `interactive` joining it, + `stop`, `wait`, `retire` and `cleanup` refuse another daemon, and each + command keeps the Docker context it started with. `stop` stops by ID + only containers that carry the engagement's token (`--numbered`: the + numbered agents only), first without the engagement lock, then under + it, after waiting up to its grace for a start still launching agents, + and fails when one still runs or the lock stays held; `wait` checks + the token again before it goes on; + `SWARM_ENGAGEMENT_TOKEN` binds `stop`, `post-process`, `interactive` + and `harvest.sh` to one engagement, and the dashboard's `s` and `p` go + through `launch.sh`. `start`, and `post-process` or `interactive` when + they would create the engagement, refuse while the runtime directory + holds anything of one; nothing is removed by force any more (no + `docker rm -f`, no `alpine` helper, no `rm -rf` recipes): end an + engagement with `harvest.sh` and `./launch.sh cleanup`, which gains + `--allow-unharvested` for work a rewritten upstream left behind and + checks the volumes before it retires anything. `post-process` refuses + while numbered agents of the engagement run, and while an earlier + post-process container is there (`./launch.sh retire --post-process`). + Bare repos and mirrors are cloned to `.new` and renamed once ready, + and a mistake in the swarmfile or a failed image build stops `start` + before the engagement is created. The runtime directory must be a + place cleanup can work in. Engagements of earlier releases are not + taken over: every command but `status` and `logs` refuses them. Finish + such an engagement with the release that started it (check it out, + stop the agents, run `harvest.sh`, follow its cleanup steps), then + come back. +- **Fix: `tests/runtime_pi.sh` on Linux hosts whose UID is not 1000.** + The test handed its Codex auth fixture to the container's user before + the driver read it on the host, so the Codex checks failed, and its + cleanup could not remove what the agents wrote into the bare repo. It + now reads the fixture first and removes the bare repo as root with + its test image. +- **Feature: `launch.sh cleanup [--discard-uncommitted]`.** Deletes this + checkout's engagement (containers, through retire's checks; workspace + volumes; the bare repo; the mirrors; leftovers of an earlier cleanup) + after copying everything Git keeps in each location into the checkout + as recovery refs (`refs/swarm-recovery/e//s/`, + `` being the engagement id in hex, with a `sources` blob naming + each) and verifying them with `git fsck --full`. It refuses while the + bare repo holds agent work the checkout lacks, while a session is open + or starting, when a location holds work outside Git's history (unless + `--discard-uncommitted`, which drops exactly what it lists), when + the checkout cannot keep the copies by itself, or when something is + mounted where it would delete. A checkpoint in the state file lets a + run that stopped half way be finished by the next; every run checks + everything again and checks the recovery refs earlier runs wrote, and + start, post-process, interactive, stop and harvest.sh refuse + meanwhile. Files the containers wrote are deleted through a throwaway + engine-image container as root, only once a token shows it sees the + same directory. The scanner gains a `files` mode for what a deletion + left of a repository. retire and cleanup ignore the caller's `GIT_*` + variables (except the choice of config files). +- **Fix: launch.sh leaves no temporary files behind.** The profile file + of `interactive` and the agents' list of `start` were removed only + when the command got that far; any earlier exit, a failure or Ctrl-C + left them, on macOS in the user's temporary directory whatever + `TMPDIR` said. They are now made in `TMPDIR` (or `/tmp`) and removed + when `launch.sh` exits, however it does. +- **Feature: `launch.sh retire [--post-process]`.** Removes this + checkout's stopped engine containers (or only the post-process one) by + ID, without `-f`, after checking every selected container first: + exited or created, no restart policy or auto-remove, exactly one mount + at `/workspace` that is its own workspace volume and nothing mounted + or on tmpfs below it, no open interactive session, no git config in + the container that keeps repository data elsewhere (the global and + system files its git names with `git var`, so Git 2.42 or later in the + image), no repository data outside the volume (worktrees, links, + alternates, config), no symlink in the volume to data the container + wrote (compared with `docker diff` and with the original image; a name + `docker diff` cannot report exactly, not being UTF-8, refuses), and no + Git repository anywhere outside the volume, made by the container or + shipped by its image, changed or not (`docker diff` cannot show every + change). The checks run in a `docker commit` snapshot of the + container. A container that changed since its check is left. Workspace + volumes stay. It needs no swarmfile, takes the engagement lock and + checks the engagement's Docker daemon. `tests/runtime_retire.sh` (in + `tests/test.sh --runtime` and `--all`) runs it on real stopped + containers. +- **Feature: recovery snapshots (building block).** + `lib/preserve-scan.sh` packs everything Git keeps in a repository + location (every ref, reflog entry, stash, operation state, object, + and every file of its git dirs) into a bundle and reports work-tree + content outside Git's history; `swarm_recovery_import` copies such a + bundle into the checkout under `refs/swarm-recovery// + /` and checks it, and `swarm_recovery_verify_checkout` + re-hashes the checkout's objects. Cleanup uses them in a later + change; no command calls them yet. +- **Fix: open interactive sessions keep the bare repo and the + mirrors.** `interactive` held the lock in a helper process until the + container ran, and `start` could still replace the bare repo and the + mirrors under a running session. `interactive` now records the + session under the lock with its pid, that process's start time and + the Docker daemon, releases the lock and execs `docker run`. While + that process runs, or the container runs after a detach, `start` + fails, and so do `post-process` and `interactive` when they would + create the bare repo. A joining session no longer replaces the + mirrors; it only adds missing ones. `post-process` now mirrors the + submodules when it creates the bare repo, as `start` and + `interactive` do. +- **Fix: the harness keeps commits that no push shipped.** Each + loop iteration reset the workspace to `origin/agent-work`, so + commits left from a session whose push and park both failed, or + from a restart before the push, stayed only in the reflog. The loop + now fetches with `--prune` and checks HEAD against origin's + `agent-work`, `swarm/*` and `agent-parked/*`; commits on none of + them go through the session-end push first, and if they are still + unshipped the harness exits without resetting. After a scratch + worktree push, the branch moves to the pushed commits, since the + originals were shipped as copies, but only when the tree is clean + and `git cherry` finds every local commit on origin. The scratch + push also stopped reporting success when it could not list the + commits to transplant. +- **Fix: a new bare repo starts from the checkout's HEAD.** The + clone copied every local branch, so an old local `agent-work`, + `swarm/*` or `agent-parked/*` branch in the checkout reached the new + engagement and harvest could take it for new agent work. + `create_bare_repo` now deletes the `swarm/*` and `agent-parked/*` + branches in the new bare repo, points `agent-work` and + `refs/swarm/base` at the checkout's HEAD, and records that commit as + `base_commit` in the state file. A checkout without a commit is + refused. - **Fix: a signal that ends costs.sh, progress.sh or dashboard.sh no longer prints "stopped before the end of its run" on Bash 4.1 and later.** Bash runs the EXIT trap with status 0 when a signal ends a diff --git a/README.md b/README.md index 8c0bc36..0221a5e 100644 --- a/README.md +++ b/README.md @@ -50,7 +50,7 @@ Host runtime dir (mode 700) ~/project/ ── git clone ──> upstream.git (rw) --bare mirrors/*.git (ro) | - | docker volumes + | bind mounts | .-----------.----------+-----------. | | | | @@ -61,6 +61,8 @@ Host runtime dir (mode 700) v v /workspace/ /workspace/ (agent-work) (agent-work) + volume volume + -ws- -ws- ``` All containers mount the same bare repo. Each runs @@ -68,6 +70,15 @@ All containers mount the same bare repo. Each runs run one agent session, push. When one agent pushes, others see the changes on the next fetch. +Each container's `/workspace` is a Docker volume of its own, so its +files outlive the container. One engagement goes from `start` to +`harvest.sh`, which merges the agents' work into the checkout, then to +`./launch.sh retire`, which removes the stopped containers once nothing +lives only in them, and `./launch.sh cleanup`, which copies everything +Git kept in the volumes, the bare repo and the mirrors into the +checkout as recovery refs before it deletes them. The next `start` +needs a runtime directory cleaned up that way. + The runtime directory is private to the user who runs the swarm, `~/.local/state/swarm-core/` by default (see [USAGE.md](USAGE.md#runtime-directory)). @@ -89,6 +100,10 @@ SWARM_CONFIG=swarm.json ./launch.sh wait # Open a human-guided session from a named agent profile: SWARM_CONFIG=swarm.json ./launch.sh interactive hunter +# Once the work is harvested (wait does it), delete the engagement; +# what Git had in it stays in the checkout as recovery refs: +./launch.sh cleanup + # Or place swarm.json in your repo root and launch. ./launch.sh start --dashboard ``` diff --git a/USAGE.md b/USAGE.md index 28a4f76..c9371ca 100644 --- a/USAGE.md +++ b/USAGE.md @@ -8,6 +8,9 @@ SWARM_CONFIG=swarm.json ./launch.sh start --dashboard # Later, after agents are running or have exited: SWARM_CONFIG=swarm.json ./launch.sh wait + +# Then delete the engagement before the next start: +./launch.sh cleanup ``` All configuration lives in the swarmfile (JSON). Place a @@ -53,8 +56,11 @@ names, and existing bind mounts still refer to their original paths. ## Commands ```bash -./launch.sh start [--dashboard] # Launch numbered agents. -./launch.sh stop # Stop all agents. +./launch.sh start [--dashboard] # Start an engagement: launch + # numbered agents. +./launch.sh stop [--numbered] # Stop this engagement's + # containers (or only the + # numbered agents). ./launch.sh status # Show containers. ./launch.sh logs N # Tail agent N logs. ./launch.sh wait # Wait for already-started @@ -66,6 +72,10 @@ names, and existing bind mounts still refer to their original paths. ./launch.sh interactive hunter # Start a human-guided driver # session from agents[].name. ./launch.sh shell --agent hunter # Start a shell in that profile. +./launch.sh retire # Remove stopped containers. +./launch.sh cleanup # Keep what Git had as recovery + # refs, then delete the + # engagement. ``` ## Environment variables @@ -89,12 +99,16 @@ Credentials stay as env vars (not in shell history). | `SWARM_RUNTIME_DIR` | | See [Runtime directory](#runtime-directory). | | `SWARM_PROJECT` | | See [Project identity](#project-identity). | | `SWARM_ENGAGEMENT_ID` | | See [Engagements](#engagements). | +| `SWARM_ENGAGEMENT_TOKEN` | | See [Engagements](#engagements). | | `SWARM_LOCK_FD` | | See [Engagement lock](#engagement-lock). | | `SWARM_SKIP_DEP_CHECK` | | Set to `1` to silence dependency version warnings. | | `SWARM_ACTIVITY_TIMEOUT` | `0` | Seconds of logfile silence before the in-container watchdog SIGTERMs the agent CLI's process group. `0` disables. See [Activity watchdog](#activity-watchdog). | | `SWARM_ACTIVITY_POLL` | `10` | Watchdog mtime-poll interval, in seconds. Rarely needs tuning. | | `SWARM_WATCHDOG_GRACE` | `10` | Grace window between watchdog SIGTERM and SIGKILL. Rarely needs tuning. | -| `SWARM_STOP_TIMEOUT` | `60` | Seconds `./launch.sh stop` passes to `docker stop -t`, so the harness's SIGTERM trap has time to ship any in-flight local commits via `_session_end_push` before SIGKILL hits. See [Stopping the swarm](#stopping-the-swarm). | +| `SWARM_STOP_TIMEOUT` | `60` | Seconds `./launch.sh stop` passes to `docker stop -t`, so the harness's SIGTERM trap has time to ship any in-flight local commits via `_session_end_push` before SIGKILL hits. Also how long `stop` waits for the engagement lock. See [Stopping the swarm](#stopping-the-swarm). | +| `SWARM_WAIT_POLL` | `10` | Seconds between the checks of `./launch.sh wait`. | +| `SWARM_RETIRE_LAYER_LIMIT_MB` | `100` | The most, in megabytes, that the Git checkouts a container has outside its workspace volume may take as bundles for retire to keep them. | +| `SWARM_HELPER_OUTPUT_LIMIT_MB` | `16384` | The most, in megabytes, that a helper container of retire or cleanup may write to any one file on the host (its output, its errors, a scan archive). | Per-group credentials (`api_key`, `auth_token`, `base_url`) are set in the swarmfile. Use `$VAR` references to pull @@ -225,9 +239,10 @@ gap. Both rarely need adjustment outside the test suite. ### Stopping the swarm -`./launch.sh stop` sends each numbered agent and post-process -container `docker stop -t 60`. The 60 s grace gives the -harness's SIGTERM trap time to ship +`./launch.sh stop` sends every running container of the engagement +(numbered agents, post-process, interactive sessions) +`docker stop -t 60`; `--numbered` stops only the numbered agents. +The 60 s grace gives the harness's SIGTERM trap time to ship any in-flight local commits (in-place rebase → scratch worktree → `agent-parked/*` salvage) before docker forces SIGKILL. Without this window, commits the agent made during @@ -237,6 +252,11 @@ session-end push pipeline only runs at the bottom of each loop iteration and never gets to run on a mid-session SIGTERM. +Commits that no push shipped, because the push and the park both +failed or the container restarted first, are not reset away: at the +top of the next iteration the harness pushes them, and when that +fails it exits and leaves them on the workspace branch. + For larger swarms where 45+ agents race the same push lock, raise the grace: @@ -250,6 +270,28 @@ The harness handles SIGTERM and SIGINT identically; an `emergency shutdown complete` indicates the trap fired and the emergency push attempt completed. +`stop` acts on the engagement the state file records when it starts, +through its token: it lists the running containers that carry it, +reads each one's labels again right before it stops it, and stops it +by ID, so a container of a later engagement is never stopped. It does +this first without the engagement lock, which a post-process holds for +its whole run, and lists them again after each pass. A start that is +still launching agents holds the lock until it is done, so `stop` then +waits for the lock, up to `SWARM_STOP_TIMEOUT` seconds, and under it +stops what came meanwhile. After a success no agent or post-process of +the engagement runs or is about to start, with two exceptions: an +interactive session holds the lock only while it prepares, so a +container it is starting is not waited for, and a `wait` already +running goes on to its post-process or harvest once it sees the agents +stopped, as it does when they finish. `stop` fails when the lock stays +held that long (run it again: it stops what runs then before it waits +for the lock), when containers keep appearing, when one of them still +runs afterwards, or when Docker cannot tell. It refuses an +engagement of an earlier release (see +[Engagements of earlier releases](#engagements-of-earlier-releases)), +and stops nothing while Docker talks to another daemon than the +engagement's. + ### Extra Docker arguments Pass arbitrary flags to every `docker run` invocation via the @@ -268,6 +310,11 @@ This is useful for mounting the host Docker socket, adding devices or capabilities, setting network modes, or passing any other flags that the harness does not manage natively. +What the engine sets comes after `docker_args`, and Docker takes the +last value of a flag given twice, so an entry there cannot replace it: +the workspace volume at `/workspace`, the engagement labels, +`--restart no`, `--rm=false` and the container's name. + Docker's `-e` flag accepts two forms: `-e VAR=value` passes a literal value, and `-e VAR` (no `=value`) inherits `VAR` from the caller's environment at `launch.sh start` time, omitting it @@ -363,14 +410,23 @@ Interactive containers appear as `I1`, `I2`, etc. with their branch and `dirty`, `unharvested`, or `harvested` state; they do not count toward numbered-agent completion. +A numbered agent or the post-process that has no container but a +workspace volume of the engagement shows `volume only`: `./launch.sh +retire` removed its container (or `docker run` never made it), and its +files stay on the volume until `./launch.sh cleanup`. Once no container +is left, a line under the recent commits counts those volumes. A +cleanup that has not finished (running, or stopped before the end) is +shown there instead, until it finishes; if it is not running, run +`./launch.sh cleanup` again. + | Key | Action | |-----|--------| | `q` | Quit. | | `1`-`9` | Logs for agent N. | | `P` | Post-process logs (the `P` row), if it exists. | | `h` | Harvest results. | -| `s` | Stop numbered agents and post-process. | -| `p` | Start post-process after confirmation; replaces an exited run. | +| `s` | Stop the containers of the engagement shown (`./launch.sh stop`). | +| `p` | After confirmation, stop the numbered agents and start post-process; refuses while an earlier post-process container is there. | The Model column appends the agent's reasoning effort as a parenthesised letter: `(h)` high, `(m)` medium or minimal, `(l)` low, @@ -512,6 +568,13 @@ or automatically via `./launch.sh wait` after numbered agents have already been started. `./launch.sh wait` does not launch numbered agents. +`post-process` refuses while numbered agents of the engagement run: +stop them (`./launch.sh stop --numbered`) or wait for them first. A +post-process container is never replaced: while +`-agent-post` is there, `post-process` refuses. Stop it first +if it runs, then remove it with `./launch.sh retire --post-process`; +its workspace volume stays for cleanup. + The post-process agent clones the same bare repo, sees all commits on `agent-work`, runs its prompt, and pushes. @@ -522,8 +585,31 @@ post-processing, run `./launch.sh post-process`; use that command directly when you intentionally want to run only the post-process agent and then harvest. +`harvest.sh` merges a branch built on the commit the engagement started +from (`base_commit` in the state file) as long as HEAD's history still +has that commit, so it also merges agent work after you merged an +interactive branch or made commits of your own. It refuses when HEAD's +history no longer has it (rewritten after the engagement started, by a +force-push or a rebase): take what you need by hand, then end the +engagement with `./launch.sh cleanup --allow-unharvested`. It also +records what each harvest merged, and refuses when HEAD no longer has +that (a rebase dropped some of it): merging again would bring it back. +Each merge is recorded before it starts, and committed whatever merge +options the Git config sets (`--no-commit`, `--squash`). A merge that +stops on a conflict says how to finish it or give it up. Finished by +hand, it counts as merged; given up, or cut short with the harvest +(killed, say), the next harvest refuses as well, as does one run while +the merge is still in progress. After undoing a harvest on purpose +(`git reset --hard `, below) or giving up such a merge, +`./harvest.sh --again` merges that work again, also what the bare +repo's branches no longer bring back (a branch deleted there, or +`agent-work` moved back): it merges the recorded commits by id from +this checkout, which keeps them recorded until HEAD has them, and +refuses one the checkout no longer has. + Before merging, `harvest.sh` tags the current branch tip with a -local `swarm-harvest--