[orsissimo]

Summary

Step 2 of the split proposed in #115, "safe replacement, retirement, cleanup, preservation refs", based on 0782f33. This implements invariants 6 and 7 from #125 and the cleanup part of invariant 3.

• Give every engine container its own workspace volume. launch.sh retire [--post-process] [--discard-uncommitted] removes stopped containers by ID after checking them; their volumes stay.
• Make launch.sh cleanup [--discard-uncommitted] [--allow-unharvested] take the engagement lock and refuse while the bare repo holds agent work that HEAD's history lacks. Harvest first, or use --allow-unharvested to keep work left by a rewritten upstream as recovery refs only.
• Before deleting a workspace volume, bare repo, mirror or earlier cleanup remnant, copy everything Git keeps there into the checkout as recovery refs and verify it with git fsck --full. This includes refs, reflogs, objects, stashes, unfinished operations and git-dir files; work reported as uncommitted requires --discard-uncommitted to be dropped.
• Refuse start while an engine container, workspace volume or engagement runtime remains. Remove docker rm -f from engine container replacement, the alpine helper from launch.sh, and rm -rf recipes from the docs.
• 22 commits, 27 files, +16,260/-825 lines; mostly tests, including tests/test_launch.sh, tests/test_retire.sh and tests/test_preserve.sh.

Choices for step 2

@moodmosaic, I chose option B for step 2. Previously, a stopped container kept the workspace in its own layer, so removing it could lose commits that never reached the bare repo. Option A was to copy each stopped container's Git metadata out with docker cp first. With option B, each engine container's /workspace is a named Docker volume labeled with the project, checkout, engagement and a per-engagement token. retire leaves that work in place; cleanup inspects every workspace with real Git in a helper container before deleting anything.

state.json records base_commit on the host, and harvest uses it instead of the bare repo's refs/swarm/base, which agents can write. Recovery refs live in the checkout under refs/swarm-recovery/...; the engine never deletes them. What do you think of this approach, and does it work for you? Would you prefer the three small fixes for temporary files, Bash 3.2 exit status and runtime_pi.sh on UID != 1000 in a separate PR?

Changes

• af7958d: add lib/preserve.sh checks that HEAD contains the bare repo's agent-work, swarm/* and agent-parked/* tips, plus refs/swarm-runs/ reserved for Multi-Agent Post-Process Proposal #112.
• 4ce87d8: start the bare repo from the checkout's HEAD, remove old agent branch refs and record base_commit in the state file.
• cab56ba: make lib/harness.sh push unshipped commits before the loop resets the workspace; if commits remain unshipped, exit without resetting.
• b6baa96: record interactive sessions before releasing the lock, keep the bare repo and mirrors while a session is open, and add only missing mirrors when joining.
• ef7e8ff: add recovery bundles in lib/preserve-scan.sh and lib/preserve.sh, with object inventories and checks that imported refs retain every object.
• 2fd92fe: add launch.sh retire. Check all selected containers in snapshots before removing any; require stopped containers with their own workspace volumes and leave containers that changed since inspection.
• d044c23: create launch.sh temporary files in TMPDIR or /tmp and remove them on exit, including failures and interrupts.
• 2cb1328: add launch.sh cleanup and lib/cleanup.sh. Scan and preserve repository locations before deletion; checkpoint interrupted cleanup and recheck its recovery refs on retry. --discard-uncommitted drops exactly the work listed.
• b1b76e1: fix tests/runtime_pi.sh for Linux host UIDs other than 1000: read its auth fixture before changing ownership and remove agent-owned bare-repo files through the test image.
• b55fb82: create labeled workspace volumes, record the Docker daemon and engagement token, and require a clean runtime for new engagements. Bind lifecycle commands to that engagement; add stop --numbered and SWARM_WAIT_POLL (default 10 seconds between wait checks).
• 2b05163: return failure when Bash 3.2 stops launch.sh, retire or cleanup on an unset variable; fail unit test files that exit without their summary.
• 3341839: let harvest.sh merge work based on the recorded engagement base after HEAD moves on. Record completed and pending merges; add --again for an undone or abandoned harvest, still requiring the base in HEAD's history.
• 30ed5a0: show volume only and unfinished cleanup in dashboard.sh.
• 5b660de: wrap long lines in USAGE.md and CHANGELOG.md.
• c68cb53: host reads of the bare repo and the mirrors never fetch and never check signatures. Preservation scans read no global or system Git config, and stop before reading a work tree when a repository's config sets a filter driver or includes other files. Cap what a retire or cleanup helper writes to any one host file with SWARM_HELPER_OUTPUT_LIMIT_MB (default 16384 MB), keep no container log, and check scan archives before extraction.
• 1073247: preserve Git checkouts outside workspace volumes as recovery refs before retiring containers. Add retire --discard-uncommitted and SWARM_RETIRE_LAYER_LIMIT_MB (default 100 MB of bundles per container); retire still refuses bare repositories and separate git dirs outside volumes.
• c1d5f69: clone bare repos and mirrors with git clone --bare --no-local so objects held only by earlier recovery refs are not copied into each new engagement. Each mirror keeps the base's recorded submodule commit under refs/swarm/gitlink, fetching it by ID when needed; warn if the source lacks it.
• 2fc3b60: after the Git 2.56.0 CI failure, make the swapped loose files test expect a hash-mismatch stop or both files stored, based on the copy Git reads.
• 5c55c16: also unset SWARM_ENGAGEMENT_TOKEN in tests/_test_env.sh so a token in the caller's environment does not reach the tests.
• 76fa1be: let cleanup accept an ignored .claude/ in the workspace's own checkout when it holds only the engine's settings.local.json. Anything else there, and any other ignored directory, is still reported.
• bf49b79: make harvest.sh --again also merge recorded commits by ID when HEAD lacks them and no branch of the bare repo brings them back, including after an interactive branch is deleted or agent-work is moved back. Each commit must exist locally and be built on the engagement's base; it stays recorded until merged, including across a conflict or an interrupted run.
• 0d5e973: stop first stops what runs without the engagement lock, so a post-process holding it stays stoppable, then waits up to SWARM_STOP_TIMEOUT seconds for the lock and stops what a start launched meanwhile; it fails when the lock stays held. wait waits up to a minute for the lock once the agents are done, and SWARM_STOP_TIMEOUT must be whole seconds.

Breaking changes for embedders

• Each engine container now has a named /workspace volume. The workspace mount, engagement labels, container name, --restart no and --rm=false come after docker_args.
• The state file records a Docker daemon ID and random engagement token. post-process and interactive joining an engagement, stop, wait, retire and cleanup refuse another daemon; commands keep the Docker context they started with.
• A nonempty SWARM_ENGAGEMENT_TOKEN binds stop, post-process, interactive and harvest.sh to one engagement. A mismatch refuses; with that token supplied, post-process and interactive cannot create a new engagement.
• New engagements require a clean runtime. Stop the agents, run harvest.sh, then ./launch.sh cleanup before starting another. While cleanup has a checkpoint, start, post-process, interactive, stop and harvest.sh refuse until it finishes.
• stop stops containers by ID only after checking the engagement token, first without the engagement lock, then under it after waiting up to SWARM_STOP_TIMEOUT seconds for a start still launching agents; it fails when the lock stays held. An interactive session being started, and a wait that already runs and goes on to its post-process, are not waited for. --numbered selects numbered agents only; the default includes post-process and interactive containers.
• post-process refuses while numbered agents run or an earlier post-process container exists. Stop it if needed, then use ./launch.sh retire --post-process before running it again.
• Engagements started by 0.23.0 or earlier are not taken over. launch.sh commands other than status and logs, and harvest.sh, refuse them; finish them with the release that started them.
• New bare repos and mirrors hold what the checkout's or submodule's branches, tags and HEAD reach; mirrors also keep the base's recorded submodule commits when available. Other objects held only by remote-tracking refs, stashes, reflogs or recovery refs are no longer copied.
• SWARM_STOP_TIMEOUT must be a whole number of seconds below 1000000; other values make stop refuse before it stops anything.

Self-review checklist

• This PR addresses a single concern (why?). If it covers multiple independent changes, I've split them into separate PRs.

Test plan

Results on 0d5e973 unless noted.

• ./tests/test.sh --unit: 3658 tests in 18 files passed on macOS 26.5.2 with /bin/bash 3.2.57 and Git 2.50.1; 3663 passed on Ubuntu 24.04 arm64 with Bash 5.2.21 and Git 2.43.0, as UID 501 and UID 1000; 3654 passed on Ubuntu 24.04 with Git 2.56.0.
• ./tests/test.sh --runtime: signal trap 16/16, runtime_retire.sh 83/83 and runtime_cleanup.sh 33/33 on OrbStack with Docker 29.8.2 and on Ubuntu 24.04 with Docker 29.1.3, as UID 501 and UID 1000.
• shellcheck -s bash --severity=warning is clean. Every commit was reviewed and tested on its own before it was committed.
• Mutation-test the new checks across the series: about 215 mutations; reverted checks fail tests apart from a few mutations shown to be equivalent.
• Run two short live engagements in a row per checkout with prompts specifying each agent's work, on macOS/OrbStack and Ubuntu as UID 501 and UID 1000. Exercise start, an interactive commit, stop, post-process, refusal of a second post-process, retire --post-process, post-process again, harvest and refusal to start before cleanup. retire names and refuses an untracked file; retire --discard-uncommitted preserves both scratch repositories as recovery refs. Cleanup leaves no container, volume or bare repo; git fsck --full passes. Recovery refs per engagement: 515 and 527 on every host.
• Run two engagements in a row per checkout with two agents choosing their own small changes for 20 minutes, on macOS/OrbStack and Ubuntu as UID 501. Each produces 23 to 31 commits merged by post-process and harvest. On both hosts, retire names an untracked log in a test fixture repository under /tmp; --discard-uncommitted preserves its history without the log. On macOS, cleanup also refuses an uncommitted edit left in a volume; --discard-uncommitted drops only that file. git fsck --full passes; recovery refs grow by 1,100 to 1,300 per engagement (on 3a40a98).
• Check an engagement started on Ubuntu on an earlier day by the Isolate runtime state and serialize engagement operations #125 code (what 0.23.0 ships). This branch's status worked; start, post-process, stop, harvest.sh, retire and cleanup refused with an explanation and left its containers, state file and checkout unchanged (on 3a40a98).
• On a 905 MB checkout whose branches and tags reach 21 MB, git clone --bare --no-local takes the same 0.8 s as the old copy and creates a 21 MB bare repo instead of 905 MB (on 3a40a98).
• Start an engagement in the background and run stop as soon as its first agent runs, three times per host on macOS/OrbStack and on Ubuntu as UID 501: stop returned only after start had finished launching, with no container of the engagement left running. With the engagement lock held by another process, stop with SWARM_STOP_TIMEOUT=5 stopped what ran, failed and said why, then succeeded once the lock was free.
• Phase 2 of ./tests/test.sh --all.

[Copilot]

🟡 Changes recommended

Critical defects can leave undone harvest work unrestored with --again and allow stop to report success before a concurrent start finishes launching agents.

4 open issues

• --again skips restoring missing harvested or pending commits · New
• stop can report success before concurrent startup finishes · New
• Ignored-directory collapsing hides exempt generated files · New
• Tests inherit the caller’s engagement token · New

What changed in this PR

Implements the retirement and cleanup stage of #115 and #125, preserving agent work before removing engagement resources.

Changes:

• Adds persistent workspace volumes and explicit retirement and cleanup with verified recovery refs.
• Strengthens engagement identity checks, harvest bookkeeping, and protection for unshipped commits.
• Updates lifecycle documentation, dashboard status, and unit and Docker runtime coverage.

File	Description
USAGE.md	Documents lifecycle commands, safeguards, and recovery procedures.
tests/​test.sh	Extends runtime coverage and rejects incomplete unit runs.
tests/​test_session_end_push.sh	Tests unshipped commits and scratch-push recovery.
tests/​test_runtime.sh	Updates legacy-engagement refusal checks.
tests/​test_harvest.sh	Tests base-aware merging and harvest recovery bookkeeping.
tests/​test_dashboard.sh	Tests engagement-bound actions and retained-volume status.
tests/​test_ci_workflows.sh	Verifies unit tests must reach their summaries.
tests/​runtime_signal_trap.sh	Adapts stop checks to engagement state snapshots.
tests/​runtime_retire.sh	Adds real-Docker retirement and interruption coverage.
tests/​runtime_pi.sh	Fixes fixture ownership and runtime resource cleanup.
tests/​runtime_cleanup.sh	Adds real-Docker preservation and repeated-engagement coverage.
tests/​_test_env.sh	Adds a shared SIGINT capability check.
README.md	Explains workspace persistence and the cleanup lifecycle.
lib/​preserve.sh	Adds containment checks, recovery imports, and verification.
lib/​harness.sh	Protects unshipped commits before workspace resets.
harvest.sh	Adds base-aware merges, recorded progress, and --again.
dashboard.sh	Shows retained volumes and binds stop/post-process actions to engagements.
CHANGELOG.md	Records preservation features and lifecycle compatibility changes.

_{Review effort: Balanced}