Skip to content

About

A meta-harness to drive up test coverage in poorly tended to codebases

Resources

Stars

5 stars

Watchers

0 watching

Forks

Repository files navigation

dotnet-coverage-harness

A harness that runs an agentic coding CLI (Claude Code, OpenCode or GitHub Copilot CLI) in a loop to raise the branch coverage of .NET solutions to a target. It writes outside-in component tests and can then raise the mutation score with Stryker.NET.

The harness has no model of its own. It finds the solution, builds it, runs the tests, measures coverage, decides what to ask the agent for next, checks that the agent kept to the rules, and commits or reverts each iteration. The agent only writes test code.

It's a .NET tool. You need the .NET SDK (8 or later), git, and the agent CLI you want to use, already logged in. With the .NET 10 SDK you can run it straight from its package without installing it:

dnx dotnet-coverage-harness --src "/abs/path/to/repos/*" --target 100

Or install it once and run it as a dotnet command:

dotnet tool install -g dotnet-coverage-harness
dotnet coverage-harness --src /abs/path/to/one-repo --target 90 --agent copilot

The package isn't on NuGet yet. Until it is, pack it into the dist/ folder at the root of this repository (it's gitignored) and install or run it from there:

dotnet pack src/CoverageHarness -c Release -o dist
dotnet tool install -g --add-source ./dist dotnet-coverage-harness
dnx dotnet-coverage-harness --add-source ./dist --yes -- --src /abs/path/to/one-repo --target 90

While working on the harness itself, dotnet run --project src/CoverageHarness -- <options> runs it from source and dotnet test runs the unit tests. The rest of this README writes the command as coverage-harness.

How it works

Each repo goes through the same sequence of phases. A run can be stopped at any point and restarted, and each phase works out from the repo whether it still has anything to do.

flowchart TD
    discover[Find repos and write the manifest] --> prepare[Check the tree is clean and switch to the harness branch]
    prepare --> hasTests{Is there a test project?}
    hasTests -- no --> createProject[Agent creates one]
    createProject --> baseline
    hasTests -- yes --> baseline[Build, test and measure the baseline]
    baseline --> hasComponent{Are there component tests?}
    hasComponent -- no --> scaffold[Agent scaffolds a component test project]
    scaffold --> triage
    hasComponent -- yes --> triage[Agent plans which gaps to tackle first]
    triage --> iterate[Coverage iteration]
    iterate --> atTarget{Branch coverage at target?}
    atTarget -- no --> iterate
    atTarget -- yes --> simplify{"--simplify?"}
    simplify -- yes --> prune[Agent removes redundant tests]
    prune --> mutation
    simplify -- no --> mutation{"--mutation?"}
    mutation -- yes --> stryker[Agent sets up Stryker, then kills surviving mutants until the score is met]
    stryker --> summary
    mutation -- no --> summary[Write the pull request summary]
    summary --> complete[Mark the repo complete in the manifest]
Loading

Every iteration, whether it's raising coverage, pruning tests or killing mutants, follows the same pattern:

flowchart LR
    ask[Send the agent the gaps] --> guard[Revert any change outside the tests]
    guard --> measure[Build, test and measure]
    measure -- tests fail --> fix[Ask the agent to fix them]
    fix --> measure
    measure -- still failing after the fix attempts --> reset[Reset to the last commit]
    measure -- number went down --> reset
    measure -- number went up --> commit[Commit]
Loading

A repo that stops improving for --stall-limit iterations in a row, or runs out of iterations, is marked stalled.

What the harness does itself

File paths are relative to src/CoverageHarness/.

Concern Where Notes
Repos and manifest Manifest.cs --src dir/* means every git repo directly inside dir. The manifest defaults to <dir>/coverage-harness.manifest.json and is written atomically, so parallel workers can share it.
Finding the solution Dotnet/Projects.cs Picks the shallowest .sln or .slnx, preferring one named after the repo, then the one with the most projects. Test projects are recognised from their project files.
Build and test Dotnet/DotnetCli.cs dotnet build, then dotnet test --no-build --collect "Code Coverage" --logger trx with a runsettings file the harness owns, so the agent can't change what's excluded. Coverage comes from the Microsoft collector in Microsoft.NET.Test.Sdk, so no extra package is needed.
Coverage Coverage/Cobertura.cs Reports from every test project are merged line by line. The target is branch coverage, with statement coverage reported alongside it. Code with no branches falls back to statement coverage. Program.cs and Main in executable projects don't count towards the target, though the raw figure is still calculated. Gaps are grouped by method, with async state machines mapped back to the method they came from.
Test results Dotnet/Trx.cs Reads TRX files (by element name, so any test framework's output works) for failing tests, their messages and stack traces, and the list of tests that ran.
Mutation testing Dotnet/Stryker.cs dotnet tool restore and dotnet stryker --config-file stryker-config.json --reporter json. The score is (killed + timeout) / (killed + timeout + survived + no coverage).
Keeping to the rules Guard.cs After every agent turn, any change outside the test projects is reverted and the agent is told what was undone. A few changes are allowed: public partial class Program { }, InternalsVisibleTo, solution files, Directory.Packages.props, .config/dotnet-tools.json and a root stryker-config.*.
Git Git.cs Work happens on the coverage-harness branch, created from whatever was checked out. The harness commits each successful iteration and resets failed ones to the last good commit. Its scratch files are hidden through .git/info/exclude, so the repo's .gitignore isn't touched. The harness never pushes.

The files the agent works from are written to .coverage-harness/ in each repo: coverage-gaps.md, coverage-summary.json, failing-tests.md, build-errors.md and surviving-mutants.md/.json. The logs/ folder there keeps every prompt, agent transcript and dotnet or Stryker log.

Talking to the agent

Each task is one agent session (RunTask in RepoRunner.cs):

  1. Render the prompt and send it to the CLI in non-interactive mode, with permission prompts turned off.
  2. Look for the <harness-report>{json}</harness-report> block every prompt asks the agent to finish with.
  3. If there's no report, resume the same session with continue.md. If the agent asks a question ("status": "needs-input"), resume with answer.md, which tells it how to decide for itself. It gets up to two follow-ups.
  4. The harness then measures the result itself, whatever the agent said. The report is only used for notes in the manifest, such as code the agent thinks is unreachable or mutants it thinks are equivalent.

Agents run with a wall-clock timeout and an idle timeout: if a CLI prints nothing for that long, its process group is killed. Three agent crashes in a row with no output mark the repo as error, so a missing login doesn't waste hours.

Agent adapters (Agents/)

Agent Command Output parsed Follow-ups
claude claude -p --dangerously-skip-permissions --output-format stream-json --verbose, prompt on stdin assistant and result events --resume <session_id>
opencode opencode run --auto --format json [-m provider/model] <prompt> text and tool_use events -s <sessionID>
copilot copilot -p <prompt> --yolo --no-ask-user --autopilot --output-format json --session-id=<uuid> assistant.message and result events --resume=<uuid>

To add another CLI, implement IAgentAdapter in a new file and register it in AgentCli.Create in Agents/Agent.cs.

Prompts (prompts/)

File Sent when
create-test-project.md the solution has no test project
scaffold-component-tests.md there's no component test project yet. The agent maps the app's external edges and builds the WebApplicationFactory fixture, the stubs, and a README describing them
triage-coverage.md once, before the coverage loop. The agent ranks the gaps by how many branches each is worth for the effort and writes a plan to .coverage-harness/coverage-plan.md, quick wins first. It can't change code in this step. The plan is kept between runs; delete it to plan again
raise-coverage.md each coverage iteration, with the largest gaps, the plan and the outcome of earlier iterations
fix-build.md, fix-tests.md the build or tests fail after an agent turn, or at the baseline
simplify-tests.md each pass of --simplify
setup-stryker.md with --mutation, when there's no Stryker tool manifest or root config
kill-mutants.md each mutation iteration, with the surviving mutants
write-pr-summary.md once, when a repo completes, to describe each change and each new test
continue.md, answer.md follow-ups within a session
_rules.md, _testing-standards.md, _report.md, _generated-tests-*.md shared sections included in the prompts above

_testing-standards.md sets out how tests should be written. Apps are hosted in memory with WebApplicationFactory. Dependencies are replaced at the smallest seam the DI container offers: the HttpMessageHandler for outbound HTTP, in-memory SQLite for relational databases, and in-memory stubs for document stores and queues. Stubs live in the test project, are registered in ConfigureTestServices each time a host starts, and every test checks how they were called.

To change a prompt, pass --prompts <dir> with a file of the same name. Templates use {{var}} for values and {{> name}} to include _name.md. Rendering fails if a value is missing.

CLI

coverage-harness [run] --src <path> --target <percent> [options]
coverage-harness discover --src <path>      # write or refresh the manifest only
coverage-harness measure --src <repo>       # build, test and measure, without an agent
coverage-harness status --manifest <file>

coverage-harness --help lists every option, and coverage-harness <command> --help shows one command. Options can go before or after the command. Unknown options and invalid values are rejected before anything runs, with exit code 2.

Option Default
--src /abs/dir/* for every git repo in a folder, or /abs/repo for one
--target branch coverage to reach, as a percentage
--agent claude claude, opencode or copilot
--model the CLI's default passed to the agent CLI
--skip-triage off skip the planning step and take the largest gaps first
--simplify off once coverage is met, remove tests the component tests make redundant, without losing branch coverage
--max-simplify-iterations 10 simplification passes per repo
--identify-generated[=file|attribute] off mark the tests the harness writes so you can tell them from your own. On its own the flag means file. See Telling generated tests apart
--mutation off once coverage is met, run mutation testing with Stryker. This can take hours on a large codebase
--mutation-target 80 mutation score to reach
--concurrency 1 repos processed in parallel
--retry off also re-run repos marked blocked, stalled or error
--max-iterations, --max-mutation-iterations 40, 20 iterations per repo
--stall-limit 3 iterations in a row without progress before a repo is stalled
--fix-attempts 3 attempts to fix a failing build or test run before reverting
--agent-timeout, --agent-idle-timeout 90, 20 minutes
--test-timeout, --stryker-timeout 30, 240 minutes
--stryker-concurrency half the cores parallel test runners for Stryker, overriding concurrency in stryker-config.json
--stryker-shards 1 split mutation testing into this many shards by directory, largest first. Each finished shard is saved and reused if the run is interrupted or retried, and the reports are merged. --stryker-timeout applies to each shard
--stryker-exclude extra directory or file names not to mutate, on top of Models, Configuration, HealthChecks, Migrations, Program.cs, obj and bin
--heartbeat 5 minutes how often long steps log that they're still running (0 turns it off)
--verbose off also print agent activity to the console. It always goes to the log files
--env-file ./.env settings file, see below

Settings from the environment or .env

Every option can also be set as COVERAGE_HARNESS_<OPTION>, with hyphens as underscores: COVERAGE_HARNESS_MAX_ITERATIONS=60, COVERAGE_HARNESS_MUTATION=true. ./.env is loaded if it exists, or you can pass --env-file <path>. A command-line flag wins over an environment variable, which wins over the .env file. Any other variables in the file, such as API keys, are passed through to the agent CLIs. See .env.example.

Telling generated tests apart

By default the agent writes tests wherever they fit. --identify-generated makes them easy to spot in review and later on.

With --identify-generated or --identify-generated=file, each class gets at most one .generated.cs file:

  • When the agent adds tests to an existing class, say OrderTests in OrderTests.cs, it makes the class partial and puts the new tests in OrderTests.generated.cs. Adding partial is the only change it makes to your file.
  • When it creates a new class, whether a test class, stub or fixture, the whole class goes in <ClassName>.generated.cs and isn't marked partial.

With --identify-generated=attribute, tests go in ordinary files, either in existing classes or in new ones, and every test the agent adds is tagged with a Generated category: [Trait("Category", "Generated")] in xUnit, [Category("Generated")] in NUnit and TUnit, [TestCategory("Generated")] in MSTest. You can then run or skip them with dotnet test --filter Category=Generated or Category!=Generated. It's a per-test category rather than an xUnit [Collection], because a collection changes how tests share fixtures and run in parallel, and a class can only belong to one.

The instructions for each mode are in prompts/_generated-tests-file.md and prompts/_generated-tests-attribute.md. The harness doesn't check that the agent followed them.

Pull request summary

When a repo completes, the harness writes <repo>/.coverage-harness/pr-summary.md for you to paste into the pull request for the harness branch. Its path is also saved in the manifest as prSummary. It contains:

  • coverage, test count and, with --mutation, mutation score, before and after;
  • each file changed outside the test projects, such as Program.cs or the solution file, and each new test project, with a sentence on what changed and why;
  • each test added, grouped by class with its file, with a sentence on the behaviour it checks;
  • the tests removed, if --simplify removed any.

The harness works out the lists itself. It records the commit the branch started from (baseSha in the manifest) and the tests that existed at the baseline (.coverage-harness/baseline-tests.json), then runs the tests one last time to see which are new. Theory cases count as one test. The agent only writes the sentences, and any change it makes to the code in that step is thrown away. If it misses a test, that test is still listed, marked no description generated. If the summary can't be written, the harness logs a warning and the repo still completes.

Simplifying the suite (--simplify)

After the coverage loop, the agent removes tests that the component tests make redundant. A unit test is kept only if it covers complex logic that's much cheaper to test directly, such as calculations, parsers or validation rules with many combinations, or code that can't be reached from outside. If a unit test covers branches no component test reaches, the agent writes a component test that does before removing it.

The harness re-measures after each pass and reverts it if fewer branches are covered. A pass is only committed if the suite got smaller. The loop ends when a pass removes nothing, after --stall-limit reverted passes in a row, or at --max-simplify-iterations, and it never fails the repo. It runs before mutation testing, so Stryker has fewer tests to run.

Stopping before the target

Sometimes the last few percent cost far more than they're worth. The coverage and mutation prompts let the agent finish a session with "status": "diminishing-returns" and a reason. The harness still measures and commits that session's work, but it only ends the loop if the agent says so in two iterations in a row and statement coverage has reached the target. Otherwise the loop carries on. When the loop does end this way, the repo moves on to the next phase and finishes complete below the target, rather than running until it's marked stalled.

Watching a long run

Everything is logged to files, with or without --verbose:

File Contents
<manifest dir>/coverage-harness.log A readable log for the whole run: phases, iterations with their timings and outcomes, each agent turn and the tools it used, measurements, commits, resets, reverted changes and heartbeats. Good for tail -f.
<manifest dir>/coverage-harness.events.jsonl The same run as JSON events: run.start, repo.end, iteration.end, agent.end, measure, commit, reset, guard.revert, stryker, pr-summary, heartbeat and so on.
<repo>/.coverage-harness/harness.log, events.jsonl The same, for one repo.
<repo>/.coverage-harness/logs/ Each prompt sent (*.prompt.md), each agent transcript (*.agent.log) and the dotnet and Stryker output.

Stryker progress is logged every 10%, for example Stryker tested 1200/4484 mutants (27%) ... ETA ~1h10m. Heartbeats show the agent's last action and how long ago it was, which makes a stuck agent easy to spot:

claude raise-coverage (coverage-7) still running (25m00s elapsed; last activity 12s ago: Bash dotnet test)

Some useful queries:

tail -f repos/coverage-harness.log
jq -c 'select(.type=="iteration.end") | {repo, phase, iteration, outcome, durationMs}' repos/coverage-harness.events.jsonl
jq -c 'select(.type=="measure" and .result=="green") | {repo, label, coverage}' repos/coverage-harness.events.jsonl

Where the agent CLI reports usage, it's logged after every turn and totalled per repo in the manifest and the status table. Claude Code reports cost and tokens, OpenCode reports cost and tokens per step, and Copilot reports premium requests only.

To stop a run, press Ctrl-C. The agent's processes are killed and the manifest is saved, and running the same command again carries on from where it stopped. Uncommitted work left on the harness branch is stashed, never deleted, so you can get it back with git stash list.

Manifest

{
  "target": 100, "mutationTarget": 80, "agent": "claude",
  "repos": [{
    "name": "OrdersApi", "path": "/repos/OrdersApi", "status": "complete", "phase": "done",
    "solution": "OrdersApi.sln", "branch": "coverage-harness", "baseSha": "6348c14...",
    "baselineCoverage": 0, "coverage": 100, "statementCoverage": 100, "mutationScore": 86.2,
    "iterations": { "coverage": 3, "mutation": 2 }, "agentInvocations": 9,
    "prSummary": "/repos/OrdersApi/.coverage-harness/pr-summary.md",
    "completedAt": "...", "notes": ["..."],
    "history": [{ "phase": "coverage", "iteration": 1, "outcome": "committed: branches 0% → 71.4%", "coverage": 71.4 }]
  }]
}

A repo's status is one of:

  • pending
  • in-progress
  • complete
  • stalled: no progress, or out of iterations
  • blocked: something the agent couldn't fix, such as uncommitted changes, a production build that fails, or a test project, scaffold or Stryker setup that didn't work
  • skipped: no .NET projects
  • error

Limitations

  • Equivalent mutants make a 100% mutation score unrealistic, so set --mutation-target with that in mind. Mutants the agent reports as equivalent are noted in the manifest but still count against the score.
  • Solutions that use the Microsoft.Testing.Platform mode of dotnet test (set in global.json) are built with -p:UseMicrosoftTestingPlatformRunner=true, and each test project's executable is run directly, because dotnet test can report that no tests ran.
  • The harness works on a branch in your existing clone, so don't use those clones while it's running.
  • OpenCode retries forever if its provider can't be reached. The idle timeout catches this, but pass --model provider/model if your default provider is a local server.

About

A meta-harness to drive up test coverage in poorly tended to codebases

Resources

Stars

5 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages