Skip to content

Feat: run drafted E2E scenarios through a declared executor with fixtures - #216

Merged
ivory-code merged 1 commit into
mainfrom
feat/e2e-scenario-executor
Aug 22, 2026
Merged

Feat: run drafted E2E scenarios through a declared executor with fixtures#216
ivory-code merged 1 commit into
mainfrom
feat/e2e-scenario-executor

Conversation

@ivory-code

Copy link
Copy Markdown
Contributor

Summary

QAMap already answers what to verify and drafts E2E scenarios, but verification still restarted from scratch after that hand-off: hand-built fixtures, step-by-step browser driving, screenshots judged by eye. This adds qamap e2e run <scenario-id>: it resolves one compiled scenario to its draft, materializes the fixtures the repository declared for that scenario, invokes the executor the repository configured, and returns a receipt with pass/fail per assertion, timing, and failure-only artifacts. Receipts persist under .qamap/runs/e2e, so rerunning the same id after a fix compares receipts instead of re-driving the flow. qamap qa now marks which drafted scenarios are executable.

Behavioral Contract

  • qamap.config.json gains executors (runner playwright or command, an argument-vector command run without a shell that must reference {file} and may use {grep}, {scenarioId}, {fixtureDir}, {artifactDir}), fixtures (file copies a repository file; seed runs a hook without a shell; paths must stay inside the repository), and scenarioFixtures (scenario id to fixture ids). Invalid shapes fail config loading with a specific message.
  • qamap qa attaches executable (executable, executor-missing, fixtures-missing, not-compiled) to each scenario automation receipt when executors are configured, lists executable scenarios under "Executable Evidence Available Now" as qamap e2e run <id> (explicitly not run by qa), and the compact agent payload carries executable: true.
  • qamap e2e run <scenario-id> accepts the full id or a unique hash prefix, regenerates the draft index, and returns blocked — nothing executed — when the scenario is not found, not compiled, the draft file is not written, no executor is configured, or any declared fixture fails to materialize. Otherwise it runs the executor with QAMAP_SCENARIO_ID, QAMAP_FIXTURE_DIR, QAMAP_ARTIFACT_DIR, and PLAYWRIGHT_JSON_OUTPUT_NAME set, parses the Playwright JSON reporter into per-assertion results (exit code only for command executors), keeps artifacts only when the run failed, hashes stdout/stderr instead of storing them, records git-observable worktree changes, writes the receipt, and compares it with the previous receipt for the same id (same, regressed, recovered, changed). Exit codes: passed 0, failed 1, blocked 2.
  • Scenarios without a configured executor keep today's draft-only behavior; e2e plan, setup, and draft are unchanged.

Evidence

Closes #210.

  • Execution contract: web-repeated-action-guard in execution-bench.config.json now also runs the compiled "Duplicate renewal request" scenario through qamap e2e run with a real Playwright executor — failed on the seeded regression, passed on the fix, comparison verdict recovered. The scenario is resolved from the committed range so its id stays stable while the overlays change the working tree the executor runs against.
  • Focused tests (test/e2e-run.test.mjs): a Playwright-shaped executor stand-in proves fixture materialization (file sha256, seed hook executed), per-assertion results with project names, failure-only artifacts, receipt persistence, regressed/recovered/same comparisons, every blocked path (no executor, unknown id, missing fixture, fixture path escaping the repository, uncompiled scenario), qamap qa executable marking in markdown and the recoverable agent report, CLI json/markdown output with exit codes, and config validation errors. Negative control: a blocked run stores no receipt and reports performed: false.
  • test/cli-help.test.mjs pins the new usage line and the execution-boundary wording.

Checks

  • Focused regression test
  • pnpm test
  • pnpm bench:ci for inference, routing, trace, or output
  • pnpm bench:execution for E2E compiler or execution fixtures
  • pnpm scan for scanner, security, or repository policy
  • pnpm plugin:check and pnpm plugin:smoke for plugin changes
  • Documentation links and commands verified

Public OSS Check

  • No private repository, source, path, customer data, credential, or internal smoke output is included.
  • Shared inference has unrelated positive and negative coverage, or this is not applicable.
  • User-facing commands and claims match actual behavior.

Review Notes

pnpm bench:context (10/10), pnpm plugin:check, and pnpm plugin:smoke also pass locally; plugin checks are marked N/A because no plugin asset changed. Maestro executors are out of scope for this change: the executor interface is runner-neutral, but only the Playwright JSON reporter is parsed today and command executors report the exit code as a single assertion. Scenario selection uses --grep on the scenario title, which matches how routed scenarios are named in drafts; primary-flow tests are not selected by id. Receipts store hashes of executor output, never the output itself, matching the qa run evidence boundary. Docs updated: commands, configuration fields and example, architecture execution boundary, README tables (en, ko), and an Unreleased changelog section that also records the two scanner additions merged since 0.4.14.

@ivory-code ivory-code added type: feat Feature work that adds product or CLI capability area: e2e E2E planning, draft generation, and runner support labels Aug 22, 2026
@ivory-code ivory-code self-assigned this Aug 22, 2026
@ivory-code ivory-code added the area: agent-readiness Agent instructions, readiness, and workspace hygiene label Aug 22, 2026
@ivory-code
ivory-code merged commit ba35dcd into main Aug 22, 2026
2 checks passed
@ivory-code
ivory-code deleted the feat/e2e-scenario-executor branch August 22, 2026 11:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: agent-readiness Agent instructions, readiness, and workspace hygiene area: e2e E2E planning, draft generation, and runner support type: feat Feature work that adds product or CLI capability

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feat: run drafted E2E scenarios through a pluggable executor with declared fixtures

1 participant