Feat: run drafted E2E scenarios through a declared executor with fixtures - #216
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
QAMap already answers what to verify and drafts E2E scenarios, but verification still restarted from scratch after that hand-off: hand-built fixtures, step-by-step browser driving, screenshots judged by eye. This adds
qamap e2e run <scenario-id>: it resolves one compiled scenario to its draft, materializes the fixtures the repository declared for that scenario, invokes the executor the repository configured, and returns a receipt with pass/fail per assertion, timing, and failure-only artifacts. Receipts persist under.qamap/runs/e2e, so rerunning the same id after a fix compares receipts instead of re-driving the flow.qamap qanow marks which drafted scenarios are executable.Behavioral Contract
qamap.config.jsongainsexecutors(runnerplaywrightorcommand, an argument-vectorcommandrun without a shell that must reference{file}and may use{grep},{scenarioId},{fixtureDir},{artifactDir}),fixtures(filecopies a repository file;seedruns a hook without a shell; paths must stay inside the repository), andscenarioFixtures(scenario id to fixture ids). Invalid shapes fail config loading with a specific message.qamap qaattachesexecutable(executable,executor-missing,fixtures-missing,not-compiled) to each scenario automation receipt when executors are configured, lists executable scenarios under "Executable Evidence Available Now" asqamap e2e run <id>(explicitly not run byqa), and the compact agent payload carriesexecutable: true.qamap e2e run <scenario-id>accepts the full id or a unique hash prefix, regenerates the draft index, and returnsblocked— nothing executed — when the scenario is not found, not compiled, the draft file is not written, no executor is configured, or any declared fixture fails to materialize. Otherwise it runs the executor withQAMAP_SCENARIO_ID,QAMAP_FIXTURE_DIR,QAMAP_ARTIFACT_DIR, andPLAYWRIGHT_JSON_OUTPUT_NAMEset, parses the Playwright JSON reporter into per-assertion results (exit code only forcommandexecutors), keeps artifacts only when the run failed, hashes stdout/stderr instead of storing them, records git-observable worktree changes, writes the receipt, and compares it with the previous receipt for the same id (same,regressed,recovered,changed). Exit codes: passed 0, failed 1, blocked 2.e2e plan,setup, anddraftare unchanged.Evidence
Closes #210.
web-repeated-action-guardinexecution-bench.config.jsonnow also runs the compiled "Duplicate renewal request" scenario throughqamap e2e runwith a real Playwright executor —failedon the seeded regression,passedon the fix, comparison verdictrecovered. The scenario is resolved from the committed range so its id stays stable while the overlays change the working tree the executor runs against.test/e2e-run.test.mjs): a Playwright-shaped executor stand-in proves fixture materialization (file sha256, seed hook executed), per-assertion results with project names, failure-only artifacts, receipt persistence,regressed/recovered/samecomparisons, every blocked path (no executor, unknown id, missing fixture, fixture path escaping the repository, uncompiled scenario),qamap qaexecutable marking in markdown and the recoverable agent report, CLI json/markdown output with exit codes, and config validation errors. Negative control: a blocked run stores no receipt and reportsperformed: false.test/cli-help.test.mjspins the new usage line and the execution-boundary wording.Checks
pnpm testpnpm bench:cifor inference, routing, trace, or outputpnpm bench:executionfor E2E compiler or execution fixturespnpm scanfor scanner, security, or repository policypnpm plugin:checkandpnpm plugin:smokefor plugin changesPublic OSS Check
Review Notes
pnpm bench:context(10/10),pnpm plugin:check, andpnpm plugin:smokealso pass locally; plugin checks are marked N/A because no plugin asset changed. Maestro executors are out of scope for this change: the executor interface is runner-neutral, but only the Playwright JSON reporter is parsed today andcommandexecutors report the exit code as a single assertion. Scenario selection uses--grepon the scenario title, which matches how routed scenarios are named in drafts; primary-flow tests are not selected by id. Receipts store hashes of executor output, never the output itself, matching theqa runevidence boundary. Docs updated: commands, configuration fields and example, architecture execution boundary, README tables (en, ko), and anUnreleasedchangelog section that also records the two scanner additions merged since 0.4.14.