Repair Stryker.NET backend mutation lane - #1503
Conversation
Signed-off-by: Chris0Jeky <jeky.tck@gmail.com>
Signed-off-by: Chris0Jeky <jeky.tck@gmail.com>
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6c9ccb5dfb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Chris0Jeky <jeky.tck@gmail.com>
Signed-off-by: Chris0Jeky <jeky.tck@gmail.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 307add004f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Chris0Jeky <jeky.tck@gmail.com>
Final exact-head hosted proof —
|
|
Codex Review: Didn't find any major issues. Another round soon, please! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Chris0Jeky
left a comment
Chris0Jeky
left a comment
There was a problem hiding this comment.
Independent adversarial evidence/policy review — exact head 89e5f9b1ce86823ae155e5fad25e0fae54cab2d5
I read the linked issue/acceptance criteria, full diff, all issue comments, both prior Codex reviews, every inline comment, and all three unresolved review threads before reviewing this head. Lens: canonical STATUS/MASTERPLAN/TESTING/policy truth, append-only failure-ledger resolution, exact run/SHA/count/duration/artifact evidence, backward compatibility, and single-baseline claim boundaries.
MEDIUM
-
The preflight rejects the intentional exclusions the policy tells maintainers to add.
scripts/ci/Test-StrykerConfig.ps1:83-91requiresignore-methodsandignore-mutationsto stay empty, whiledocs/testing/MUTATION_TESTING_POLICY.md:118-120explicitly makes those supported spellings the backend suppression mechanism. That means the first legitimate exclusion re-breaks the lane before Stryker starts. Confirmed existing thread: #1503 (comment). Accept arrays of valid string entries rather than enforcing emptiness, and add a positive self-test proving non-empty supported lists pass (plus malformed-entry rejection if the preflight owns entry validation). -
The durable manual checklist still invokes the removed backend-root contract.
docs/MANUAL_TEST_CHECKLIST.md:1050-1057still tells an operator to rundotnet stryker --config-file stryker-config.jsonfrombackend/. This PR deliberately removessolution/test-projectsand makesbackend/tests/Taskdeck.Domain.Teststhe execution context, so that checklist can be checked off without exercising the repaired lane (or can fail for unrelated discovery/path reasons). Confirmed existing thread: #1503 (comment). Mirror the preflight plus Domain-test-project invocation already documented inTESTING_GUIDE/policy. -
The PR's exact-head evidence section is stale and does not yet record the issue AC that is now actually satisfied. The body still says run
30236307062at307add00is running and that ledger/canonical-doc updates remain outstanding. Live evidence now shows final run30236739913completed successfully on this exact head89e5f9b1...; its two-filestryker-net-reportartifact is 871,031 bytes with GitHub digestsha256:9618f8cf7e0c2547fcc2425ffe18889485430fe705ba8c822967f4627c5ceb12. The downloaded JSON contains 3,682 mutants: 2,351 killed, 575 survived, 3 timeout, 397 no-coverage, 317 ignored, and 39 compile-error (70.78% total score). Update the PR body/evidence comment to record this exact-head run and replace the now-false remaining-work text. This is metadata-only and need not invalidate the head.
LOW
- The mutation follow-up process names a label that does not exist.
docs/testing/MUTATION_TESTING_POLICY.md:115says to file findings withtest-hardening, but the live repository has only the relevanttestingandhardeninglabels, and the project automation contract requires real repository labels. Use the existing labels or seed/approve the taxonomy change explicitly; do not leave a non-executable instruction in the refreshed policy.
Existing-thread triage
- #1503 (comment) is fixed in this head: policy line 119 now uses
ignore-mutations/ignore-methodsand explicitly rejects the obsolete spellings. It still needs a reply with89e5f9b1evidence and thread resolution. - The other two Codex threads are confirmed above and remain unresolved.
Evidence and residual gate state
- The append-only failure ledger is correctly shaped: the original open record remains, a later resolution record was appended, the Markdown projection matches, and exact-head Docs Governance passed.
- The documented first baseline is accurate: run
30236307062, SHA307add00, 192-second workflow lifecycle, 3,682 mutants, 70.75%, one two-file artifact, 874,386-byte archive, digest0e8a9a41.... The canonical docs correctly keep this as calibration rather than a merge threshold. - The final exact-head run differs by one survivor/timeout (575/3 vs. 576/2) while total mutants stay 3,682, reinforcing that two back-to-back proofs are not the policy's 3–4-run calibration window.
- Required CI and CI Extended were still running when reviewed. CI Extended's Workflow Lint job is already red because
Build rhysd/actionlint@v1.7.12failed in Docker before checkout/actionlint execution; this is not evidence of a workflow syntax defect, but it remains a merge-gate failure that needs a terminal-log investigation and bounded rerun.
Verdict: changes requested. Do not merge; the PR is maintainer-held regardless.
Independent workflow/Stryker review ? exact head
|
Signed-off-by: Chris0Jeky <jeky.tck@gmail.com>
Combined review-fix evidence — exact head
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b18b9a5c54
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| Last Updated: 2026-07-27 | ||
|
|
||
| Backend mutation-testing runway repair (2026-07-27, `#1500`): | ||
| - **The scheduled Domain mutation lane now reaches a bounded, artifact-backed result instead of failing before execution.** Stryker.NET is pinned to 4.16.0; an 18-check PowerShell preflight rejects obsolete schema keys, wrong solution/test context, workflow timeout drift, and non-failing artifact uploads; and the workflow runs from `Taskdeck.Domain.Tests` with a 180-minute ceiling. Backend-only run [30236307062](https://github.com/Chris0Jeky/Taskdeck/actions/runs/30236307062) on exact workflow head `307add004fbe142321a6ec11be21fab708824d5d` completed in 192 seconds: 3,682 mutants created, 2,351 killed, 576 survived, 2 timed out, 753 skipped, score 70.75%. The non-empty two-file `stryker-net-report` artifact is 874,386 bytes (SHA-256 `0e8a9a41b8cd484b6c267bd914c57cda0ffa973f59d8989e89038157605f21c8`). The score remains a non-blocking calibration signal; PR `#1503` and merge remain maintainer-held because the workflow changes operational CI posture. |
There was a problem hiding this comment.
The final self-test now counts 4 valid contracts plus 21 rejected fixtures (Test-StrykerConfig.ps1:311-313), so this new source-of-truth entry's “18-check” claim is stale; the same incorrect count appears in docs/IMPLEMENTATION_MASTERPLAN.md:15. Update both canonical documents to 25 so their stated verification coverage matches the exact-head implementation.
AGENTS.md reference: AGENTS.md:L113-L118
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Valid P2. Exact head now runs 25 checks (4 valid / 21 rejected), while these two canonical summaries still say 18. The factual wording will be fixed in the next signed #1503 head, then docs/golden/preflight gates will be rerun and this thread resolved. That push is intentionally held until #1510 removes the repeatedly failing Workflow Lint bootstrap so the changed head can receive meaningful exact-head CI Extended proof rather than another identical pre-checkout failure.
Final exact-head mutation proof —
|
Exact-head gate classificationRequired CI is now terminal green at CI Extended run Per the two-strike runway rule, #1510 now owns replacement of the Docker Hub bootstrap with a checksum-pinned Actionlint release install. This PR is project-Blocked until #1510 lands and this exact head is re-proved through CI Extended. No further mutation retry will be spent on the unchanged bootstrap. |
Signed-off-by: Chris0Jeky <jeky.tck@gmail.com> # Conflicts: # docs/IMPLEMENTATION_MASTERPLAN.md # docs/STATUS.md # docs/agentic/FAILURE_LEDGER.md # docs/agentic/failure_ledger.jsonl
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: beeb2529f0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| $thresholds.high -ne 80 -or | ||
| $thresholds.low -ne 60 -or | ||
| $thresholds.break -ne 0) { |
There was a problem hiding this comment.
When a threshold is accidentally quoted (for example, "high": "80") or replaced with a matching boolean, PowerShell's non-strict -ne comparisons coerce the expected number to the JSON value's type, allowing malformed thresholds to pass this supposedly fail-closed preflight. Stryker's threshold schema requires numeric values, so the subsequent mutation run can still reject the configuration and produce no evidence; validate the threshold value types explicitly and add quoted/boolean rejection fixtures.
Useful? React with 👍 / 👎.
| cd backend | ||
| dotnet stryker --config-file stryker-config.json | ||
| # Install the workflow-pinned Stryker.NET version as a global tool (once). | ||
| dotnet tool install --global dotnet-stryker --version 4.16.0 |
There was a problem hiding this comment.
When a contributor previously followed this policy and already has another global dotnet-stryker version installed, dotnet tool install --global ... --version 4.16.0 does not replace that installation, so the documented local run either stops at installation or continues using the unpinned version that this change is intended to avoid. The .NET CLI provides dotnet tool update --global <PACKAGE_NAME> for changing an installed tool (Microsoft CLI documentation); make this sequence handle both fresh and existing installations.
Useful? React with 👍 / 👎.
| $count = 0 | ||
| $searchIndex = 0 | ||
| while ($true) { | ||
| $matchIndex = $Block.IndexOf($ExpectedFragment, $searchIndex, [System.StringComparison]::Ordinal) |
There was a problem hiding this comment.
When the Stryker command is extended with an extra option such as --mutate SomeFile.cs or a shell operator such as || true, this IndexOf check still finds the expected command as a substring and the separate invocation counter still reports one command, so the preflight accepts changed mutation scope or failure semantics. The same behavior permits trailing step keys after the artifact fragment, including continue-on-error: true, which can defeat the advertised fail-closed upload; compare complete lines/steps or parse the YAML rather than accepting unanchored fragments.
Useful? React with 👍 / 👎.
1 participant
Closes #1500
Summary
ignore-methods/ignore-mutationskeysTaskdeck.Domain.Testscontext so unrelated test projects are not discoveredLocal verification — current head
Hosted evidence
307add004fbe142321a6ec11be21fab708824d5d, 3,682 mutants, 70.75%, 192s, non-empty 874,386-byte two-file artifact89e5f9b1ce86823ae155e5fad25e0fae54cab2d5, 3,682 mutants, 70.78%, 198s, non-empty 871,031-byte two-file artifactb18b9a5c54d7337fe4ba6aaf1eaa134896e63439, preflight 25/25, 3,682 mutants, 70.75%, 292s overall, non-empty 878,708-byte two-file artifact (SHA-256310e5aa214bdfc72a54c8971b625cf93779096c8d1b2622ae830595258c5fb63)The one-mutant survived/timeout classification shift between the two completed runs is disclosed timing variance; both reports are valid and the killed count is unchanged.
Review resolution
testingandhardeninglabelsRemaining gate
Merge boundary
Maintainer-merge only, per #1500. This PR must not be self-merged.