Repository navigation
ci(mutants): run the lane once a night on main, never on a push - #24
Open
systemfsoftware-maker wants to merge 5 commits into
Open
systemfsoftware-maker wants to merge 5 commits into
systemfsoftware-maker wants to merge 5 commits into
Conversation
The Mutants lane must stop running on every push to main (ci-wall-clock/mutation-runs-nightly-on-main, enrolled in #23). The plan covers the schedule and dispatch inputs, the main-only guard, per-run concurrency, shards sized from measured timings, and the night record and fix task the lane writes itself while no triage exists.
Mutants now runs on a 03:17 UTC schedule and on workflow_dispatch, as ci-wall-clock/mutation-runs-nightly-on-main requires, and no longer on every push to main. - A guard job refuses any ref but the default branch, a sha main does not reach and a malformed night id before anything installs, and every other job needs it. - Each run is its own concurrency group, so a dispatch and the night's run never replace each other. - Dispatch takes sha, packages and night; the run is titled "Mutants <sha>" plus " for <night>". - The plan sizes shards from the newest measured run's per-file mutant times and per-shard overhead, so each job aims at 10 minutes. - Shards restore main's Nix store bundle and take their tools from the new gritlint-mutants-tools package instead of the dev shell. - A night tests the diff from the last green night, or everything. - The report job writes the night's record artifact, files or updates the fix task on a red night, and ends red; a green night writes one line. - Every job writes its wall time and, on failure, a reason code and a reproduce command.
…flow, bound a red night's candidates From the full-path review of the nightly lane: - The night record upload overwrites its own name, so a re-run's report writes the record instead of failing on a name the first attempt took. - The jobs, artifacts and open-issue reads follow every page, so a night past 100 shards still sizes, reports and routes from all of them. - The green-night lookup moves from mutants-night.sh into the plan job, which hands the green record to the report job in the plan artifact; the scripts again hold only decisions over files. - With no green night, a red night names the 20 newest first-parent commits as candidates instead of main's whole history, and one failed pulls lookup leaves that commit unattributed instead of failing the record. - A malformed artifacts body fails the plan instead of being read as a run with no shard artifacts. - mutants-night.sh tasks selects the open fix tasks; a test files a task body and routes the next night to it. - flake.nix and rust-toolchain.toml join the full-run triggers, since the lane's tools are built from them. - The shard's outcome and upload steps carry ids and reason codes.
verdict writes the record route reads; one case now feeds the first into the second, so a renamed record key fails the gritlint-mutants-plan check.
…candidate bound From the cross-model pass over the review fixes: - The plan job publishes the green night's sha and id as outputs as soon as it finds them, so a report whose plan failed later still names the range from that night instead of "no green night yet". - With no green night, the fix task says that only the newest 20 first-parent commits are named and older merges are not; plan R8 says the same. - The decision table runs the full-night case for flake.nix and rust-toolchain.toml, and plan R5 lists them as lane inputs.
systemfsoftware-maker
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Mutants lane stops running on every push to
main. It now runs once a night onmainat 03:17 UTC (cron: "17 3 * * *") and onworkflow_dispatch. A red night opens a fix task. This followsci-wall-clock/mutation-runs-nightly-on-main, which #23 enrolled. Plan:docs/plans/2026-10-13-2100-ci-mutants-nightly-plan.md.What changes
scheduleandworkflow_dispatchonly. Thepushtrigger is gone. The rulenever on a pull requestfrom R21 of the standalone plan still holds. The run name isMutants <sha>, plusfor <night>on a re-test.sha(a commitmainreaches, 40 hex digits; empty meansmain's tip),packages(a JSON array of package names; empty means every package) andnight(the run id a re-test is for; a re-test resumes nothing). The oldmodeandbaseinputs are gone.guardjob refuses any ref that is not the default branch (mutants-off-main), a malformed or unreachablesha(mutants-sha-invalid,mutants-sha-off-main) and a malformednight(mutants-night-invalid) before anything installs. Every other job needs it.mutants-${{ github.run_id }},cancel-in-progress: false). The old setup keyed groups by commit and ref, so with GitHub's defaultqueue: singlea third run in a group cancelled the one still pending, and two dispatches onmaincould replace each other.main's Nix store bundle and take their tools from the newpackages.x86_64-linux.gritlint-mutants-tools, not from a dev shell.mutants.yml,.cargo/mutants.toml, either script,flake.lock,flake.nixorrust-toolchain.toml.reportjob writes the record artifactmutants-night-<run id>itself. On a red night it updates the open fix task that already names a survivor or failure, files a new task for anything no task names (listing every merge in the range as a candidate, with the opener mentioned and a reproduce command), and ends red. A green night writes one line. The decisions live inscripts/tools/mutants-plan.shand the newscripts/tools/mutants-night.sh.checks.<system>.gritlint-mutants-planruns both decision tables.Proof before merge
checks.x86_64-linux.gritlint-mutants-planbuilds locally. On CI run 38473088412 the x86_64 job built it: 72oklines,all passedtwice.actionlint .github/workflows/mutants.ymlexit 0.shellcheck -S warning scripts/tools/mutants-*.shexit 0.cargo mutants --workspace --listthroughgritlint-mutants-toolslists 627 mutants, byte-identical to the dev shell's list.--shard k/3 --sharding round-robingiveslist[k::3]line for line.cargo test --workspace --no-run --offlinebuilds with only that package onPATH.5f88dbeac. The head shows no Mutants run. Times below.verdictgives red with caught 105, unviable 50, missed 10, timeout 21, complete.routewith no open task files the 10 survivors. The rendered task routes back to itself on the next night.No mutant was built or run locally.
cargo mutantsran only with--list.CI job times (jobs API, run 38473088412 and Changeset Check 38473088832)
Proof after merge
schedulerun after merge, the night of the merge day at 03:17 UTC: title,guardsuccess, plan summary, every shard's wall time, themutants-night-<id>artifact, then the task or the green lineworkflow_dispatchonmainruns, and one on a branch is refused byguardwithmutants-off-mainPOST …/actions/workflows/mutants.yml/dispatchesreturns HTTP 403Resource not accessible by personal access token.The first night tests everything (no green night yet). Its predicted price is in the workflow's cost header: 93 shards and 900 job-min from run 38447766912's timings, and 25 shards and 236 job-min if the store restore brings the overhead down to about 160 s.
Review receipt
Full-path
ce-code-review, run20261013-204731-a3bd48a6, base3919672c8(main, #23), reviewed headaf068a080. Eight reviewers ran: correctness, project-standards, testing, maintainability, learnings, reliability, security, and the cross-model adversarial peer. Every reviewer returned and none timed out. Turns served, from each reviewer's session transcript:No Grok or xai model served any turn. The body check filed two P3s (which head the Item column's line numbers refer to, and a shortened job name); both are fixed here. The peer ran as an omp subagent, not through CE's peer runner, so CE records
independence_verified: false, and its findings took no promotion bonus.Verdict. The review found five primary findings at
af068a080: one P1, three P2, one P3. The validator confirmed the P1. CE's report leaf wrote "Ready to merge", but an open P1 caps the verdict at "Ready with fixes" under CE's own rule, so the verdict ataf068a080is Ready with fixes. All five are fixed: #1 to #5 in74de64575. The delta pass checked all eight findings the reviewers filed and found each one fixed at74de64575. Its three P3 regressions are fixed in5f88dbeac.1be5a6a41adds a verdict-to-route test.Every finding, risk and gap, ruled
Each item is fixed by a named commit, rejected with its evidence, or given an owner and a trigger. Line numbers in the Item column are the reviewers', at the reviewed head
af068a080. Line numbers in the Ruling column are at the final head5f88dbeac.74de64575: both jobs reads paginate (mutants.yml:208-209, :417-418). Live check over run 38447766912 atper_page=10: all 23 jobs came back.night-tagged re-test dispatch, the plan's triage hook, writes no record and cannot end red:record,route,upload-recordandverdictare g…per_page=100(mutants.yml:158, 176) while a night may upload more than 99 shard art…74de64575: both artifacts reads paginate (mutants.yml:180-181, :199-200).routereads the open tasks withper_page=100(mutants.yml:428) while tasks are filed on every red night and the lane never closes them; past 100 o…74de64575: the open-issues read paginates (mutants.yml:466-467).artifacts: an API error body is a failure) is bypassed a…74de64575:$PLAN artifactswrites its own file and fails the step on a bad body. Only thegrepfor shard names maycontinue(mutants.yml:201-202).plan,shardandreportjobs:plan/args,plan/mutants.diff, the `mutants-plan-<a…ghglue needs a test harness over the GitHub API, which is a new CI gate; this repo admits a gate only with operator sign-off (GATE1). The glue is proved by the first scheduled night and the two dispatches after merge (plan V6, V7), which the post-merge comment cites by run id.verdictorroutefor a night whose shards uploaded artifacts from two run attempts at once, the shape a re-run attempt produces (`p…mergeunions every directory it is given and does not read names (mutants-plan.shmerge, covered by themergecases in mutants-plan.test.sh), so two attempts' shard directories merge like any two shards.74de64575:last_greenis gone from mutants-night.sh. The plan job makes theghcalls and calls the purenightsandgreen-record(mutants.yml:152-167).mutants-plan.sh(it holds the list), and that script is itself a trigger, so the rename night runs everything.ghglue needs a test harness over the GitHub API, which is a new CI gate; this repo admits a gate only with operator sign-off (GATE1). The glue is proved by the first scheduled night and the two dispatches after merge (plan V6, V7), which the post-merge comment cites by run id.ghglue needs a test harness over the GitHub API, which is a new CI gate; this repo admits a gate only with operator sign-off (GATE1). The glue is proved by the first scheduled night and the two dispatches after merge (plan V6, V7), which the post-merge comment cites by run id.74de64575by removal:last-greenno longer exists;nightsandgreen-recordhave decision tables.74de64575: a test files atask-bodyoutput, finds it withtasksand routes the next night to it (mutants-night.test.sh:126-133).ghglue needs a test harness over the GitHub API, which is a new CI gate; this repo admits a gate only with operator sign-off (GATE1). The glue is proved by the first scheduled night and the two dispatches after merge (plan V6, V7), which the post-merge comment cites by run id.verdict…1be5a6a41: one case feedsverdict's record intoroute(mutants-night.test.sh:135-141).74de64575(task-body's machine-read part is now tested). Rejected forcomment-body,mentions,reproduceandgreen-line: their output is prose for people, and a test of wording is banned here.ghglue needs a test harness over the GitHub API, which is a new CI gate; this repo admits a gate only with operator sign-off (GATE1). The glue is proved by the first scheduled night and the two dispatches after merge (plan V6, V7), which the post-merge comment cites by run id.1be5a6a41(same case as 15).74de64575: the marker lives only in mutants-night.sh:5, and the workflow calls$NIGHT tasks(mutants.yml:468).for commit in $(git log --first-parent --format=%H "${green:+$green..}$TARGET")degenerates togit log --first-parent $TARGETw…74de64575: with no green night, the record walks at most 20 commits (mutants.yml:448-452). Since5f88dbeac, the task also says that older merges are not named.Job reportsteps (guard, plan, shard, report) repeat identicalif: always()+STEPS/printf steps.json+$PLAN job-reportboilerplat…unclassified, never to a silent pass.ghglue needs a test harness over the GitHub API, which is a new CI gate; this repo admits a gate only with operator sign-off (GATE1). The glue is proved by the first scheduled night and the two dispatches after merge (plan V6, V7), which the post-merge comment cites by run id.ghglue needs a test harness over the GitHub API, which is a new CI gate; this repo admits a gate only with operator sign-off (GATE1). The glue is proved by the first scheduled night and the two dispatches after merge (plan V6, V7), which the post-merge comment cites by run id.74de64575: the cost header now prices the fallback at 105 shards and 1023 job-min (mutants.yml:10-11). That is whatmutants-plan.sh shardsprinted for the 627-mutant list.flake.lock. Trigger: that bump; the solution doc's Verification section names the list-only probe to re-run, and aflake.lockchange also forces a full night. An executable check would be a new gate (GATE1).74de64575(flake.nixandrust-toolchain.tomladded) and tested in5f88dbeac(mutants-plan.test.sh:90-91).74de64575: the shard's outcome and upload steps have ids and reason codes (mutants.yml:345, :348).unclassifiedand still names the step.timeout-minutes: 60is read in review; actionlint is clean.timings: nothing measured falls back to the labelled model).74de64575(same as 1 and 3).resume(mutants.yml:150-168) andtimings(173-188) steps run underset -euo pipefail, so one tra…mutants-plan-history), which the failures rule wants; falling back silently would turn one API error into a full night of about 900 job-min.mutants-night-${{ github.run_id }}(mutants.yml:410) whileresume_runsexplicitly handles `attempt…74de64575: the record upload setsoverwrite: true(mutants.yml:495). The pinned action's README says that without it, a name collision fails the upload.74de64575(same as 20).GITHUB_REF/DEFAULTvalues, the report job'stoJSON(needs)-> failures mapping, and the …ghglue needs a test harness over the GitHub API, which is a new CI gate; this repo admits a gate only with operator sign-off (GATE1). The glue is proved by the first scheduled night and the two dispatches after merge (plan V6, V7), which the post-merge comment cites by run id.timingshas no case where a succeeded shard is present in the jobs API but itsoutcomes.jsonis missing (a partially uploaded run); that path sile…outcomes.jsonwould lower the overhead estimate; the next run's jobs table marks any job over 600 s as over the target, so the miss is loud and the next night's timings correct it.main(nix copy --no-check-sigs: ci.yml:71, mutants.yml:133, :314). Since #17, only the default branch exports and saves that key (ci.yml:96, :107).scripts/tools/mutants-night.sh:135builds ` ($merges[…mainby users with write access, and GitHub renders issue bodies without scripts.run:script instead of routing it through an environm…env:(mutants.yml:60-61, :96-97), and review checks that.shard(.github/workflows/mutants.yml:255: ` if: nee…needs: guardwhen the guard fails, unless the job'sif:callsalways(). Onlyreportcalls it, and itsif:also requiresneeds.guard.result == 'success'(mutants.yml:373).74de64575(same as 34); the validator confirmed it ataf068a0.74de64575: a failed pulls lookup yields[], and that commit is left unattributed (mutants.yml:454).74de64575(same as 20)..../runs/$run/jobs?per_page=100) and mutants.yml:391 (`.../attempts/$GITHUB_RUN_ATTEMPT/jobs?pe…74de64575(same as 1).--list --jsonover this tree, through the new tools package, gives 627 objects, each withpackage: "gritlint_core", the shapepackagesreads.record's merges loop andlast-greentogether make up the report job's API budget: one call per first-parent commit plus up to 50 x 2 calls in last…74de64575: the candidate walk is at most 20 commits. The green lookup lists only nights that endedsuccess, and a red night's report fails its run, so the first unexpired record it reads is normally green.record/route(empty green => unboundedgit logand a task with no named range); the tests cove…ghglue needs a test harness over the GitHub API, which is a new CI gate; this repo admits a gate only with operator sign-off (GATE1). The glue is proved by the first scheduled night and the two dispatches after merge (plan V6, V7), which the post-merge comment cites by run id.recordorroutestep fails (transientgh apierror) to pin the intended behavior: unrouted red run vs. best-…ghglue needs a test harness over the GitHub API, which is a new CI gate; this repo admits a gate only with operator sign-off (GATE1). The glue is proved by the first scheduled night and the two dispatches after merge (plan V6, V7), which the post-merge comment cites by run id.5f88dbeac: the plan publishesgreenandgreen-nightas step outputs as soon as it finds them, and the report job builds a base record from them when the plan artifact is missing.5f88dbeac: the task says only the newest 20 are named, and plan R8 states the bound.5f88dbeac: the decision table covers both, and plan R5 lists them.Constitution treatment (16 laws,
CONSTITUTION.mdatde684e25, evidence ataf068a080)scripts/tools/mutants-plan.sh:37scripts/tools/mutants-plan.sh:35scripts/tools/mutants-night.sh:74scripts/tools/mutants-night.sh:164flake.nix:75flake.nix:142flake.nix:145.github/workflows/mutants.yml:17docs/plans/2026-10-13-2100-ci-mutants-nightly-plan.md:17docs/solutions/tooling-decisions/cargo-mutants-resume-and-shard-semantics.md:49flake.nix:145scripts/tools/mutants-plan.test.sh:155scripts/tools/mutants-night.test.sh:107scripts/tools/mutants-plan.sh:90docs/plans/2026-10-13-2100-ci-mutants-nightly-plan.md:45flake.nix:146Compound Pack rules (all 56 loaded rules, packs at
de684e25, evidence ataf068a080)11 apply and comply; 45 do not apply.
failures-name-the-gate-and-location.mdscripts/tools/mutants-plan.sh:235flake-outputs-build-once-on-main.md.github/workflows/mutants.yml:124gates-share-one-job.md.github/workflows/mutants.yml:17heavy-lanes-state-their-daily-minutes.md.github/workflows/mutants.yml:1heavy-suites-judge-every-main-commit.md.github/workflows/mutants.yml:35light-checks-run-the-pinned-tool.md.github/workflows/mutants.yml:19local-gate-equals-pr-gate.mdflake.nix:145mutation-runs-nightly-on-main.mdscripts/tools/mutants-plan.sh:33no-dev-shell-on-verdict-lanes.mdflake.nix:77no-lane-grades-the-repos-own-docs.md.github/workflows/mutants.yml:174one-stuck-job-backstop.md.github/workflows/mutants.yml:51pr-gate-runs-only-what-the-diff-affects.md.github/workflows/mutants.yml:255red-main-reverts-its-culprit-same-day.md.github/workflows/mutants.yml:347release-promotes-the-staging-proven-build.md.github/workflows/mutants.yml:43setup-under-a-minute.md.github/workflows/mutants.yml:16kill-survivors-with-contract-properties.mdscripts/tools/mutants-night.sh:58suppress-by-shape-in-an-ignorer.mdscripts/tools/mutants-plan.test.sh:175unkillable-survivors-are-dead-code.mdscripts/tools/mutants-night.sh:71expected-values-never-come-from-the-subjects-output.mdscripts/tools/mutants-night.test.sh:79fake-and-real-store-laws.mdscripts/tools/mutants-plan.sh:1no-mocks-on-internal-glue.mdscripts/tools/mutants-plan.sh:154pin-dependency-semantics.mdscripts/tools/mutants-plan.sh:154port-fakes-pass-the-port-contract.mdscripts/tools/mutants-plan.sh:1real-system-oracles.md.github/workflows/mutants.yml:174staged-protocol-evidence.mdscripts/tools/mutants-night.sh:1callable-vs-resource-syntax.mdflake.nix:75decode-never-cast.mdscripts/tools/mutants-plan.sh:1four-channel-contracts.mdscripts/tools/mutants-night.sh:1handle-state-privacy.md.github/workflows/mutants.yml:42pipeable-dual-parity.mdflake.nix:75pipeline-composition.mdscripts/tools/mutants-plan.sh:1pure-decision-workflows.mdscripts/tools/mutants-night.sh:1resource-vs-handle-duality.md.github/workflows/mutants.yml:42sandwich-phase-order.mdflake.nix:75scoped-lifecycle-boundaries.mdscripts/tools/mutants-plan.sh:1service-and-layer-boundaries.mdscripts/tools/mutants-night.sh:1single-namespace-barrel.md.github/workflows/mutants.yml:42staged-lawful-builders.mdflake.nix:75store-serializable-unit-of-work.mdscripts/tools/mutants-plan.sh:1store-unit-of-work-handle.mdscripts/tools/mutants-night.sh:1condition-branch-agreement.mdflake.nix:142declared-entry-points.mdflake.nix:75enumerated-entry-modules.mdflake.nix:146import-time-inertness.mdflake.nix:142one-access-path.mdflake.nix:75public-signature-types.mdflake.nix:146surface-changes-are-versioned.mdflake.nix:142arbitrary-filter-floors.mdscripts/tools/mutants-plan.sh:1cross-field-invariants-as-struct-checks.mdscripts/tools/mutants-night.sh:1data-only-schema-classes.mdscripts/tools/mutants-night.test.sh:79invariants-as-refinements.mdscripts/tools/mutants-plan.sh:1recursive-schema-suspend.mdscripts/tools/mutants-night.sh:1refusal-properties-cover-unobservable-boundaries.mdscripts/tools/mutants-night.test.sh:79rich-type-over-foreign-encoded.mdscripts/tools/mutants-plan.sh:1tagged-unions-over-state-by-presence.mdscripts/tools/mutants-night.sh:1tests-own-no-schemas.mdscripts/tools/mutants-night.test.sh:79ce-compound
Nothing new written: the documentation was skipped. The durable learning, that the plan must size from the same filtered listing the shards split, round-robin by index, and that a reproduce must not resume, is in this PR's update to
docs/solutions/tooling-decisions/cargo-mutants-resume-and-shard-semantics.md. The other lessons from this session are either in the plan's KTDs or not yet observed, such as how a re-run behaves on a real night, so writing them up now would document claims nobody has checked.Privacy
A case-insensitive grep for private project and person names, email addresses and local filesystem paths finds nothing in the diff
origin/main..5f88dbeac. In the commit messages it finds only the author line, the identity this repository already commits under. The same grep over this body finds nothing.