You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The production selector has the same resource-only gap as the local selector: changes below a skill's references, scripts, or assets directories do not match its SKILL.md source path, so the workflow skips the affected skill evaluation. Match skill components using the parent skill directory. .github/workflows/agent-harness-evaluation.yml:346
This unconditional return leaves a PR without rerun instructions when a synchronize event introduces its first harness-affecting change: there are no prior evaluation comments, so the stale notice above is not created, and synchronize runs are intentionally not authorized to evaluate. Return here only when a stale notice already covers the existing evaluations; otherwise continue to create the manual-required prompt. eng/harness-evaluation/src/harness.mjs:370
Relative repository paths are resolved only from the copied eval directory. Consequently, an authored target reference such as src: .agents/skills/example/SKILL.md with an unrelated destination is retained in the control, leaking the evaluated skill into the supposedly unskilled arm. Check both eval-relative and repository-relative interpretations before retaining a path. eng/harness-evaluation/src/harness.mjs:502
Skill resources are part of the evaluated component, but this matches changes only against SKILL.md. A resource-only edit such as .agents/skills/make-skill/references/testing-patterns.md therefore selects no evaluation. Match skill changes against the skill directory while retaining file matching for other component kinds. .agents/skills/sqlite-adonet/SKILL.md:17
This is phrased as current implementation behavior, but SqliteConnection.Close() currently iterates _commands and calls command.Dispose() (src/Microsoft.Data.Sqlite.Core/SqliteConnection.cs:372-380). That contradiction will mislead the evaluation away from the ownership-boundary bug it is meant to diagnose. State the expected ADO.NET lifetime rule normatively rather than claiming the current code already follows it.
When a synchronize commit removes the last harness-related change, selected is empty but prior evaluation comments still make this branch create/update a “rerun required” notice. The code then returns at line 327, so the PR simultaneously reports no affected components and asks for an impossible /eval rerun. Handle the empty selection before creating the stale notice and remove any existing rerun prompt in that path.
if (context.payload.action === 'synchronize' && evalComments.length > 0) {
const body = `${rerunMarker}\n### Agent harness evaluation is stale\n\n` +
`New commits were added after the previous evaluation. A contributor with write access can comment ` +
`\`/eval\` to run all affected components or \`/eval component-name\` to run one component.`;
GitHub's pull-request files endpoint caps results at 3,000 files. This branch paginates but never detects that cap, so a very large PR can silently omit changed customizations and produce an incomplete evaluation matrix. Fail closed when the result reaches 3,000 entries, as the historical-commit branch already does for its lower API cap.
defaults.runs is an eval-level field, not a field that each stimulus sets; all of the evals added here declare it once beside stimuli. As written, this rule says every stimulus must contain an impossible/invalid setting and misstates what the validator enforces.
Every stimulus must:
1. Set `defaults.runs: ${RUNS=5}`.
The stimulus is based on behavior the staged provider does not have: SqliteConnection.Close() disposes tracked commands, so it cannot preserve a reusable command across close/reopen. This eval will reward the skill for proposing behavior contrary to the actual lifecycle; replace the premise with a scenario consistent with the implementation, or stage the corresponding provider change before evaluating it. eng/harness-evaluation/skills/sqlite-adonet/eval.yaml:16
The stimulus is built around behavior the staged implementation does not have: SqliteConnection.Close() disposes tracked commands, so the requested conclusion to preserve command reuse after reopening is false. This makes the treatment/control evaluation reward incorrect guidance and can drive a proposed fix in the wrong direction; require the analysis to recognize that commands are not reusable after close/reopen, or stage an implementation that actually supports that contract. eng/harness-evaluation/skills/sqlite-adonet/eval.yaml:16
The eval asks the agent to diagnose command state surviving DbConnection.Close() and to preserve command reuse after reopening, but the staged SqliteConnection.Close() implementation disposes every tracked command before closing. This makes the evaluation reward an impossible/incorrect diagnosis rather than repository-grounded guidance; rewrite the stimulus around the actual close/dispose lifecycle or stage a source change that establishes a different contract.
repoRoot is the candidate checkout, so this makes a PR-controlled harness.experiment.yaml part of the executed runner. A PR that changes both an eval and this file can therefore alter variants or other experiment behavior before merge, contradicting the trusted-runner boundary documented in README.md:35 and the workflow comment at .github/workflows/agent-harness-evaluation.yml:244. Load the experiment definition from the trusted harness checkout instead; the candidate should supply only its customization, eval spec, and declared inputs. eng/harness-evaluation/src/cli.mjs:71
This makes the trusted CLI load harness.experiment.yaml from the candidate checkout. A PR that changes an eval together with this file can alter baseline, variants, or vary so the selected treatment/control comparison is no longer the committed experiment, even though infrastructure changes are supposed to be evaluated only after merge. Load the experiment definition from the trusted runner (packageRoot) or reject mixed infrastructure/component runs. eng/harness-evaluation/src/harness.mjs:330
A non-empty set of successful trials is treated as complete here; there is no check that all configured runs/stimuli were produced. Thus a truncated experiment containing one passing trial can satisfy --require-pass (the test fixture itself has this shape), even though the acceptance contract requires the configured number of control and treatment trials. Validate the result against Vally's plan snapshot or expected trial keys before computing the verdict, as the workflow verifier already does.
Affected-component selection checks only each component's source and eval directory, but evaluations also consume shared candidate files through stimulus-level agent_environment.files. For example, the make-skill, make-instructions, and make-custom-agent evals stage .github/copilot-instructions.md and this harness README. Changing either shared input can therefore skip those affected evaluations and leave their reported result stale. Track declared eval input paths when selecting components, or conservatively fan out evaluations when shared inputs change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace repository MCP configuration and duplicated skills with plugins.
Update skill guidance and add matching repository-grounded evaluations.
Add Dependabot coverage for Vally dependencies.