Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 33 additions & 8 deletions workshop/27-evaluate-workflow-quality.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,20 @@ You'll add an `evals:` block to your workflow, define binary quality questions,

## Steps

### Add an `evals:` block
### Add an `evals:` block with the skill

Open `.github/workflows/daily-status.md` and add binary questions to frontmatter.
In your Copilot CLI session in the terminal, paste:

```prompt
/agentic-workflows add three binary eval questions to daily-status.md: one checking that a status issue was created, one checking the output includes a summary of repository activity, and one checking that no writes happened outside declared safe outputs.
```

The skill adds the `evals:` block to your frontmatter, compiles the lock file, and shows you the diff.

<details>
<summary>:desktop_computer: Terminal path — add the evals block directly</summary>

Open `.github/workflows/daily-status.md` and add binary questions to frontmatter:

```markdown .github/workflows/daily-status.md
---
Expand All @@ -48,17 +59,25 @@ evals:
---
```

Each question should test one observable claim and be answerable from agent output alone.
Each question should test one observable claim and be answerable from agent output alone. Compile after editing:

```bash
gh aw compile daily-status
```

</details>

### Compile and run
### Commit and trigger a run

Compile after editing:
Commit both the workflow source and the recompiled lock file, then trigger a run from the Actions UI:

```bash
gh aw compile daily-status
git add .
git commit -m "feat: add evals to daily-status workflow"
git push
```

Trigger a run from the Actions UI (or use `gh aw run daily-status` if you prefer CLI).
Go to **Actions → Daily Status Report → Run workflow** and click **Run workflow**.

### Inspect evaluation results

Expand All @@ -78,8 +97,14 @@ Example record:

When you update your prompt or tools, rerun the workflow and compare answers across runs. A question that flips from `YES` to `NO` is a fast signal that quality regressed and needs investigation.

To refine your questions, return to Copilot CLI and describe what you observed:

```prompt
/agentic-workflows the eval question "includes_summary" is too broad — update it to check that the output includes at least one open issue or pull request from the last 24 hours.
```

> [!TIP]
> Prefer using an agent with `/agentic-workflows` to add or refine `evals:` questions, then run `gh aw compile --watch` while iterating.
> Use `gh aw compile --watch` while iterating on eval questions to get instant feedback on compile errors.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inaccurate flag: --watch does not exist for gh aw compile

The tip references gh aw compile --watch, but this flag is not part of the gh aw compile command. The documented flags are --strict, --validate, --fail-fast, --purge, and --approve — no --watch flag exists.

Suggested fix:

> Use `gh aw compile daily-status` after each edit to get immediate feedback on compile errors.


## ✅ Checkpoint

Expand Down
Loading