Skip to content

Pattern quality: Show concrete YAML schema for skip-if-match/expires in generated prompts - #180

Merged
pelikhan merged 1 commit into
mainfrom
fix/skip-if-match-schema-guidance-9ae881c7133b1aaa
Aug 29, 2026
Merged

Pattern quality: Show concrete YAML schema for skip-if-match/expires in generated prompts#180
pelikhan merged 1 commit into
mainfrom
fix/skip-if-match-schema-guidance-9ae881c7133b1aaa

Conversation

@github-actions

Copy link
Copy Markdown
Contributor

What was evaluated

Read /tmp/gh-aw/data/generated-patterns-and-prompts.json (6 archetype samples), patterns/manifest.json, patterns/workflow-generation.json, patterns/archetypes/*.json, and the generator (src/js/workflow.js, src/js/patterns.js). Cross-checked the generated status-report-daily and dependency-monitor-daily prompts against gh-aw's actual reference docs (workflow-patterns.md, report.md, triggers.md) fetched live from github/gh-aw.

Recurring quality issue found

Four archetypes (status-report, security-scanner, code-improvement, dependency-monitor, documentation-updater, code-health-auditor) carry a tip telling the downstream agent to add skip-if-match and expires to prevent duplicate scheduled findings. That tip already reaches the generated prompt (doNotConstraints/duplicatePreventionTips in workflow.js), but only as prose:

"Prevent duplicate scheduled findings, for example: Use skip-if-match and expires on create-issue so scheduled reports don't accumulate as duplicate open issues"

The prompt's own "Suggested workflow file" starter YAML never demonstrates this — its safe-outputs: block is a bare create-issue: with no skip-if-match/expires anywhere. Per gh-aw's actual syntax, skip-if-match is a sibling key of schedule: under on:, while expires/max nest under the specific safe-output key (e.g. create-issue:) inside safe-outputs:. With prose-only guidance and no example, a downstream agent generating the real workflow has no concrete model and is likely to misplace these fields (e.g. put skip-if-match under safe-outputs:) or drop the feature entirely — silently reintroducing the exact duplicate-issue-spam problem the tip exists to prevent.

What changed

In src/js/workflow.js, immediately after the existing "Prevent duplicate scheduled findings" bullet, the generated prompt now includes a concrete YAML snippet showing the correct schema:

on:
  schedule: every 30 minutes
  skip-if-match: 'is:issue is:open "gh-aw-workflow-id: <workflow-id>" in:body'
safe-outputs:
  create-issue:
    max: 1
    expires: 7

This only appears for archetypes whose pattern tips already mention skip-if-match/expires/tracker-id, so it doesn't add noise to prompts that don't need it.

Added a targeted test in test/workflow.test.js asserting the schema snippet appears with the correct key nesting for the status-report archetype.

Validation

  • npm test — 205/205 passed (added 1 new test)
  • npm run build — succeeded

Follow-ups deliberately left out

  • Not adding a similar structured example for other duplicate-prevention primitives (tracker-id, close-older-issues) — scoping this fix to the specific ambiguity found in the sampled prompts.
  • Not regenerating patterns/ from scan data, per task constraints.

Generated by Pattern Quality Eval · auto · 101.1 AIC · ⌖ 7.18 AIC · ⊞ 7.6K ·

The generated prompt only told downstream agents to 'Use skip-if-match
and expires on create-issue' in prose, with no example of where these
fields belong in the YAML frontmatter. The suggested starter workflow
in the same prompt also omitted them entirely from its safe-outputs
block, so a downstream agent had no concrete model to follow and could
easily misplace skip-if-match under safe-outputs instead of on:, or
omit the feature altogether.

This adds a short YAML snippet (based on gh-aw's workflow-patterns.md
and report.md) showing skip-if-match as a sibling of schedule: under
on:, and expires/max nested under the specific safe-output key.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@pelikhan
pelikhan marked this pull request as ready for review August 29, 2026 01:43
@pelikhan
pelikhan merged commit 28d8f88 into main Aug 29, 2026
1 check passed
@pelikhan
pelikhan deleted the fix/skip-if-match-schema-guidance-9ae881c7133b1aaa branch August 29, 2026 01:43

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer kind: matt-pocock (JS-only change to prompt-generation logic).

Reviewed the diff in src/js/workflow.js and the added test in test/workflow.test.js. The change adds a concrete YAML schema example clarifying skip-if-match (sibling of schedule: under on:) vs expires/max (nested under the safe-output key). This is a template-string addition with no control-flow changes, guarded behind the existing duplicateTips.length check, and is covered by a new targeted test that asserts the exact string shape. No correctness, type-safety, or test-coverage issues found — no blocking feedback.

Generated by Specialist PR Review for #180 · auto · 15.2 AIC · ⌖ 1.94 AIC · ⊞ 7.9K

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant