Repository navigation
feat: add maintainer-triggered PR description assessment - #4902
Conversation
Port the complete pr-assess workflow with concise reviewer-facing comments, bounded outcome-label updates, focused tests, and usage guidance. Keep the reviewed gh-aw v0.89.21 runtime pin isolated from existing workflows. Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ee0ab16-f074-4303-82b9-d11bfad16175
There was a problem hiding this comment.
🟡 Changes recommended
Authorization exceeds the stated maintainer scope, required labels are not provisioned, and the AI disclosure is incomplete.
3 open findings
What changed in this PR
Adds a maintainer-triggered workflow that evaluates PR description alignment and reports bounded comments and outcome labels.
Changes:
- Adds the
pr-assessagentic workflow and generated lock file. - Adds focused workflow configuration and prompt-contract tests.
- Documents usage and pins the isolated gh-aw runtime dependency.
| File | Description |
|---|---|
.github/workflows/pr-assess.md |
Defines assessment behavior and safeguards. |
.github/workflows/pr-assess.lock.yml |
Provides the compiled GitHub Actions workflow. |
.github/aw/actions-lock.json |
Pins gh-aw setup v0.89.21. |
tests/test_github_workflows.py |
Tests triggers, permissions, outputs, and reporting. |
docs/guides/agentic-sdlc.md |
Documents maintainer usage and outcomes. |
🧠 Review effort: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Port the tested built-in label replacement and standalone-comment behavior. Keep matching, conflicting, or unreadable outcome labels unchanged. Limit suggested updates to the PR description, not changes to the code. Include offline digest-checked probes for the pinned MIT-licensed handler. Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ee0ab16-f074-4303-82b9-d11bfad16175
Skip test if Node.js is not available. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Follow the extension-submission remove/add pattern: remove up to two stale outcomes and add the selected outcome only when absent. Keep matching outcomes unchanged, post fresh standalone comments, and limit suggested updates to the description. Remove the obsolete replacement-handler tests and fixtures. Make no transactional or concurrent-manual-edit guarantee. Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ee0ab16-f074-4303-82b9-d11bfad16175
|
Updated in Prepared on behalf of KSchlobohm with GitHub Copilot (GPT-6.1 Sol, autonomous); AI-assisted change summary. |
Compare title text with the existing captured inputs before reporting. Require an inconclusive explanation when the title changes during assessment. Update the existing prompt contract and regenerate its pinned workflow lock. Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9ee0ab16-f074-4303-82b9-d11bfad16175
|
Updated in Prepared on behalf of KSchlobohm with GitHub Copilot (GPT-6.1 Sol, autonomous); AI-assisted change summary. |
Description
Adds an optional workflow that collaborators with write access or higher trigger with the
pr-assesslabel. It compares the PR description with the code changes, flags material omissions or contradictions, and posts a concise assessment with an outcome label to help reviewers spot gaps.Testing
I tested outcome-label handling on my fork. All four scenarios passed: first assessment, changed verdict, unchanged verdict with a new comment, and cleanup of two stale outcome labels. Suggested corrections were limited to the PR description.
Outcome labels follow the existing extension-submission remove/add pattern; a matching outcome is left unchanged.
Title stability was also tested on the fork: a stable-title control returned the normal verdict, and a title-change retry returned inconclusive and explained the change. The title now joins head, base, and body in the final input check. An earlier attempt did not change the title and was not evidence of the safeguard.
Automated check:
.\.venv\Scripts\python.exe -m pytest tests\test_github_workflows.py -q— 136 passed, 16 skipped.gh aw compile pr-assess --strict --validate --no-check-updatepassed with the expectedpull_request_targetwarning.uv tool run ruff check tests\test_github_workflows.py,npm exec --offline -- markdownlint-cli2 docs\guides\agentic-sdlc.md, andgit diff --checkpassed.uv run specify --helpuv sync && uv run pytestAI Disclosure
AI disclosure: Prepared and created with GitHub Copilot (GPT-6.1 Sol) under human supervision.