[skill-optimizer] Daily Skill Optimizer Improvements - 2026-08-01

[github-actions]

Summary

• Run mode: dry-run
• Status: ⚠️ (suite execution skipped — benchmark quality signal not collected)

Key Findings

1. Configure OPENROUTER_API_KEY to enable benchmark mode — the daily run stays in dry-run indefinitely, so no real skill-quality metrics are ever produced. Impact: unlocks the actual benchmark suite, turning this workflow from a no-op check into a real quality signal.
2. Split .github/skills/developer/SKILL.md (2027 lines) into focused sub-skills — it is ~4–20x larger than every other skill file in the repo and works against the repo's own lazy-loading/token-efficiency policy documented in AGENTS.md. Impact: faster, cheaper skill loads and better fusion (partial-section reads) for agents that only need one topic from it.
3. Add a lightweight, key-free smoke suite (e.g. a suite.yml under .github/skills/ or examples/, modeled on examples/workbench/mcp/suite.yml in the skill-optimizer source) that can run in dry-run/CI without OPENROUTER_API_KEY. Impact: gives daily signal on skill/doc drift even on days the full benchmark can't run (missing secret, rate limits, etc.).

Evidence from Artifact

• summary.json: {"repository":"github/gh-aw","run_mode":"dry-run","run_status":0,"run_url":"https://github.com/github/gh-aw/actions/runs/30682690432"}
• run.log: dry-run: Docker available but OPENROUTER_API_KEY not set; skipping suite execution — confirms no benchmark data was collected this run.
• No suite-results/ directory was present in the artifact (only produced when run_mode=benchmark), so there is currently no way to measure skill-guidance quality end-to-end.
• .github/skills/developer/SKILL.md is 2027 lines vs. the next largest skill (custom-agents/SKILL.md at 591 lines) and the repo median of ~110–150 lines — a clear outlier relative to the repo's own "keep skills lazily loaded / small fragments" guidance in AGENTS.md.
• The skill-optimizer source checkout (skill-optimizer-src/examples/workbench/{mcp,pdf}/suite.yml) shows the expected suite.yml shape that could be adapted for a repo-local, no-API-key smoke suite.

Recommendations

1. Add OPENROUTER_API_KEY as a repository/organization secret and wire it into the skill_optimizer job so the workflow runs in benchmark mode instead of perpetually falling back to dry-run.
2. Refactor .github/skills/developer/SKILL.md into smaller topic-scoped skill files (mirroring the granularity already used for custom-agents, debugging-workflows, etc.), keeping only a short index/router in the main file.
3. Introduce a minimal, API-key-independent suite.yml-style smoke check (structural/lint-only, no LLM calls) covering the highest-traffic skills (agentic-workflows, developer, github-mcp-server) so the daily run always produces some actionable quality signal, even without benchmark credentials.

Generated by ⚡ Daily Skill Optimizer Improvements · auto · 13.6 AIC · ⌖ 8.15 AIC · ⊞ 6.2K · ◷

• expires on Aug 7, 2026, 7:54 PM UTC-08:00