Feature: forge test-skill — local skill testing without Jira/GitHub

[ItzikEzra-rh]

Problem

Iterating on Forge skills requires a full end-to-end run: real Jira ticket, hosted beta execution, real GitHub PR, and trace analysis. Most of the pipeline (Jira webhooks, GitHub PR creation, fork sync) adds no value during skill development — only the agent execution matters. This makes skill iteration slow and expensive.

Proposal

Add a forge test-skill subcommand that runs a skill locally against pre-fetched input without Jira or GitHub integration:

forge test-skill \
  --skill generate-prd \
  --skill-dir skills/osac/generate-prd \
  --input test-cases/OSAC-1110/input.yaml \
  --output output/

This would:

1. Read pre-fetched requirements from an input file (no live Jira needed)
2. Construct the same system prompt and user message that Forge uses in production
3. Set up the same tool environment (read_file, write_file, edit_file, ls, glob, grep, write_todos)
4. Run the agent with the configured model
5. Save the output file(s) — no GitHub PR creation
6. Save the conversation trace as JSON for debugging

What Forge provides that must be reproduced

From analyzing Langfuse traces, the skill-agnostic context Forge wraps around every skill is:

• System prompt (~14K chars) — agent identity + behavior rules, write_todos instructions, skills metadata list, filesystem tool docs, subagent spawner
• User message format — per-workflow-step template wrapping the input content
• 8 tools — read_file, write_file, edit_file, ls, glob, grep, write_todos, task
• File structure — skills at /opt/forge/skills/, workspace at /home/user/

All of this is deterministic and skill-agnostic. The only variables are the skill content and the input.

Use Cases

• Skill development — iterate on skill instructions and test locally before deploying to the beta
• Eval harness integration — run test suites against pre-fetched datasets (e.g., with agent-eval-harness or custom evaluators)
• A/B testing — compare two skill versions on the same input without full Forge overhead
• CI — validate skill changes don't regress on gold-standard test cases before merging

Workaround

We're building a standalone test runner that extracts Forge's context from traces and reproduces it via the Anthropic API. This works but is fragile — if Forge changes its system prompt or tool setup, the runner falls out of sync. A first-class forge test-skill would stay aligned automatically.