OSAC-3968: add skill testing and evaluation framework

[ItzikEzra-rh]

Summary

Local test runner and LLM judge for evaluating Forge skills without the hosted beta.

• run.py — runs a skill against test cases using deepagents + FilesystemBackend (same code path as Forge), with a --legacy fallback to raw Anthropic API
• evaluate.py — scores generated artifacts against gold standards using Sonnet with configurable per-skill criteria
• forge test-skill — CLI subcommand wrapping both tools
• MLflow integration — --mlflow flag auto-traces all API calls, logs metrics (scores, tokens, cost), links traces to runs

Skill-agnostic — adding a new skill requires a prompt template and criteria YAML, no code changes.

What changed since the first draft

Based on feedback from @eranco74 and @EllaSh:

• Replaced manual tool emulation with deepagents create_deep_agent() + FilesystemBackend for exact parity with Forge's agent execution
• Added forge test-skill run and forge test-skill eval CLI subcommands
• Kept --legacy flag for raw API fallback

Validation

Tested on 7 OSAC features (generate-prd skill):

• deepagents runner produces comparable results to hosted Forge (75% avg score)
• Full MLflow traces captured for both generation (Opus) and evaluation (Sonnet) calls
• Judge reliability measured: +/-1 point avg across 3 runs of the same inputs

Test plan

• forge test-skill run --help and forge test-skill eval --help work
• Single case run with deepagents produces a valid PRD
• Batch dataset run (7 cases) completes successfully
• Evaluation scores match expected ranges against gold PRDs
• MLflow traces linked to runs when --mlflow flag provided
• --legacy fallback works when deepagents not available

🤖 Generated with Claude Code

[eshulman2]

I found three issues while reviewing this:

1. P1 — filesystem sandbox escape (devtools/test-skill/run.py:231)

   remap_path() joins model-controlled paths onto the temporary workspace without resolving the result and verifying that it remains inside the workspace. For example, with the directories created by setup_workspace(), /home/user/../../../../etc/hostname resolves to /etc/hostname and can be read successfully. The same issue affects write_file and edit_file, so model-controlled tool calls can read or overwrite host files outside the advertised temporary workspace.

   Please resolve the candidate path and reject it unless it is contained beneath the intended workspace root (and account for symlinks as well).

2. P2 — fail_on_critical is ignored (devtools/test-skill/evaluators/judge.py:62)

   EvalReport.overall_pass always requires critical_failures to be empty. Consequently, a criteria file setting fail_on_critical: false still fails whenever a critical criterion fails. The critical-failure condition should be applied only when self.fail_on_critical is true.

3. P3 — dynamic report content is not HTML-escaped (devtools/test-skill/evaluators/reports.py:85-111)

   Quotes and reasoning returned by the judge, plus skill/path values, are interpolated directly into report.html. Crafted generated content echoed by the judge can therefore inject markup or script when a developer opens the report. Please pass all dynamic text through html.escape() before interpolation.