feat: LLM-generated skill-specific edge cases

[csoceanu]

Summary

Adds LLM-generated edge cases specific to each skill's domain. The pipeline reads SKILL.md at runtime and generates 2-3 edge case scenarios via LLM with tailored assertions. Also removes the unused _check_failure_mode() sub-check.

• LLM generation: new pipeline step reads SKILL.md, calls LLM to produce skill-specific edge case evals
• Per-edge-case ASE runs: each generated edge case runs as a separate ASE evaluation
• Submitter edge cases: edge_cases/*.md still supported alongside LLM-generated ones
• Source tagging: aggregation tags results as generated or submitter
• failure_mode removal: sub-check removed, behaviors covered by edge case assertions. Weights rebalanced to consistency 40%, edge_case 40%, stability 20%

Certified level requirements

For a skill to pass the behavioral testing check (required for Certified in the skill profile), it needs:

• Consistency check: trial variance below threshold (always has data from the main evaluation)
• Edge case check: at least one of these must succeed:
  • LLM-generated edge cases from SKILL.md (automatic, requires successful LLM call)
  • Submitter-provided edge_cases/*.md files

If LLM generation fails (LLM down, malformed responses after retries) AND no submitter edge cases exist, the edge case sub-check has no data. Combined with the min-2 requirement, only consistency has data — behavioral testing fails, blocking Certified.

Validation

Tested on cluster with 3 diverse skills:

Skill	Domain	Edge cases generated	Pass rate	Score
hello-world-full	file creation	nested path, special chars, empty content	3/3	1.00
csv-data-processor	data processing	missing column, single-row groups, encoding	3/3	1.00
python-code-review	code quality	perfect code, security issues, non-Python	2/3	0.75

Also verified submitter + generated edge cases work together (hello-world-behavioral: 6 edge cases total, 3 generated + 3 submitter).

Generation features

• temperature=0 for reproducible results
• Retry up to 3 times on malformed LLM output
• Enforce edge- prefix on IDs, sanitize IDs for path safety, override skill_name, cap evals and assertions at 3
• JSON extraction handles markdown fences and trailing text
• Validates eval entries are dicts before accessing fields
• Mocked unit tests for validation, retry, and edge cases

[GuyZivRH]

Consolidated review: PR #55

Verdict: Request changes (two small code fixes) → then approve

CI green · 6 commits · +474/−157 · end-to-end wiring verified · 1029 tests pass on branch

---

What it does (agreed)

1. LLM-generates 2–3 skill-specific edge cases from SKILL.md at evaluate time
2. Keeps submitter edge_cases/*.md alongside, tagged generated / submitter
3. Removes never-wired _check_failure_mode; weights → consistency 40% / edge_case 40% / stability 20%
4. Caps evals/assertions at 3; retries + JSON fence/rindex recovery

Pipeline path is correct:
generate_edge_case_evals.py → _eval_tmp/edge-case-evals/ → ase-eval-edge-cases → aggregate_edge_case_evals.py → analyze merges into report.json → EdgeCaseGate / certification.

---

Blocking / must-fix before merge

#	Issue
1	Retry loop crashes on non-dict evals items — ev.get(...) raises AttributeError on e.g. {"evals": ["not-a-dict"]}, burns retries, dumps traceback instead of “Failed after N attempts”. Wrap validation in try/except (AttributeError, TypeError, KeyError) (or isinstance(ev, dict)) and continue.
2	Sanitize LLM edge IDs before using as paths — EDGE_NAME from id after stripping edge- is used in _eval_tmp/edge-case-$EDGE_NAME and results dirs with no charset check → edge-../../… / spaces / metacharacters. Normalize to [a-zA-Z0-9_-]+.

---

Strongly recommended (product / ops)

#	Issue
3	Non-reproducible Certified scores — temperature=0.3, no cache/seed → different edge packs per run. Cache generated evals, freeze in report, or temperature=0 + seed for monitoring.
4	Document Certified needs successful generation or submitter edge_cases/ (LLM failure → warning → may leave only consistency → min-2 fail).
5	AEH still skips edge-case steps (harbor|ase|both only) — state explicitly if intentional.

---

Minor / non-blocking (keep as follow-ups)

• EdgeCaseResult.source set via Tekton raw-dict merge — not Pydantic-checked until AnalysisResult deserialize
• report.md still doesn’t render edge_case_results / source (pre-existing)
• Generation uses chat_completion → Phase A metrics miss those tokens
• Runtime npm install --global agent-skills-eval (pre-existing airgap concern)
• Optional squash of the 4 fix commits for history cleanliness
• DEFAULT_ASSERTIONS remain generic for submitter-only path — fine

---

What looks solid

• Additive generated + submitter design with submitter_ prefix / source tagging
• Clean failure_mode removal (no dangling refs; weights sum to 1.0)
• Graceful Tekton degradation (|| echo WARNING)
• Guardrails (prefix, caps, fence stripping) + real unit tests + cluster validation table
• No path/filename mismatch class of bug in the traced wiring

---

Required before merge (minimum)

1. Harden retry validation against non-dict / wrong-shape eval entries
2. Sanitize edge case IDs used as directory names