FEAT: report harm scorer error split by inter-rater agreement

[WatchTree-19]

Description

Implements #2628.

A harm scorer is evaluated against one gold label per response. Where that label is the median of several human raters, the aggregate MAE hides how much the raters disagreed. On violence_multi_score.csv nearly a third of the binary harmful/not labels are a 2-1 split (58 of 187), so a scorer marked wrong on those rows was outvoted by one human rather than overruled by a consensus. The measurement in #2628 shows why that matters: out of fold, the violence classifier from #2626 is at 86.0% on unanimous rows and 51.7% on contested ones, and the single aggregate figure of 75.4% says nothing about that gap.

HarmScorerEvaluator now also reports the MAE over unanimous responses (every rater on the same side of 0.5) and over contested responses (a split across 0.5), with the count of each and the threshold used, as five new optional fields on HarmScorerMetrics:

• contested_threshold
• num_unanimous_responses, num_contested_responses
• mean_absolute_error_unanimous, mean_absolute_error_contested

The split is only computed when a gold set has more than one rater. Single-rater sets report None rather than a trivially unanimous split, because there is no disagreement there to measure, which is the other half of #2628.

Nothing existing moves. The aggregate mean_absolute_error is untouched and is the row-weighted average of the two strata, so every number in the shipped *_metrics.jsonl files is unchanged. Metrics files written before these fields still load, since from_json_file already tolerates absent keys. The pretty printer shows the two rows when they are present, and the scorer metrics page documents them.

On the shipped multi-rater gold sets the split lands as 129 unanimous and 58 contested for violence, 164 and 35 for hate speech, and 11 and 4 for the hate speech mini set, matching the audit table in #2628.

Tests and Documentation

tests/unit/score/test_scorer_evaluator.py gains four tests: the split on a mixed gold set, including that the aggregate equals the row-weighted average of the strata; None for a single rater; None for the contested stratum when no rows are contested; and that a metrics file written before these fields still loads. All four fail on an unpatched tree with AttributeError. tests/unit/output/scorer/test_pretty.py gains two: the rows print when present and are absent for a single rater.

Verified against upstream main 0ac4526: 71 passed in test_scorer_evaluator.py and 86 across it and test_pretty.py. ruff check and ruff format clean on every touched file. mypy reports the same 134 pre-existing errors before and after, none in the touched files. The notebook 4_scorer_metrics.ipynb round-trips through jupytext against its .py with only the tool version stamp differing.

Written with AI assistance.