You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Undefined t-statistic is written as bare NaN, making the metrics registry invalid JSON #2875
HarmScorerEvaluator._compute_metrics deliberately reports an undefined t-test as NaN (pyrit/score/scorer_evaluation/scorer_evaluator.py:711, and test_compute_harm_metrics_partial_agreement in tests/unit/score/test_scorer_evaluator.py pins np.isnan(metrics.t_statistic)). That is the right in-memory signal.
The serialization boundary does not carry it. ScorerMetrics.to_json (pyrit/score/scorer_evaluation/scorer_metrics.py:87) and both registry writers (scorer_metrics_io.py:330 for the atomic rewrite, :513 for the append) call json.dumps with its default allow_nan=True, so a metrics row whose statistic is undefined is written with bare NaN tokens.
Two consequences, both silent:
The JSONL stops being JSON. Python's own reader takes it back, so nothing looks wrong from inside PyRIT, but RFC 8259 has no such literal: JSON.parse in JavaScript rejects the whole document, as do the .NET and Go standard libraries. pyrit/datasets/scorer_evals/**/*_metrics.jsonl are files people read with other tools and in CI.
Readers that do get through misread it.NaN < 0.05 is False, so "the test was undefined" is indistinguishable from "not significant", and any mean or sum over that column silently becomes NaN.
Reproduction
Measured on 7b533109, a HarmScorerMetrics whose t_statistic and p_value are NaN (the shape _compute_metrics returns for a constant non-zero bias):
grep -c NaN pyrit/datasets/scorer_evals is empty today, so no shipped file is affected yet — this bites the first evaluation that hits the degenerate case and writes the registry.
Expected
null at the serialization boundary, since that is JSON's "no value", with the field types widened to float | None so the absence is visible in the dataclass too, and allow_nan=False on the writes so any future non-finite value fails loudly instead of producing a document other parsers reject. In memory nothing changes: NaN stays what _compute_metrics reports.
One consequence to decide on: doc/code/scoring/4_scorer_metrics.py:184 formats p_value with :.4f, which raises TypeError on None. The page works today only because it prints nan. I would rather not touch that file while #2860 is editing the same notebook pair — a guard there (or here) is a two-line follow-up either way.
Activity