FIX Share scenario success statistics across SDK, backend and reports

[Utkarsh Bahuguna (u7k4rs6)]

Fixes #2819.

Reworked per the discussion on #2819. The scenario success calculation now lives in one place, pyrit.analytics.scenario_statistics, and everything else just presents its results.

What moved there: the plan lookup, execution-unit identity (atomic attack name plus technique configuration, plus seed group), latest-attempt selection, counts, denominators and rounding. The shared result types are in pyrit.models: ScenarioExecutionUnit and ScenarioExecutionStatistics, which reuses the existing ScenarioProgressCounts. Historical attempts, errors and retries are reported separately from the effective-unit counts.

Who uses it now:

• the backend read model, for run detail counts and every progress rollup. ScenarioPlanLookup, ResultUnitIdentity and total_retry_pressure delegate to it and stay importable from the old module.
• the pretty, JSON and HTML reports, via scenario_overview. Group rates fold the per-atomic-attack counts using display_group_map, the same keys the reports group by. Reports now show objective executions, one objective run with one attack configuration (num_objective_executions, total_objective_executions), next to raw attempts (num_attempts, total_attempts). group_success_rate is gone.
• SDK callers, via pyrit.analytics.compute_scenario_statistics(result).

One API change: ScenarioResult.objective_achieved_rate is deprecated (removed in 1.4.0). test_import_boundary doesn't allow pyrit.models to import pyrit.analytics, even lazily, so the method can't call the shared code without adding a new tracked violation. It keeps its old behaviour until removal, and nothing in PyRIT calls it anymore. I added a filterwarnings entry like the one for Scorer.score_async so the suite can't drift back onto it. If you'd rather have a tracked lazy violation instead, that's an easy switch.

Breaking change for pyrit scan --report json (worth a release note): stats.total_results is now stats.total_objective_executions plus stats.total_attempts, and each group's num_results is now num_objective_executions plus num_attempts. The denominator really changed, so keeping the old names with a new meaning would be worse. Anything reading the old keys gets a KeyError after upgrading. If the JSON report counts as a supported contract, I can also emit total_results / num_results with their old meaning (attempt counts) until 1.4.0, same as objective_achieved_rate.

This also unifies the two identity resolvers the backend had. One fell back to the bare atomic attack name and the other to a hash of the name plus eval hash. Both now use the latter, so technique configurations that share a name are separate units everywhere.

Parity: tests/unit/analytics/test_scenario_statistics_parity.py saves a history in SQLite, then checks that the SDK, run detail, run history list (the SQL aggregate), live progress and the JSON report give the same overall numbers. For histories with a saved plan it checks the per-group numbers too. It covers retry/resume, unrecovered errors, legacy rows with no plan or seed attribution, an old error row matched to its unit by the saved plan, two technique configs sharing a name, legacy seed groups that share an objective (told apart by the atomic identifier's seeds in both the SDK and SQL), display groups, and an empty history. A strict xfail pins one known gap that predates this PR: the history list rejects a plan with two seed groups sharing an objective and falls back to legacy totals, while run detail keeps using the plan. For reference, running the same histories through main, the SDK/report disagree with the backend on 5 of 7 (e.g. 50 vs 100 after retry/resume, 33 vs 50 with unrecovered errors).

A few existing tests built several rows with the same objective in one atomic attack and expected each row to count separately. Under the shared policy those are one unit, as they already were in the backend, so I gave them distinct objectives.

Unrelated, but it made CI flaky here: test_frontend_exists_mounts_static mounted the SPA on the shared app and never removed it. That turned GET /runs/{id}/resume into a 404 for any test that ran after it in the same worker (reproducible on main by running the two tests in that order). It now restores the routes.

Cached benchmark results: since #2551 the benchmark drops cached objectives before building the plan and copies the cached results into the new run. The plan now still lists those objectives, with one seed group per cached result, and each copy is attributed to that seed group when it's saved, so cached results count as planned units.

Tests: tests/unit/{analytics,backend,output,scenario,memory,models}: 6190 passed, 1 xfailed.

[Utkarsh Bahuguna (u7k4rs6)]

Heads up on an overlap with #2551 (hannahwestra25). That PR reworks benchmark caching: it drops cached objectives from each atomic attack before the plan is built, then copies the cached results into the new run. Those copies aren't in the saved plan, so the plan-based counting here (and the backend progress view on main today) would treat them as unattributed and leave them out of the stats.

The cached-plan hook I added in run_async would go away with #2551 anyway. Whichever lands second should keep the pruned cached objectives in the run plan (or build the plan before pruning). Happy to do that rebase on this side if #2551 goes in first.

[Roman Lutz (romanlutz)]

🔴 Must Fix: Give identifier-derived and attributed seeds the same SQL identity

These two branches produce different keys for the same logical seed group: seed_group_id is its canonical hash, while the identifier fallback is "seeds:" + joined_seed_hashes. The SDK resolves both to AtomicAttackIdentifier.logical_seed_group_id.

I reproduced this in SQLite with no saved plan and two attempts for the same attack, configuration, objective and prompt context:

# Older ERROR: atomic identifier contains the seeds, no seed_group_id.
# Later SUCCESS: same identifier, seed_group_id = seed_group.logical_id.

The SDK, run detail and live progress report 1 completed unit, 100%, 1 retry. The history list reports 2 completed units, 50%, 0 retries. This is one recovered execution, not two separate units.

Can we normalize both SQL branches to the same logical seed identity in both backends, and add a mixed identifier-only / attributed history to the parity suite? The current legacy-seed fixture gives every row identifier-only attribution, so it cannot catch this mismatch.

[Utkarsh Bahuguna (u7k4rs6)]

Fixed in 11b3fd2. Both backends now key an attempt by the identifier's seed key whenever it has one, and rows that only have the attributed seed_group_id borrow that key from a row in the same run that carries both, since those two spellings name the same logical seed group. SQL can't hash the seeds itself, so that pairing is the bridge. Added identifier_and_attributed_rows_of_one_seed_group to the parity histories: an identifier-only ERROR, an attributed-only ERROR and a row with both. Everything reports 1 unit, 100%, 2 retries; before the fix the history list said 50%. One case it can't bridge: a seed group whose only rows are identifier-only and attributed-only, with none carrying both. That needs an old run resumed by new code where every new attempt failed before enrichment, so I left it.

[Roman Lutz (romanlutz)]

This is partially fixed, but the remaining no-bridge case still needs to be addressed before resolving this thread.

I reproduced it again on e394bff with no saved plan and two attempts for the same logical seed group:

# Earlier ERROR: identifier seeds only, no seed_group_id.
# Later SUCCESS: seed_group_id only, no identifier seeds.
# No third row carrying both.

The SDK and run detail report 1 objective execution, 100%, 1 retry. History reports 2 objective executions, 50%, 0 retries.

The new test's third row supplies the bridge, so it does not cover this case. Please add this two-row history to the parity suite and make the counts agree without requiring another row carrying both identity forms. The history is already sufficient for the SDK to identify one recovered execution.

_{Auto-replied by the GitHub Copilot app}

[Utkarsh Bahuguna (u7k4rs6)]

Fixed in 8f0382c. SQL can't derive the logical seed group from stored seeds, so instead of approximating it, the history aggregate now flags any run with an attempt that has identifier seeds but no explicit attribution, and list_runs recounts those runs with compute_scenario_statistics, the same path run detail uses. Runs without such rows stay on the SQL aggregate. Your two-row history is identifier_only_then_attributed_only in the parity suite (1 execution, 100%, 1 retry everywhere), plus a direct test for the flag. Only legacy runs from before seed attribution take the slower path.

[Roman Lutz (romanlutz)]

🔴 Must Fix: Keep explicit seed attribution authoritative

This now prefers the identifier's seeds over seed_group_id, but resolve_execution_unit() does the opposite. Those values are not always interchangeable: the benchmark cache copier deliberately assigns the current seed group while retaining the original result's identifier.

I persisted two rows without a saved plan, with the same attack/configuration/objective and the same stored seed identifier, but different explicitly attributed logical seed groups:

# FAILURE: identifier seeds = old context, seed_group_id = current group A
# SUCCESS: identifier seeds = old context, seed_group_id = current group B

The SDK and run detail correctly report 2 objective executions, 50%, no retries. History merges them into 1 execution, 100%, 1 retry. The reverse case also fails: one attributed seed group with two different stored identifiers is split into two SQL executions instead of one.

Can we keep explicit attribution authoritative and normalize identifier-only rows onto that canonical identity, rather than replacing attributed identities with identifier keys? Please cover both cases in the parity suite. Simply putting attribution first again would reintroduce the original mixed-attribution bug, so this needs a consistent normalization rule.

[Utkarsh Bahuguna (u7k4rs6)]

Fixed in 8f0382c. The SQL now keys attempts by explicit seed attribution first, then the objective hash, same order as resolve_execution_unit, and the identifier bridge is gone. Both of your cases are in the parity suite: explicit_attribution_wins_over_identifier_seeds (same identifier, different attribution, 2 executions at 50%) and one_attributed_seed_group_with_two_identifiers (same attribution, different identifiers, 1 execution at 100%, 1 retry). Both fail on e394bff.