Upgrade PolicyEngine.py, compact SPM provenance, and deploy UK credentials

[anth-volk]

Fixes #3862
Fixes #3865
Fixes #3867

Summary

• Install policyengine[models]==6.2.1, selecting PolicyEngine Core 3.32.10, PolicyEngine US 2.2.1, PolicyEngine UK 2.102.3, and spm-calculator 1.0.0.
• Replace the previous large Supplemental Poverty Measure result metadata with explicitly typed compact household and baseline-versus-reform provenance receipts.
• Require completed US household and society-wide calculations to carry those compact receipts, validate them before persistence or caching, and include the comparison schema version in cache identity.
• Configure the dedicated PE_UK_PRIVATE_HF_READ_TOKEN credential through GitHub Actions, Google Secret Manager, Cloud Run, and the Modal secret synchronization step.
• Keep the automatic PolicyEngine bundle updater atomic by changing the single bundle requirement, regenerating the lockfile, and validating all installed distribution versions against the bundle manifest.

This PR consolidates and supersedes #3863 and #3866 while preserving their logical commits. It also incorporates the PolicyEngine.py 6.2.1 upgrade that was already present in this PR.

API behavior

The compact SPM contract deliberately removes the earlier spm_config field and the large per-execution provenance structure. A completed household calculation returns one canonical-spm-provenance-v2 receipt. A completed society-wide calculation returns one canonical-spm-comparison-v2 object containing one receipt and an execution count for each of the baseline and reform simulations.

The receipt records only the selected forecast identity and content hash, scenario, geography selection, county-data vintage, optional as-of date, calculated years, and the four runtime package versions needed to identify the calculation implementation.

Simulation service relationship

The corresponding simulation-service changes have already landed through PolicyEngine/policyengine-sim-api#703, #716, and #721. The API validates the worker's installed bundle and compact receipt contract before accepting completed US economy results.

Credential deployment

The deployment workflows synchronize PE_UK_PRIVATE_HF_READ_TOKEN from the GitHub environment into Google Secret Manager, grant the Cloud Run runtime identity access, attach it to Cloud Run under the same explicit name, and include it in the Modal secret set. The prior ambiguous token names are not accepted by the new path.

Validation

• uv lock --check
• uv run --frozen --extra dev mypy
• uv run --frozen --extra dev ruff format --check .
• uv run --frozen --extra dev ruff check .
• uv run python scripts/export_migration_contracts.py (no generated changes)
• uv run python scripts/run_quality_guards.py
• Focused combined suite: 592 passed
• Contract and generated migration-artifact suite: 47 passed
• Initial full configured suite: 2,352 passed and nine stale test-double failures
• Exact affected test files after correcting those fixtures: 19 passed

[codecov]

Codecov Report

❌ Patch coverage is 79.80769% with 63 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.56%. Comparing base (0ebfc08) to head (c04d6a5).

Files with missing lines	Patch %	Lines
policyengine_api/spm.py	84.12%	10 Missing and 10 partials ⚠️
policyengine_api/services/report_output_service.py	57.14%	12 Missing and 3 partials ⚠️
policyengine_api/worker_spm_release.py	75.86%	10 Missing and 4 partials ⚠️
...engine_api/runtime_cache/household_calculations.py	78.26%	2 Missing and 3 partials ⚠️
...gine_api/services/household_calculation_service.py	71.42%	2 Missing and 2 partials ⚠️
policyengine_api/services/simulation_service.py	91.66%	2 Missing ⚠️
policyengine_api/worker_spm.py	80.00%	1 Missing and 1 partial ⚠️
policyengine_api/services/v2/reports/types.py	87.50%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #3855       +/-   ##
===========================================
+ Coverage   45.76%   87.56%   +41.79%     
===========================================
  Files         182      198       +16     
  Lines       10812    12211     +1399     
  Branches     1837     2159      +322     
===========================================
+ Hits         4948    10692     +5744     
+ Misses       5456      922     -4534     
- Partials      408      597      +189     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[anth-volk]

CI follow-up completed in commits 736c26c3 and dd5a0ac7.

• Applied Ruff formatting to the four reported test modules.
• Confirmed the existing changelog fragment already passed; no changelog edit was needed.
• Removed the live simulation compatibility job as a prerequisite for the API test and container-build jobs. The compatibility check remains visible.
• Merged current master and resolved the v6 SPM test fixtures against the newly merged observability and cache behavior.

Validation:

• Local full suite: 2,311 passed, 94 skipped.
• Hosted lint, changelog, API tests, environment tests, both container builds, both database migration checks, cross-database integration tests, type checking, quality checks, and Codecov checks pass.

The only remaining unsuccessful check is the live simulation compatibility check because the deployed simulation service does not yet advertise PolicyEngine.py 6.1.2. No service was deployed or merged as part of this fix.

[juaristi22]

@anth-volk, could this PR and PolicyEngine/policyengine-sim-api#703 target policyengine.py 6.2.0 instead of 6.1.2 before they merge?

A UK household calculator I'm building needs the duties added in policyengine-uk 2.102.0 (PolicyEngine/policyengine-uk#1876: alcohol, tobacco, vehicle excise and road fuel gas). API v1 runs policyengine-uk 2.90.2 today, and 6.1.2 still selects 2.90.2. 6.2.0 was released on 29 Sep at 18:02 UTC, after this PR was opened, and selects 2.102.3 (PolicyEngine/policyengine.py#536).

What moves between 6.1.2 and 6.2.0. Only policyengine-uk, from 2.90.2 to 2.102.3. Core 3.32.5, US 2.2.1, spm-calculator 1.0.0 and both certified data releases (populace-us-2024-spm-20260915, policyengine-uk-data-1.56.16) stay the same, and the release's only other change is a docs note (PolicyEngine/policyengine.py#534). Here I'd expect it to need the two pins, a re-lock, the package table in docs/canonical-spm.md and new counts in test_v2_catalog_installed.py, since 2.102.3 adds variables.

What else it changes. 6.2.0 moves every UK result the API serves, household and economy, onto 2.102.3. Its changelog lists the absolute poverty line rebased to FYE 2025, which raises UK absolute poverty in 2026-27 on the certified enhanced FRS by about 4 points overall and 7 points for children after housing costs, and relative poverty measured on the median over individuals (PolicyEngine/policyengine-uk#1865). Versions 2.102.1 and 2.102.2 also fix Class 4 National Insurance.

If you and Max would rather land the v6 migration on 6.1.2 first, when would 6.2.0 follow? No bot PR will open for it: every policyengine-release dispatch from 6.0.0 to 6.2.0 failed at uv lock in both repos on the direct spm-calculator==0.3.1 pin (6.2.0 runs: policyengine-api, policyengine-sim-api). Once this PR and PolicyEngine/policyengine-sim-api#703 merge, both updaters would need a manual run with version 6.2.0.