Skip to content

Calibration-stage cache/checkpoint identity omits solver config (no solver_config_sha256) #326

Description

@MaxGhenis

Summary

No single cached/checkpointed artifact in the US release builder records the solver/calibration configuration that produced a given calibrated result. During the Build F campaign (populace#299) this did not cause a wrong result, but it nearly misled a diagnosis. Filing so the identity gap is closed before it bites a run where config genuinely changes the cached values.

What's already correct (so this issue is narrow)

Two things I initially suspected turned out to be fine, and I want to record that so nobody "fixes" them:

  • target_registry_version is already a content hash. TargetRegistry.version hashes the canonical spec JSON (every spec's name/value/source/sign/period). A feed edit that changes the target registry (e.g. v5→v6→v7 removing M-CHIP CHIP rows) therefore does invalidate the target-frame checkpoint. No feed-collision risk there.
  • The target-frame checkpoint is legitimately solver-independent. _target_frame_checkpoint_identity (keys: base_dataset_sha256, policyengine_us_version, seed, target_period, target_registry_version, congressional_district_vintage_crosswalk_sha256) caches pre-calibration PE-US engine outputs over the base pool. Those don't depend on λ / max_weight_ratio / epochs, so omitting solver config from this checkpoint's key is correct.

The real gap

The calibrated result (weights, loss, diagnostics, exported H5) depends on the full solver config — l2_lambda, l0_refit_lambda_share, refit_l2_lambda, max_weight_ratio, epochs, learning_rate, l2_anchor/refit_l2_anchor, warm-start identity, mass policy — but no artifact's identity records that config as a hash. The release manifest records the values in prose (default_dataset, options), but there is no solver_config_sha256 you can compare across runs to answer "did the calibration config change?" from identity alone.

Why it matters (Build F evidence, framed accurately)

The campaign ran back-to-back invocations differing only in --l2-lambda (0.08 vs default 0.0) and --max-weight-ratio (5 vs 50). An l2_lambda=0.08 run that produced +258% income tax immediately preceded a default-λ dense run. When the default-λ run then failed a downstream gate, checkpoint bleed was a natural first hypothesis.

It was NOT the cause — attempt 5 was run in a fresh --checkpoint-root (clean room), and its dense result was byte-identical on the flagged masses to the prior attempt despite the clean room, proving the behavior was deterministic pipeline logic, not stale cache. But had the clean-room control not been run, the identical numbers could have been misattributed to a stale checkpoint. A solver_config_sha256 on the calibrated artifacts (and any calibration-stage cache added later) would make that question answerable from identity, without a control run.

Proposed fix

  • Add solver_config_sha256 (canonical-JSON hash of the full calibration option set) to the identity/manifest of the calibrated artifacts and any future calibration-stage checkpoint whose values depend on solver config.
  • Add a test pinning the invariant that the target-frame checkpoint is solver- and (given target_registry_version) feed-independent, so that independence is a stated contract rather than an accident.

Context

Found during the Build F certified-lineage rerun (populace#299). Not a blocker for that campaign's conclusions.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions