EDC Phases 2–4f: geometry features, distribution-free certificates, generalization & baselines - #1
Merged
Merged
Conversation
…TT abstention Phase 2 (geometry features): basin/energy/curvature/dynamics -> 14-feature vector (geometry/features.py), batched curvature in the JAX core, TrajectoryRecord.h_x, single_feature_auroc, and the `edc.cli geometry` diagnostic. Phase 3 (conformal + falsification + abstention): - eval/metrics.py: risk_coverage_curve, tie-robust aurc, paired_bootstrap_delta_aurc (the invariant-8 falsification test), selective_accuracy_at_coverage, ece. - conformal/: logistic nonconformity mapper + 1-rho_basin fallback; selective_predict; Hoeffding-Bentkus p-value + Bonferroni LTT calibration; CRC finite-sample calibrator. - eval/evaluate.py: disjoint fit/calib/test folds; geometry vs raw-energy baselines; ΔAURC bootstrap + LTT abstention. experiments/run_experiment.py wired to the ledger. - tasks/arithmetic.py: ood_max_operand magnitude-shift OOD. - plotting.py + analysis/make_figures.py: F2 (risk-coverage) + F3 (coverage validity). E1 result (run_id c1557364f51d, seed 0, ID acc 0.81): the learned geometry mapper beats the best raw-energy baseline — ΔAURC +0.074, 95% CI [+0.054, +0.093] (excludes 0). LTT answers 69% at selective risk 0.075 <= alpha=0.1. Tests: 52 passing, offline + CPU. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Verifies the Phase-3 discovery is real (multi-seed) and mechanism-driven (restarts matter).
- experiments/run_sweep.py: grid+override expansion; every cell reuses the factored
run_experiment.run_and_append (single train->evaluate->ledger path). evaluate() gains
include_ood to skip the OOD fold in the sweep.
- analysis/aggregate.py: group selective ledger rows by (K, seed); summarise multi-seed
ΔAURC from each run's own point estimate + 95% CI (mean±std, #seeds whose CI excludes 0).
- analysis/make_tables.py: T1 (main, primary K), T2 (K-ablation), T3 (repro) -> paper/tables/.
- plotting.k_restart_lift + make_figures: S1 K-lift figure. plotting._selective_row now
picks the full-fold headline run for F2/F3 (was fragile "latest row").
- configs/sweeps/k_restarts.toml: K in {1,2,4,8,16} x 5 seeds, reduced folds for CPU.
- tests: run_sweep grid expansion, aggregation, table smoke, evaluate include_ood (+9 tests).
S1 result (5 seeds/K): the geometry lift over raw energy grows monotonically with K —
K=1 (a single descent, no basin geometry, ≈EBT) separates on only 1/5 seeds; K>=8 on 5/5,
ΔAURC +0.083 at K=16. The restart geometry is the mechanism; the "K>1 gives no lift over
K=1" falsifier does not fire. Tests: 61 passing, offline + CPU.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes the "two guarantees" pillar: LTT abstention (Phase 3) + CRC adaptive halting. - Inference core (opt-in, off by default so E1/sweep are unchanged): langevin_descent(record_z) emits the latent trajectory; restarts.solve(record_steps) decodes it per step into a new optional TrajectoryRecord.step_pred (B,K,T+1). - halting/adaptive.py: per_step_agreement (basin agreement per step), halting_policy, halting_losses (L = 1[early best-of-N != full best-of-N]), halted_predictions, and calibrate via the existing conformal.crc.calibrate with a lambda = 1 - tau reparametrisation. Reports a risk-monotone-in-tau flag (CRC's assumption); crc scans conservatively; LTT is the documented fallback if materially non-monotone. - eval/evaluate_halting.py + experiments/run_halting.py (split="halting") + configs/experiments/arithmetic_halting.toml (modest folds for per-step decode memory). - plotting.halting_pareto (F4) + make_figures; plotting._halting_row selector. - tests: halting math exact values + CRC control, per-step recording correctness, e2e smoke (+15 tests; 70 passing total, offline + CPU). H1 result (run_id 685d46f13e58, seed 0, alpha 0.1): CRC chose tau_hat=0.917 with the empirical risk monotone in tau; the descent uses 42% of the step budget (57.8% compute saved) at a halting risk of 0.9% <= alpha=10% with no accuracy loss (full 0.804 -> halted 0.807). F4 Pareto: the CRC point sits at the elbow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…mechanism) - evaluate.py: two new metric blocks. F6 `ood_validity`/`ood_ltt` calibrate the LTT selective-risk threshold on the ID calib fold and apply it to the OOD test fold (guarded by include_ood) to exhibit the guarantee failing under shift. F5 `feature_diagnostics`: per-feature single_feature_auroc + 20-bin correct/incorrect histograms (compact, ledger-regenerable). - plotting.py: feature_diagnostics (F5, separation bar + basin/curvature histograms) and ood_stress (F6, achieved-vs-target risk, ID on-diagonal vs OOD above) + _halting_row-style reuse of _selective_row; registered in make_figures. - tests: test_ood_validity (F5/F6 block structure, e2e), test_plotting_f5_f6 (render + graceful skip). +4 tests; 74 passing, offline + CPU. F6 result (run_id 1acd65c19072, seed 0): at alpha=0.1 the ID-calibrated threshold achieves selective risk 0.075 in-distribution (valid) but 0.762 on OOD (answers 66% of OOD inputs at 76% error) — the guarantee breaks once exchangeability is violated, which is the concrete argument for abstention/shift-detection in critical systems. F5: basin-agreement features separate correct from incorrect best on arithmetic; curvature separates weakly (reported honestly). The arithmetic figure set (F2-F6, S1) and tables (T1-T3) are now complete. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tests whether "geometry beats scalar energy" replicates beyond additive arithmetic on a structurally distinct relational/combinatorial family. - tasks/graph_planning.py: GraphPlanningTask — random Erdős–Rényi graph, random (source,target), BFS shortest-path-length label (capped; class 0 = unreachable/farther), fixed feature_dim via max_nodes-padded adjacency + source/target one-hots, larger-graph (size-shift) OOD. Registered. - analysis/aggregate.py: task-aware (selective_rows/by_k/aggregate_by_k gain a `task` filter + tasks_present) so multi-task ledgers don't cross-contaminate per-K aggregates. - analysis/make_tables.py: T1 is now one row per task (cross-task main result). - plotting.k_restart_lift: restricted to the single K-sweep task so graph's fixed-K rows can't contaminate the arithmetic S1 figure. - configs: graph_selective.toml (tuned to ID~0.72, majority baseline ~0.41) + sweeps/graph_seeds.toml (5 seeds). tests: test_graph_task (BFS/shapes/OOD/determinism), extended test_aggregate + test_make_tables for multi-task. 80 passing, offline + CPU. E2 result (5 seeds): geometry beats the best raw-energy baseline on 5/5 seeds — AURC 0.159 vs 0.270, ΔAURC +0.111 ± 0.054, every CI excludes 0. With arithmetic (ΔAURC +0.083 ± 0.034, 5/5), T1 now shows the discovery holds across two structurally distinct tasks — not an arithmetic artifact. (On graph, LTT abstention at α=0.1 with reduced calib folds certifies zero coverage — conservative, not a failure; ΔAURC is the threshold-free generalization evidence.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stress-tests "geometry beats energy" against the standard confidence baselines the research plan calls for, and finds a real boundary. - eval/baselines.py: MSP, temperature-scaled MSP (T fit on the fit fold, invariant 7), predictive entropy — from the mean decoder logits over K restarts. - eval/evaluate.py: wire the three baselines into scores_for + AURC + F2; add delta_aurc_vs_best_baseline (geometry vs the best of ALL baselines) + geometry_wins_vs_baseline, alongside the sacred delta_aurc_vs_best_energy (invariant 8, unchanged). - analysis/aggregate.py: baseline ΔAURC aggregation + has_baseline flag + headline_cell (largest same-config, baseline-carrying seed sweep). make_tables T1: per-task ΔAURC vs energy AND vs best baseline. plotting: MSP/entropy/temp lines in F2. - configs/sweeps/arith_seeds.toml (5-seed arithmetic sweep w/ baselines). tests: test_baselines + extended aggregate/evaluate. 85 passing, offline + CPU. Result (5 seeds/task): geometry beats scalar ENERGY on both tasks (invariant 8 holds: +0.085 arithmetic, +0.111 graph, 5/5) BUT does NOT beat softmax confidence — ΔAURC vs best baseline is -0.011 (arithmetic, 0/5) and -0.034 (graph, 0/5); MSP/entropy tie-or-beat geometry (F2 shows them overlapping, all far below energy). The "geometry is the best nonconformity score" claim fails against softmax confidence under the Phase-1 basin-center reasoner (shallow decoder -> softmax already informative). Reported honestly. Next: IRED landscape training as the lever (SESSION_HANDOFF). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Answers "does geometry's win over energy just re-use the energy features it contains?" — no. - eval/evaluate.py: _feature_ablation refits the logistic geometry mapper on feature subsets (leave-one-group-out + group-only across basin/energy/curvature/dynamics) on the fit fold and scores the test fold; stored as the feature_ablation metric block. - analysis/aggregate.py: aggregate feature_ablation across seeds (mean/std per subset). - analysis/make_tables.py: T2b_feature_ablation.tex (per-task AURC by subset). - tests: ablation block present/finite; aggregation + table well-formed. 86 passing, offline+CPU. - README status refreshed to v0.0.1 with the honest findings incl. the 4e softmax caveat. Result (5 seeds/task): drop_energy AURC ~= full (arith 0.062 vs 0.059; graph 0.162 vs 0.159), both far below raw energy (0.144 / 0.270) — the win over energy comes from the NON-energy features. Basin agreement is the dominant driver (dropping it nearly doubles arithmetic AURC toward energy); descent dynamics helps too (most on graph); curvature/energy contribute little. Consistent with F5; does not change the 4e caveat (basin/dynamics beat energy, not softmax). Full IRED landscape training remains the documented priority lever, scoped as the next research phase (not in v0.0.1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds the Energy Descent Certificates method end-to-end on top of the Phase-0/1 scaffold: turning the geometry of an EBM reasoner's inference-time descent into distribution-free reliability certificates, then rigorously stress-testing the discovery.
What's in this PR
Phase 2 — geometry features. 14-feature per-input vector (basin agreement · terminal-energy stats · curvature via HVPs · descent dynamics);
edc.cli geometrydiagnostic.Phase 3 — conformal falsification + abstention. AURC / risk–coverage / paired-bootstrap ΔAURC; logistic nonconformity mapper (+
1−ρ_basinfallback); split-conformal, Learn-then-Test selective prediction (Hoeffding–Bentkus + FWER), CRC calibrator; disjoint fit/calib/test pipeline; F2/F3.Phase 4a–4f.
run_sweep+ T1/T2/T3 tables.halting.adaptive, F4.Findings (honest)
drop_energy ≈ full).Invariants / quality
JAX confined to the core; offline + CPU-only tests (86 passing), deterministic via
edc.seeding; append-only ledger; every figure/table regenerates fromresults/ledger.jsonl.ruffclean.🤖 Generated with Claude Code