EDC Phases 2–4f: geometry features, distribution-free certificates, generalization & baselines

.claude/CLAUDE.md

@@ -26,6 +26,20 @@ inference-time descent** into distribution-free reliability certificates. Read
 
 ## Phase map (see docs/EXPERIMENTS.md)
 
-- Phase 0 skeleton ✅ / Phase 1 base reasoner ✅ (this pass)
-- Phase 2 geometry features · Phase 3 conformal + halting · Phase 4 experiments/OOD/Modal
-  · Phase 5 paper — stubbed, `NotImplementedError` with a phase note until built.
+- Phase 0 skeleton ✅ / Phase 1 base reasoner ✅ / Phase 2 geometry features ✅
+- Phase 3 conformal falsification + LTT abstention ✅ (E1: geometry beats energy, ΔAURC CI
+  excludes 0; F2/F3 regenerate from the ledger).
+- Phase 4a robustness + K-ablation ✅ (`run_sweep` + `aggregate` + `make_tables` T1/T2/T3 + S1
+  K-lift figure; S1: geometry lift grows monotonically with restarts, K=1≈EBT barely separates).
+- Phase 4b adaptive halting ✅ (opt-in per-step decode + `halting.adaptive` CRC + F4; H1: ~58%
+  compute saved at halting risk ≤ α, no accuracy loss). Both guarantees now live.
+- Phase 4c arithmetic figure set complete ✅ (F5 mechanism + F6 OOD stress; F6: selective risk
+  0.075 ID vs 0.762 OOD — guarantee breaks under shift, motivating abstention).
+- Phase 4d generalization ✅ (graph shortest-path task; E2: geometry beats energy 5/5 seeds,
+  ΔAURC +0.111 — discovery replicates on a 2nd structurally distinct family; per-task T1).
+- Phase 4e baseline stress test ✅ ⚠️ (MSP/temp/entropy; geometry beats scalar energy 5/5 but
+  does NOT beat softmax confidence — ties arith, loses graph). Key caveat; see EXPERIMENTS.md.
+- Phase 4f feature-group ablation ✅ (A1/T2b; basin agreement is the dominant driver, drop_energy≈full
+  so the win is non-energy features — rebuts "it's just energy").
+- Next: IRED landscape training (the priority lever from 4e) · E3/E4 · Modal · Phase 5 paper —
+  stubbed, `NotImplementedError` / Phase-1 placeholder until built.

README.md

@@ -58,7 +58,7 @@ PYTHONPATH=src python -m edc.cli smoke --config configs/smoke.toml
 src/edc/
   energy/       encoder → E_θ(h_x, z) → decoder            (JAX/Flax)
   inference/    K-restart Langevin descent + trajectory logging
-  geometry/     basin · curvature (HVP) · energy stats · dynamics   [Phase 2]
+  geometry/     basin · curvature (HVP) · energy stats · dynamics   ✓
   conformal/    Learn-then-Test (abstention) · Conformal Risk Control (halting)  [Phase 3]
   halting/      adaptive compute policy                    [Phase 3]
   tasks/        synthetic reasoning families (+ OOD splits)
@@ -67,12 +67,32 @@ src/edc/
 configs/  experiments/  analysis/  results/ledger.jsonl  tests/  docs/  paper/
 ```
 
-## Status
-
-Phase 0 (skeleton) and Phase 1 (runnable base reasoner) are in. Geometry features,
-the conformal layer, full experiments, and the paper are scaffolded and tracked in
-[`docs/EXPERIMENTS.md`](docs/EXPERIMENTS.md). Current state:
-[`docs/SESSION_HANDOFF.md`](docs/SESSION_HANDOFF.md).
+## Status (v0.0.1)
+
+The full method and its distribution-free certificate machinery are implemented and tested
+(offline, CPU): base reasoner (Phase 1), 14-feature landscape geometry (Phase 2), the conformal
+layer — split-conformal, Learn-then-Test **selective prediction/abstention**, and Conformal Risk
+Control **adaptive halting** (Phase 3, 4b) — a multi-seed experiment/sweep/table harness, and the
+figure set F2–F6 + S1, all regenerable from `results/ledger.jsonl`.
+
+**Findings so far (honest).**
+- **Geometry beats the EBT scalar energy** as a nonconformity score on two structurally distinct
+  tasks — arithmetic (E1) and graph shortest-path (E2) — with the paired-bootstrap ΔAURC CI
+  excluding 0 on 5/5 seeds each. This is the project's specific falsification target (invariant 8).
+- The restart geometry is the mechanism: the lift **grows monotonically with K** (S1 ablation);
+  K=1 (≈ EBT, no basin geometry) barely separates.
+- Both guarantees hold in-distribution (F2/F3 abstention validity, F4 halting saves ~58% compute
+  at a guaranteed error budget) and **break under distribution shift** (F6) — the argument for
+  abstention in critical systems.
+- **Key caveat (Phase 4e):** geometry does **not** beat a standard *softmax-confidence* baseline
+  (MSP / temperature-scaled / entropy) — it ties on arithmetic and loses on graph. So the current
+  contribution is the certificate machinery + beating the EBT scalar-energy signal specifically,
+  **not** a universal win over all confidence signals. The likely lever — a fully-learned
+  **IRED-style landscape** (vs the current supervised basin-center reasoner) — is the priority next
+  experiment.
+
+Experiments and figures/tables: [`docs/EXPERIMENTS.md`](docs/EXPERIMENTS.md). Rolling state and next
+steps: [`docs/SESSION_HANDOFF.md`](docs/SESSION_HANDOFF.md). Decisions: [`docs/DECISIONS.md`](docs/DECISIONS.md).
 
 ## License
 

analysis/aggregate.py

@@ -0,0 +1,118 @@
+"""Aggregate ``split='selective'`` ledger rows across seeds for the tables/figures. [Phase 4]
+
+Pure NumPy over already-computed per-seed metrics (invariant 6: everything derives from the
+ledger; no re-training, no re-bootstrapping). The multi-seed ΔAURC is summarised from each row's
+own point estimate + 95% CI — we report the mean±std over seeds and how many seeds' CIs exclude 0,
+rather than pooling raw scores across seeds (which would break the per-run exchangeability).
+"""
+
+from __future__ import annotations
+
+import numpy as np
+
+from edc.ledger import read_all
+
+
+def selective_rows(rows: list[dict] | None = None, task: str | None = None) -> list[dict]:
+    """Latest ``split=='selective'`` row per ``(config_hash, seed)`` (cf. latest_per_config).
+
+    ``task`` filters to one reasoning family (``row["task"]``) so multi-task ledgers do not
+    cross-contaminate per-K aggregates.
+    """
+    if rows is None:
+        rows = read_all()
+    latest: dict[tuple[str, int], dict] = {}
+    for r in rows:
+        if r.get("split") == "selective" and (task is None or r.get("task") == task):
+            latest[(r["config_hash"], r["seed"])] = r  # later rows win
+    return list(latest.values())
+
+
+def tasks_present(rows: list[dict] | None = None) -> list[str]:
+    """Sorted distinct task names among selective rows."""
+    return sorted({r.get("task") for r in selective_rows(rows)} - {None})
+
+
+def _k(row: dict) -> int:
+    m = row["metrics"]
+    return int(m.get("k_restarts", row["config"]["inference"]["k_restarts"]))
+
+
+def by_k(rows: list[dict] | None = None, task: str | None = None) -> dict[int, list[dict]]:
+    """Group selective rows by ``k_restarts`` (ascending), optionally filtered to one ``task``."""
+    out: dict[int, list[dict]] = {}
+    for r in selective_rows(rows, task=task):
+        out.setdefault(_k(r), []).append(r)
+    return dict(sorted(out.items()))
+
+
+def _ms(vals) -> tuple[float, float]:
+    a = np.asarray(vals, dtype=float)
+    return float(a.mean()), float(a.std(ddof=0))
+
+
+def aggregate_cell(rows_for_k: list[dict]) -> dict:
+    """Aggregate one K's per-seed rows into mean/std stats + the multi-seed verdict."""
+    ms = [r["metrics"] for r in rows_for_k]
+    dvb = [m["delta_aurc_vs_best_energy"] for m in ms]  # each [delta, lo, hi]
+    n = len(ms)
+    delta_mean, delta_std = _ms([d[0] for d in dvb])
+    ci_excludes_0 = sum(1 for d in dvb if d[1] > 0.0)  # lo > 0 => positive & CI clears 0
+    geo_mean, geo_std = _ms([m["aurc"]["geometry"] for m in ms])
+    best_mean, best_std = _ms([m["aurc"][m["best_energy_baseline"]] for m in ms])
+    # vs the strongest of ALL baselines (energy + softmax); older rows fall back to energy.
+    dvbl = [m.get("delta_aurc_vs_best_baseline", m["delta_aurc_vs_best_energy"]) for m in ms]
+    dbl_mean, dbl_std = _ms([d[0] for d in dvbl])
+    bl_ci = sum(1 for d in dvbl if d[1] > 0.0)
+    return {
+        "task": rows_for_k[0].get("task"),
+        "k_restarts": _k(rows_for_k[0]),
+        "n_seeds": n,
+        "seeds": sorted(r["seed"] for r in rows_for_k),
+        "run_ids": [r["run_id"] for r in rows_for_k],
+        "accuracy_id": _ms([m["accuracy_id"] for m in ms]),
+        "aurc_geometry": (geo_mean, geo_std),
+        "aurc_best_energy": (best_mean, best_std),
+        "best_energy_names": sorted({m["best_energy_baseline"] for m in ms}),
+        "best_baseline_names": sorted(
+            {m.get("best_baseline", m["best_energy_baseline"]) for m in ms}),
+        "delta_aurc": (delta_mean, delta_std),
+        "seeds_ci_excludes_0": ci_excludes_0,
+        "geometry_wins_all": bool(ci_excludes_0 == n and delta_mean > 0),
+        "delta_aurc_baseline": (dbl_mean, dbl_std),
+        "seeds_ci_excludes_0_baseline": bl_ci,
+        # True only when every row genuinely carried the Phase-4e baseline field (not a fallback).
+        "has_baseline": all("delta_aurc_vs_best_baseline" in m for m in ms),
+        # Feature-group leave-one-out ablation (Phase 4f), mean/std per subset over seeds.
+        "feature_ablation": (
+            {k: _ms([m["feature_ablation"][k] for m in ms])
+             for k in ms[0]["feature_ablation"]}
+            if all("feature_ablation" in m for m in ms) else {}),
+        "ltt_coverage": _ms([m["ltt"]["coverage"] for m in ms]),
+        "ltt_selective_risk": _ms([m["ltt"]["selective_risk"] for m in ms]),
+        "ltt_abstain_rate": _ms([m["ltt"]["abstain_rate"] for m in ms]),
+    }
+
+
+def aggregate_by_k(rows: list[dict] | None = None, task: str | None = None) -> dict[int, dict]:
+    """``{K: aggregate_cell(...)}`` for every K present (optionally filtered to one ``task``)."""
+    return {k: aggregate_cell(v) for k, v in by_k(rows, task=task).items()}
+
+
+def headline_cell(rows: list[dict] | None = None, task: str | None = None) -> dict:
+    """Aggregate the headline seed-sweep for a task: the largest single-config seed group.
+
+    Prefers rows carrying the Phase-4e baseline field, then picks the ``config_hash`` group with the
+    most seeds — the multi-seed sweep — so T1 is not polluted by mixing fold sizes / configs that
+    happen to share a K value. Falls back to all rows for the task if none carry the field.
+    """
+    sel = selective_rows(rows, task=task)
+    with_bl = [r for r in sel if "delta_aurc_vs_best_baseline" in r["metrics"]]
+    pool = with_bl or sel
+    # Group by the experiment config *excluding seed* (config_hash bakes in the seed). (K, n_test)
+    # separates a seed-sweep (many seeds, one fold size) from one-off full-fold runs at the same K.
+    groups: dict[tuple, list[dict]] = {}
+    for r in pool:
+        m = r["metrics"]
+        groups.setdefault((_k(r), m.get("n_test")), []).append(r)
+    return aggregate_cell(max(groups.values(), key=len))

analysis/make_figures.py

@@ -7,16 +7,41 @@
 
 from __future__ import annotations
 
+from pathlib import Path
+
+from edc import plotting
 from edc.ledger import read_all
 
+FIG_DIR = Path(__file__).resolve().parent / "figures"
+
+# (filename, plotting fn). F1 (schematic) is TikZ; F4/F6 (halting, OOD stress) are Phase 4/5.
+_FIGURES = [
+    ("F2_risk_coverage.png", plotting.risk_coverage_curve),
+    ("F3_coverage_validity.png", plotting.coverage_validity),
+    ("F4_halting_pareto.png", plotting.halting_pareto),
+    ("F5_feature_diagnostics.png", plotting.feature_diagnostics),
+    ("F6_ood_stress.png", plotting.ood_stress),
+    ("S1_k_restart_lift.png", plotting.k_restart_lift),
+]
+
 
 def main() -> int:
     rows = read_all()
     print(f"[figures] {len(rows)} ledger rows available.")
-    raise NotImplementedError(
-        "Phase 3/5: emit F1..F6 via edc.plotting from the ledger. "
-        "Run `make smoke` first to populate results/ledger.jsonl."
-    )
+    FIG_DIR.mkdir(parents=True, exist_ok=True)
+    n_ok = 0
+    for fname, fn in _FIGURES:
+        out = FIG_DIR / fname
+        try:
+            fn(rows, str(out))
+        except (ValueError, NotImplementedError, ImportError) as e:
+            print(f"[figures] skip {fname}: {e}")
+            continue
+        n_sel = sum(1 for r in rows if r.get("split") == "selective")
+        print(f"[figures] wrote {out}  (from {n_sel} selective rows)")
+        n_ok += 1
+    print(f"[figures] {n_ok}/{len(_FIGURES)} figures generated -> {FIG_DIR}")
+    return 0
 
 
 if __name__ == "__main__":

analysis/make_tables.py

@@ -1,18 +1,155 @@
-"""Regenerate LaTeX tables (T1..T3) from results/ledger.jsonl. [Phase 3/5]
+"""Regenerate LaTeX tables (T1..T3) from results/ledger.jsonl. [Phase 4]
 
-Main-results, ablation, and reproducibility-appendix tables, emitted to paper/. Reads only the
-ledger. See paper/README.md for the table->claim map.
+T1 main results (primary K, aggregated over seeds), T2 K-ablation, T3 reproducibility appendix.
+Reads only the ledger (invariant 6) and writes ``\\input``-able ``.tex`` to ``paper/tables/``, each
+with a provenance comment naming the run_ids it consumed. See paper/README.md for the table->claim
+map.
 """
 
 from __future__ import annotations
 
+from edc.config import REPO_ROOT
 from edc.ledger import read_all
 
+try:  # sibling import when run as a script (analysis/ on sys.path)
+    from aggregate import aggregate_by_k, headline_cell, selective_rows, tasks_present
+except ImportError:  # pragma: no cover
+    from analysis.aggregate import aggregate_by_k, headline_cell, selective_rows, tasks_present
+
+TABLE_DIR = REPO_ROOT / "paper" / "tables"
+
+
+def _esc(s: str) -> str:
+    return str(s).replace("_", r"\_")
+
+
+def _pm(mean_std: tuple[float, float], nd: int = 3) -> str:
+    m, s = mean_std
+    return f"${m:.{nd}f} \\pm {s:.{nd}f}$"
+
+
+def _tabular(header: list[str], rows: list[list[str]], colspec: str, caption: str,
+             label: str, provenance: str) -> str:
+    lines = [
+        f"% auto-generated by analysis/make_tables.py — do not edit. {provenance}",
+        r"\begin{table}[t]", r"  \centering", f"  \\caption{{{caption}}}", f"  \\label{{{label}}}",
+        f"  \\begin{{tabular}}{{{colspec}}}", r"    \toprule",
+        "    " + " & ".join(header) + r" \\", r"    \midrule",
+    ]
+    lines += ["    " + " & ".join(r) + r" \\" for r in rows]
+    lines += [r"    \bottomrule", r"  \end{tabular}", r"\end{table}", ""]
+    return "\n".join(lines)
+
+
+def _t1(rows: list[dict]) -> str:
+    """Main results, one row per task (each at its own primary K, aggregated over seeds).
+
+    Reports geometry's ΔAURC against **both** the best raw-energy baseline (invariant-8 sacred test)
+    and the best of *all* baselines (energy + MSP + temperature + entropy), with the seeds-win count
+    for the stronger, all-baseline comparison.
+    """
+    body, all_ids = [], []
+    for task in tasks_present(rows):
+        a = headline_cell(rows, task=task)             # largest same-config baseline-carrying sweep
+        win = f"{a['seeds_ci_excludes_0_baseline']}/{a['n_seeds']}"
+        body.append([
+            _esc(task), str(a["k_restarts"]), _pm(a["accuracy_id"], 2), _pm(a["aurc_geometry"]),
+            _pm(a["delta_aurc"]), _pm(a["delta_aurc_baseline"]), win, _pm(a["ltt_coverage"], 2),
+        ])
+        all_ids += a["run_ids"]
+    header = ["task", "$K$", "acc", "AURC geom", r"$\Delta$AURC vs energy",
+              r"$\Delta$AURC vs best base", "seeds win", "LTT cov"]
+    return _tabular(
+        header, body, "lrcccccc",
+        "Main selective-prediction results across reasoning families (mean$\\pm$std over seeds): "
+        "geometry's $\\Delta$AURC over the best raw-energy baseline and over the best of all "
+        "baselines (energy, MSP, temperature-scaled MSP, predictive entropy). $\\Delta$AURC $>0$ "
+        "with the CI clearing 0 means geometry wins; ``seeds win`` counts seeds beating the best "
+        "overall baseline.",
+        "tab:main", f"run_ids={all_ids}")
+
+
+def _t2(agg: dict[int, dict]) -> str:
+    rows = []
+    all_ids: list[str] = []
+    for k, a in agg.items():
+        rows.append([
+            str(k), _pm(a["aurc_geometry"]), _pm(a["aurc_best_energy"]), _pm(a["delta_aurc"]),
+            f"{a['seeds_ci_excludes_0']}/{a['n_seeds']}", _pm(a["ltt_coverage"], 2),
+        ])
+        all_ids += a["run_ids"]
+    header = ["$K$", "AURC geom", "AURC energy", r"$\Delta$AURC", "seeds win", "LTT cov"]
+    return _tabular(
+        header, rows, "rccccc",
+        "Restart ablation: does landscape geometry across $K$ restarts add signal over a single "
+        "descent ($K{=}1\\approx$ EBT)? Lift is $\\Delta$AURC $>0$ with the CI clearing 0.",
+        "tab:kablation", f"run_ids={all_ids}")
+
+
+def _t2b(rows: list[dict]) -> str:
+    """Feature-group leave-one-out ablation (A1): AURC per subset, one column per task.
+
+    ``full`` is the whole geometry vector; ``drop_energy`` is geometry *without* energy columns —
+    if it stays near ``full`` (and below raw energy), the basin/curvature/dynamics features are
+    genuinely carrying the win, not merely re-using energy.
+    """
+    tasks = tasks_present(rows)
+    cells = {t: headline_cell(rows, task=t).get("feature_ablation", {}) for t in tasks}
+    variants = ["full", "drop_basin", "drop_energy", "drop_curv", "drop_dynamics"]
+    body = []
+    for v in variants:
+        row = [_esc(v)]
+        for t in tasks:
+            fa = cells[t].get(v)
+            row.append(_pm(fa) if fa else "--")
+        body.append(row)
+    header = ["subset (AURC, lower=better)", *[_esc(t) for t in tasks]]
+    colspec = "l" + "c" * len(tasks)
+    return _tabular(
+        header, body, colspec,
+        "Feature-group leave-one-out ablation: test AURC of the geometry mapper refit on subsets "
+        "of the 14 features (mean$\\pm$std over seeds). ``drop\\_X`` removes that group; comparing "
+        "``drop\\_energy`` to ``full`` isolates the non-energy geometry contribution.",
+        "tab:ablation", "feature-group leave-one-out")
+
+
+def _t3(rows: list[dict]) -> str:
+    sel = selective_rows(rows)
+    env = sel[-1]["env"] if sel else {}
+    body = [
+        [_esc(r["run_id"]), str(r["seed"]),
+         str(r["metrics"].get("k_restarts", "?")), _esc(r["config_hash"]), _esc(r["git_sha"])]
+        for r in sorted(sel, key=lambda r: (r["metrics"].get("k_restarts", 0), r["seed"]))
+    ]
+    caption = (f"Reproducibility appendix. Env: python {env.get('python', '?')}, "
+               f"jax {env.get('jax', '?')} ({env.get('jax_backend', '?')}). "
+               f"Every row regenerates from seed + config via \\texttt{{run\\_sweep}}.")
+    return _tabular(["run\\_id", "seed", "$K$", "config hash", "git sha"], body, "lrrll",
+                    caption, "tab:repro", f"n_rows={len(body)}")
+
 
 def main() -> int:
     rows = read_all()
-    print(f"[tables] {len(rows)} ledger rows available.")
-    raise NotImplementedError("Phase 3/5: emit T1..T3 LaTeX from the ledger.")
+    tasks = tasks_present(rows)
+    print(f"[tables] {len(rows)} ledger rows; tasks = {tasks}")
+    if not tasks:
+        print("[tables] no split='selective' rows — run experiments/run_sweep.py first.")
+        return 0
+    TABLE_DIR.mkdir(parents=True, exist_ok=True)
+
+    # T2 (K-ablation) uses whichever task has a K-sweep (>1 distinct K); default the first.
+    kablation_task = next((t for t in tasks if len(aggregate_by_k(rows, task=t)) > 1), tasks[0])
+    outputs = {
+        "T1_main.tex": _t1(rows),
+        "T2_kablation.tex": _t2(aggregate_by_k(rows, task=kablation_task)),
+        "T2b_feature_ablation.tex": _t2b(rows),
+        "T3_repro.tex": _t3(rows),
+    }
+    for name, tex in outputs.items():
+        (TABLE_DIR / name).write_text(tex)
+        print(f"[tables] wrote {TABLE_DIR / name}")
+    print(f"[tables] {len(outputs)} tables -> {TABLE_DIR}  (K-ablation task={kablation_task})")
+    return 0
 
 
 if __name__ == "__main__":