Phase 4k: richer geometry (full Hessian spectrum + mode connectivity)

.claude/CLAUDE.md

@@ -51,4 +51,11 @@ inference-time descent** into distribution-free reliability certificates. Read
 - Phase 4j complementarity ✅ (T4): geometry adds *conditional* signal over softmax ONLY where it
   wins head-to-head (arithmetic-IRED 4/5); on graph-IRED the combined [geom+softmax] mapper ≈ softmax
   (0/5) — the graph tie is **redundancy, not complementarity**.
-- Next: richer geometry (full spectrum / mode connectivity) · E3/E4 · Modal · scale-up.
+- Phase 4k richer geometry ✅ (T5): opt-in `[eval] richer_geometry=true` appends the full Hessian
+  spectrum (`spectrum/*`, exact d×d eigh) + mode-connectivity barriers (`connect/*`) after the base
+  14. Re-ran the 4j test on all 4 cells: richer geometry helps **nowhere** — graph-IRED conditional
+  signal stays −0.002 (0/5). So the **graph redundancy is a task property, not thin features**; it's
+  a wash where geometry already wins (arith-IRED +0.007 4/5) and slightly worse on fixed-bowl cells.
+  Aggregation is feature-set-aware (`feature_set_of`, canonical tables default `feature_set="base"`)
+  so richer never pollutes T1/T2/T4. JAX for spectrum/connectivity lives in `curvature.py` (inv. 1).
+- Next: E3/E4 (a 3rd/4th task to characterize *when* geometry beats softmax) · Modal · scale-up.

analysis/aggregate.py

@@ -19,19 +19,30 @@ def objective_of(row: dict) -> str:
         "objective", "basin_center")
 
 
+def feature_set_of(row: dict) -> str:
+    """Geometry feature set behind a row: ``base`` (14 features) or ``richer`` (Phase 4k).
+
+    Pre-4k rows lack the field and never enabled richer geometry, so they read as ``base``.
+    """
+    return row["metrics"].get("feature_set") or (
+        "richer" if row["config"].get("eval", {}).get("richer_geometry") else "base")
+
+
 def selective_rows(rows: list[dict] | None = None, task: str | None = None,
-                   objective: str | None = None) -> list[dict]:
+                   objective: str | None = None, feature_set: str | None = None) -> list[dict]:
     """Latest ``split=='selective'`` row per ``(config_hash, seed)`` (cf. latest_per_config).
 
-    ``task`` filters to one reasoning family and ``objective`` to one reasoner (``basin_center`` vs
-    ``ired``), so multi-task / multi-reasoner ledgers do not cross-contaminate per-K aggregates.
+    ``task`` filters to one reasoning family, ``objective`` to one reasoner (``basin_center`` vs
+    ``ired``), and ``feature_set`` to one geometry set (``base`` vs ``richer``), so multi-task /
+    multi-reasoner / multi-feature-set ledgers do not cross-contaminate per-K aggregates.
     """
     if rows is None:
         rows = read_all()
     latest: dict[tuple[str, int], dict] = {}
     for r in rows:
         if (r.get("split") == "selective" and (task is None or r.get("task") == task)
-                and (objective is None or objective_of(r) == objective)):
+                and (objective is None or objective_of(r) == objective)
+                and (feature_set is None or feature_set_of(r) == feature_set)):
             latest[(r["config_hash"], r["seed"])] = r  # later rows win
     return list(latest.values())
 
@@ -46,10 +57,15 @@ def _k(row: dict) -> int:
     return int(m.get("k_restarts", row["config"]["inference"]["k_restarts"]))
 
 
-def by_k(rows: list[dict] | None = None, task: str | None = None) -> dict[int, list[dict]]:
-    """Group selective rows by ``k_restarts`` (ascending), optionally filtered to one ``task``."""
+def by_k(rows: list[dict] | None = None, task: str | None = None,
+         feature_set: str | None = "base") -> dict[int, list[dict]]:
+    """Group selective rows by ``k_restarts`` (ascending), optionally filtered to one ``task``.
+
+    Defaults to the canonical ``base`` feature set so the K-ablation is never contaminated by
+    Phase-4k ``richer`` rows (which share K with their base counterparts).
+    """
     out: dict[int, list[dict]] = {}
-    for r in selective_rows(rows, task=task):
+    for r in selective_rows(rows, task=task, feature_set=feature_set):
         out.setdefault(_k(r), []).append(r)
     return dict(sorted(out.items()))
 
@@ -100,31 +116,38 @@ def aggregate_cell(rows_for_k: list[dict]) -> dict:
         # True only when every row genuinely carried the Phase-4e baseline field (not a fallback).
         "has_baseline": all("delta_aurc_vs_best_baseline" in m for m in ms),
         # Feature-group leave-one-out ablation (Phase 4f), mean/std per subset over seeds.
+        # Aggregate only the ablation subsets present in *every* row (base and richer rows carry
+        # different group sets), so a heterogeneous cell can never KeyError.
         "feature_ablation": (
             {k: _ms([m["feature_ablation"][k] for m in ms])
-             for k in ms[0]["feature_ablation"]}
+             for k in ms[0]["feature_ablation"]
+             if all(k in m["feature_ablation"] for m in ms)}
             if all("feature_ablation" in m for m in ms) else {}),
         "ltt_coverage": _ms([m["ltt"]["coverage"] for m in ms]),
         "ltt_selective_risk": _ms([m["ltt"]["selective_risk"] for m in ms]),
         "ltt_abstain_rate": _ms([m["ltt"]["abstain_rate"] for m in ms]),
     }
 
 
-def aggregate_by_k(rows: list[dict] | None = None, task: str | None = None) -> dict[int, dict]:
-    """``{K: aggregate_cell(...)}`` for every K present (optionally filtered to one ``task``)."""
-    return {k: aggregate_cell(v) for k, v in by_k(rows, task=task).items()}
+def aggregate_by_k(rows: list[dict] | None = None, task: str | None = None,
+                   feature_set: str | None = "base") -> dict[int, dict]:
+    """``{K: aggregate_cell(...)}`` per K present (filtered to a ``task`` / ``feature_set``)."""
+    return {k: aggregate_cell(v) for k, v in by_k(rows, task=task, feature_set=feature_set).items()}
 
 
 def headline_cell(rows: list[dict] | None = None, task: str | None = None,
-                  objective: str = "basin_center") -> dict:
+                  objective: str = "basin_center", feature_set: str | None = "base") -> dict:
     """Aggregate the headline seed-sweep for one task + reasoner (largest single-config seed group).
 
     T1 uses the canonical ``basin_center`` reasoner (pass ``objective="ired"`` to report the learned
-    landscape separately). Prefers rows carrying the Phase-4e baseline field, then picks the config
-    group (K, n_test) with the most seeds — the multi-seed sweep — so the cell is not polluted by
-    mixing fold sizes / reasoners that happen to share a K value.
+    landscape separately) and the canonical ``base`` feature set (pass ``feature_set="richer"`` for
+    the Phase-4k richer-geometry cells) — so richer rows never pollute the base headline. Prefers
+    rows carrying the Phase-4e baseline field, then picks the config group (K, n_test) with the most
+    seeds — the multi-seed sweep — so the cell is not polluted by mixing fold sizes / reasoners that
+    happen to share a K value.
     """
-    sel = selective_rows(rows, task=task, objective=objective) or selective_rows(rows, task=task)
+    sel = (selective_rows(rows, task=task, objective=objective, feature_set=feature_set)
+           or selective_rows(rows, task=task, feature_set=feature_set))
     with_bl = [r for r in sel if "delta_aurc_vs_best_baseline" in r["metrics"]]
     pool = with_bl or sel
     # Group by the experiment config *excluding seed* — (K, n_test) separates a seed-sweep (many

analysis/make_figures.py

@@ -12,6 +12,11 @@
 from edc import plotting
 from edc.ledger import read_all
 
+try:  # sibling import when run as a script (analysis/ on sys.path)
+    from aggregate import feature_set_of
+except ImportError:  # pragma: no cover
+    from analysis.aggregate import feature_set_of
+
 FIG_DIR = Path(__file__).resolve().parent / "figures"
 
 # (filename, plotting fn). F1 (schematic) is TikZ; F4/F6 (halting, OOD stress) are Phase 4/5.
@@ -26,8 +31,10 @@
 
 
 def main() -> int:
-    rows = read_all()
-    print(f"[figures] {len(rows)} ledger rows available.")
+    # Canonical figures are all base-feature results; exclude Phase-4k richer rows so they never
+    # dilute the pooled figures (e.g. S1's K=12 point). The richer verdict lives in table T5.
+    rows = [r for r in read_all() if feature_set_of(r) == "base"]
+    print(f"[figures] {len(rows)} base-feature ledger rows available.")
     FIG_DIR.mkdir(parents=True, exist_ok=True)
     n_ok = 0
     for fname, fn in _FIGURES:

analysis/make_tables.py

@@ -14,6 +14,7 @@
 try:  # sibling import when run as a script (analysis/ on sys.path)
     from aggregate import (
         aggregate_by_k,
+        feature_set_of,
         headline_cell,
         objective_of,
         selective_rows,
@@ -22,6 +23,7 @@
 except ImportError:  # pragma: no cover
     from analysis.aggregate import (
         aggregate_by_k,
+        feature_set_of,
         headline_cell,
         objective_of,
         selective_rows,
@@ -156,6 +158,41 @@ def _t4(rows: list[dict]) -> str:
         "tab:complementarity", f"run_ids={all_ids}")
 
 
+def _t5(rows: list[dict]) -> str:
+    """Richer geometry (Phase 4k): does the full Hessian spectrum + mode connectivity add signal?
+
+    Two rows per (task, reasoner) — ``base`` (14 features) vs ``richer`` (14 + spectrum + connect) —
+    so the reader sees directly whether the richer geometry moves the head-to-head win over softmax
+    and the conditional ``geom adds`` signal, especially on the graph cell where geometry ties.
+    """
+    body, all_ids = [], []
+    for task in tasks_present(rows):
+        for o in sorted({objective_of(r) for r in selective_rows(rows, task=task)}):
+            sets = {feature_set_of(r) for r in selective_rows(rows, task=task, objective=o)}
+            if "richer" not in sets:                      # only cells with a richer run
+                continue
+            for fs in ("base", "richer"):
+                if fs not in sets:
+                    continue
+                a = headline_cell(rows, task=task, objective=o, feature_set=fs)
+                adds = (f"{a['seeds_ci_excludes_0_geom_adds']}/{a['n_seeds']}"
+                        if a.get("has_geom_adds") else "--")
+                body.append([
+                    _esc(task), _esc(o), fs, _pm(a["aurc_geometry"]), _pm(a["delta_aurc_baseline"]),
+                    _pm(a["delta_aurc_geom_adds"]) if a.get("has_geom_adds") else "--", adds,
+                ])
+                all_ids += a["run_ids"]
+    header = ["task", "reasoner", "features", "AURC geom", r"$\Delta$AURC vs softmax",
+              r"$\Delta$AURC geom \emph{adds}", "seeds add"]
+    return _tabular(
+        header, body, "lllcccc",
+        "Richer geometry (mean$\\pm$std over seeds): the base 14-feature vector vs the base plus "
+        "the full Hessian spectrum and mode-connectivity barriers. If ``richer`` does not raise "
+        "$\\Delta$AURC vs softmax or the conditional ``geom adds`` on the graph cell, the graph "
+        "redundancy is a task property, not an artifact of thin curvature features.",
+        "tab:richer", f"run_ids={all_ids}")
+
+
 def _t3(rows: list[dict]) -> str:
     sel = selective_rows(rows)
     env = sel[-1]["env"] if sel else {}
@@ -188,9 +225,16 @@ def main() -> int:
         "T2b_feature_ablation.tex": _t2b(rows),
         "T3_repro.tex": _t3(rows),
     }
+    def _has_body(tex: str) -> bool:
+        return r"\midrule" in tex and bool(
+            tex.split(r"\midrule")[1].split(r"\bottomrule")[0].strip())
+
     t4 = _t4(rows)                                  # only when complementarity rows exist
-    if r"\midrule" in t4 and t4.split(r"\midrule")[1].split(r"\bottomrule")[0].strip():
+    if _has_body(t4):
         outputs["T4_complementarity.tex"] = t4
+    t5 = _t5(rows)                                  # only when richer-geometry rows exist
+    if _has_body(t5):
+        outputs["T5_richer_geometry.tex"] = t5
     for name, tex in outputs.items():
         (TABLE_DIR / name).write_text(tex)
         print(f"[tables] wrote {TABLE_DIR / name}")

configs/sweeps/arith_richer_seeds.toml

@@ -0,0 +1,13 @@
+# Phase 4k: 5-seed arithmetic basin-center with RICHER geometry, for completeness so T5 has the
+# base-vs-richer comparison on all four (task, reasoner) cells. Reduced folds match arith_seeds.toml
+# (its base-feature counterpart) so the only changed variable is the feature set.
+[sweep]
+base = "configs/experiments/arithmetic_selective.toml"
+
+[sweep.override]
+"eval.n_eval" = 600
+"conformal.n_calib" = 800
+"eval.richer_geometry" = true
+
+[sweep.grid]
+"run.seed" = [0, 1, 2, 3, 4]

configs/sweeps/graph_ired_richer_seeds.toml

@@ -0,0 +1,13 @@
+# Phase 4k THE CRUX: 5-seed graph-IRED with RICHER geometry (full Hessian spectrum + mode
+# connectivity). On graph-IRED the base 14 features tie softmax and the combined mapper adds nothing
+# (Phase 4j: redundancy, 0/5). This asks whether that tie is a real task property or an artifact of
+# the thin lambda_max/trace curvature features. Same base cell as graph_ired_seeds.toml; only the
+# feature set changes.
+[sweep]
+base = "configs/experiments/graph_ired.toml"
+
+[sweep.override]
+"eval.richer_geometry" = true
+
+[sweep.grid]
+"run.seed" = [0, 1, 2, 3, 4]

configs/sweeps/graph_richer_seeds.toml

@@ -0,0 +1,13 @@
+# Phase 4k: 5-seed graph basin-center with RICHER geometry, for completeness so T5 has the
+# base-vs-richer comparison on all four (task, reasoner) cells. Reduced folds match graph_seeds.toml
+# (its base-feature counterpart) so the only changed variable is the feature set.
+[sweep]
+base = "configs/experiments/graph_selective.toml"
+
+[sweep.override]
+"eval.n_eval" = 600
+"conformal.n_calib" = 800
+"eval.richer_geometry" = true
+
+[sweep.grid]
+"run.seed" = [0, 1, 2, 3, 4]

configs/sweeps/ired_richer_seeds.toml

@@ -0,0 +1,12 @@
+# Phase 4k: 5-seed arithmetic-IRED with RICHER geometry (full Hessian spectrum + mode connectivity).
+# Positive control — arith-IRED already beats softmax 4/5 with the base 14 features; richer geometry
+# should keep or strengthen that. Mirrors ired_seeds.toml (full folds, annealed sampler) so the only
+# changed variable vs the base cell is the feature set.
+[sweep]
+base = "configs/experiments/arithmetic_ired.toml"
+
+[sweep.override]
+"eval.richer_geometry" = true
+
+[sweep.grid]
+"run.seed" = [0, 1, 2, 3, 4]

docs/DECISIONS.md

@@ -2,6 +2,23 @@
 
 Short, dated, append-only. Newest first.
 
+## 2026-07-29 — Phase 4k: the graph redundancy is a task property, not thin features
+**Finding/decision:** the 4j redundancy verdict rested on a thin curvature description (only
+`λmax`/`tr(H)`), so we added two opt-in richer feature groups — the **full Hessian spectrum**
+(`geometry/spectrum.py`, exact `d×d` eigh via new `curvature.batched_spectrum`) and **mode
+connectivity** (`geometry/connectivity.py`, energy barriers via new `curvature.batched_path_energy`)
+— behind `[eval] richer_geometry` and re-ran the 4j complementarity test on all four cells (T5).
+**Result:** richer geometry helps nowhere — the crux graph-IRED conditional signal over softmax stays
+−0.002 (0/5); it's a wash where geometry already wins (arith-IRED +0.007, 4/5) and slightly worse on
+the fixed-bowl basin-center cells (extra features add noise). So the graph tie is a genuine property
+of the task, not feature poverty. **Design choices:** (a) all new JAX lives in `curvature.py` to keep
+invariant 1 (the two feature modules are pure NumPy); (b) richer is opt-in and appended *after* the
+base 14 so every prior result is reproduced byte-identically; (c) aggregation is feature-set-aware
+(`aggregate.feature_set_of`; `headline_cell`/`by_k`/`aggregate_by_k` default `feature_set="base"`) so
+richer rows never pollute the canonical T1/T2/T4 — exactly mirroring how `objective` keeps IRED out of
+T1. **Why record it:** it closes the paper's stated "richer geometry" open question with a decisive
+negative result, strengthening (not weakening) the redundancy claim. **Reversible?** Fully additive.
+
 ## 2026-07-28 — Phase 4j: the graph tie is redundancy, not complementarity (combined-mapper test)
 **Finding/decision:** to distinguish redundancy from complementarity we measure whether a mapper on
 `[geometry ∪ softmax]` beats one on `[softmax]` alone (both fit on the fit fold; `evaluate` adds

docs/EXPERIMENTS.md

@@ -198,6 +198,40 @@ the softmax does not. Re-running this baseline stress test under IRED training i
 experiment (see SESSION_HANDOFF). MC-dropout / deep ensembles remain deferred (the K-restart
 best-of-N already supplies ensemble-like diversity).
 
+### Richer geometry — is the graph tie thin features or a task property? (Phase 4k → T5)
+
+The Phase-4j redundancy verdict rested on a thin curvature representation: of the 14 features,
+curvature was only `λmax` (power iteration) and `tr(H)` (Hutchinson). So "geometry is redundant with
+softmax on graph" was confounded with "our geometry features are impoverished." Phase 4k adds two
+richer, opt-in groups (`[eval] richer_geometry=true`) — the **full Hessian spectrum**
+(`spectrum/*`: `lmin`, negative-eigenvalue fraction, effective rank / participation ratio, log-det,
+each mean-over-restarts and at the best restart; exact `d×d` eigendecomposition since `d` is small)
+and **mode connectivity** (`connect/*`: energy barriers along the straight-line path from the best
+restart to the others) — and re-runs the Phase-4j complementarity test on all four cells. Same
+reasoners, folds, and mappers; only the feature set changes.
+
+| task | reasoner | features | AURC geom | ΔAURC vs softmax | ΔAURC geom *adds* | seeds add |
+|------|----------|----------|-----------|------------------|-------------------|-----------|
+| arithmetic | basin-center | base | $0.059$ | $-0.011$ | $+0.001$ | $0/5$ |
+| arithmetic | basin-center | **richer** | $0.061$ | $-0.012$ | $-0.002$ | $1/5$ |
+| arithmetic | IRED | base | $0.028$ | $+0.007$ | $+0.007$ | $4/5$ |
+| arithmetic | IRED | **richer** | $0.028$ | $+0.008$ | $+0.007$ | $4/5$ |
+| graph | basin-center | base | $0.159$ | $-0.034$ | $-0.006$ | $0/5$ |
+| graph | basin-center | **richer** | $0.162$ | $-0.037$ | $-0.009$ | $0/5$ |
+| **graph** | **IRED** | base | $0.056$ | $+0.001$ | $+0.001$ | $0/5$ |
+| **graph** | **IRED** | **richer** | $0.055$ | $+0.002$ | $\mathbf{-0.002}$ | **$0/5$** |
+
+**Verdict — the graph redundancy is a task property, not a feature-poverty artifact.** In the crux
+cell (graph-IRED), the full spectrum + mode connectivity leave the conditional signal over softmax at
+$-0.002$, $0/5$ — geometry still adds *nothing*. Richer geometry changes the picture nowhere: it is a
+wash on the one cell geometry already wins (arithmetic-IRED, $+0.007$, $4/5$, unchanged) and slightly
+*worse* on the fixed-bowl basin-center cells (adding features to a trivial landscape only adds noise:
+AURC $0.059\to0.061$ arith, $0.159\to0.162$ graph). So the answer to "does *more* landscape signal
+help on graph?" is **no** — a well-calibrated softmax genuinely captures what descent geometry would
+contribute there, even with the richest geometry we can extract. The base rows reproduce the Phase-4j
+numbers exactly, confirming richer geometry is purely additive and non-breaking (default stays the
+canonical 14 features; aggregation is feature-set-aware so richer never pollutes T1/T2/T4).
+
 ## Operating regime
 
 Tune each task's difficulty so base accuracy is **70–85%**. Fully-solved tasks saturate AURC and