Phase 4h: IRED on graph — the softmax flip is task-dependent (honest update)

.claude/CLAUDE.md

@@ -43,7 +43,8 @@ inference-time descent** into distribution-free reliability certificates. Read
   so the win is non-energy features — rebuts "it's just energy").
 - Phase 5 paper draft ✅ (`paper/sections/*.tex` written from the real results; figures/tables wired;
   compiles via `tectonic`).
-- Phase 4g IRED landscape ✅ (opt-in `objective="ired"` contrastive+stationarity + annealed sampler;
-  it reasons, and under the learned landscape geometry beats softmax confidence **4/5 seeds** —
-  flips the 4e caveat, confirming the landscape was the limiter).
-- Next: E3/E4 tasks · Modal · scale-up.
+- Phase 4g/4h IRED landscape ✅ (opt-in `objective="ired"` contrastive+stationarity + annealed
+  sampler; it reasons). Under the learned landscape geometry beats softmax **4/5 seeds on arithmetic**
+  (flips the 4e caveat) but only **ties on graph** (1/5) — the flip is real but **task-dependent**.
+  Aggregation is objective-aware so IRED never pollutes the basin-center T1.
+- Next: stronger IRED graph reasoner + more seeds to widen the margin · E3/E4 · Modal · scale-up.

analysis/aggregate.py

@@ -13,17 +13,25 @@
 from edc.ledger import read_all
 
 
-def selective_rows(rows: list[dict] | None = None, task: str | None = None) -> list[dict]:
+def objective_of(row: dict) -> str:
+    """Training objective behind a row: ``basin_center`` (default) or ``ired``."""
+    return row["metrics"].get("objective") or row["config"].get("train", {}).get(
+        "objective", "basin_center")
+
+
+def selective_rows(rows: list[dict] | None = None, task: str | None = None,
+                   objective: str | None = None) -> list[dict]:
     """Latest ``split=='selective'`` row per ``(config_hash, seed)`` (cf. latest_per_config).
 
-    ``task`` filters to one reasoning family (``row["task"]``) so multi-task ledgers do not
-    cross-contaminate per-K aggregates.
+    ``task`` filters to one reasoning family and ``objective`` to one reasoner (``basin_center`` vs
+    ``ired``), so multi-task / multi-reasoner ledgers do not cross-contaminate per-K aggregates.
     """
     if rows is None:
         rows = read_all()
     latest: dict[tuple[str, int], dict] = {}
     for r in rows:
-        if r.get("split") == "selective" and (task is None or r.get("task") == task):
+        if (r.get("split") == "selective" and (task is None or r.get("task") == task)
+                and (objective is None or objective_of(r) == objective)):
             latest[(r["config_hash"], r["seed"])] = r  # later rows win
     return list(latest.values())
 
@@ -99,14 +107,16 @@ def aggregate_by_k(rows: list[dict] | None = None, task: str | None = None) -> d
     return {k: aggregate_cell(v) for k, v in by_k(rows, task=task).items()}
 
 
-def headline_cell(rows: list[dict] | None = None, task: str | None = None) -> dict:
-    """Aggregate the headline seed-sweep for a task: the largest single-config seed group.
+def headline_cell(rows: list[dict] | None = None, task: str | None = None,
+                  objective: str = "basin_center") -> dict:
+    """Aggregate the headline seed-sweep for one task + reasoner (largest single-config seed group).
 
-    Prefers rows carrying the Phase-4e baseline field, then picks the ``config_hash`` group with the
-    most seeds — the multi-seed sweep — so T1 is not polluted by mixing fold sizes / configs that
-    happen to share a K value. Falls back to all rows for the task if none carry the field.
+    T1 uses the canonical ``basin_center`` reasoner (pass ``objective="ired"`` to report the learned
+    landscape separately). Prefers rows carrying the Phase-4e baseline field, then picks the config
+    group (K, n_test) with the most seeds — the multi-seed sweep — so the cell is not polluted by
+    mixing fold sizes / reasoners that happen to share a K value.
     """
-    sel = selective_rows(rows, task=task)
+    sel = selective_rows(rows, task=task, objective=objective) or selective_rows(rows, task=task)
     with_bl = [r for r in sel if "delta_aurc_vs_best_baseline" in r["metrics"]]
     pool = with_bl or sel
     # Group by the experiment config *excluding seed* (config_hash bakes in the seed). (K, n_test)

configs/experiments/graph_ired.toml

@@ -0,0 +1,45 @@
+# Phase 4h: IRED learned-landscape reasoner on graph shortest-path — does the softmax-caveat flip
+# (Phase 4g, arithmetic) generalize to a second task? IRED is a weaker graph reasoner than the
+# basin-center model (ID ~0.58 vs 0.70; still above the 0.42 majority baseline), so this is an
+# honest, harder test.
+[run]
+task = "graph_planning"
+seed = 0
+notes = "IRED learned-landscape reasoner on graph"
+
+[task.graph_planning]
+n_nodes = 7
+ood_n_nodes = 10
+edge_prob = 0.4
+max_len = 4
+
+[model]
+latent_dim = 32
+hidden_dim = 128
+context_dim = 64
+
+[inference]
+k_restarts = 12
+init_scale = 1.0
+sampler = "annealed"
+anneal_levels = 20
+anneal_steps_per_level = 10
+anneal_step_max = 0.5
+anneal_step_min = 0.003
+temperature = 0.01
+
+[train]
+objective = "ired"
+epochs = 100
+n_train = 10000
+ired_ridge = 0.2
+ired_stat_weight = 8.0
+ired_decode_weight = 4.0
+
+[eval]
+n_eval = 800
+
+[conformal]
+alpha = 0.1
+delta = 0.05
+n_calib = 800

configs/sweeps/graph_ired_seeds.toml

@@ -0,0 +1,8 @@
+# Phase 4h: 5-seed IRED-on-graph — does geometry beat softmax under a learned landscape on a second
+# task? (On graph the basin-center reasoner lost to softmax 0/5 by a larger margin, so this is the
+# harder generalization test of the Phase-4g flip.)
+[sweep]
+base = "configs/experiments/graph_ired.toml"
+
+[sweep.grid]
+"run.seed" = [0, 1, 2, 3, 4]

docs/DECISIONS.md

@@ -2,6 +2,17 @@
 
 Short, dated, append-only. Newest first.
 
+## 2026-07-27 — Phase 4h: objective-aware aggregation; the IRED flip is task-dependent
+**Decision:** `evaluate` now records `objective` (basin_center|ired) + `sampler` in its metrics, and
+`analysis/aggregate` is **objective-aware** (`objective_of`, `selective_rows(objective=…)`,
+`headline_cell(objective="basin_center")` by default) so IRED rows never contaminate the canonical
+basin-center T1 (previously the IRED and basin sweeps shared K=12 and T1 picked one by a fragile
+tie-break). **Finding:** the Phase-4g softmax-caveat flip is **task-dependent** — robust on
+arithmetic (geometry beats softmax 4/5 under IRED) but only a tie on graph (1 win/1 loss/3 ties),
+where the learned reasoner is weaker (ID 0.55–0.82). Reported narrowly; the paper/abstract/docs
+tempered accordingly. **Why:** honesty — one task's flip is not a general claim; and the fragile
+tie-break was a latent T1 bug worth fixing. **Reversible?** Yes — `objective` filter is additive.
+
 ## 2026-07-27 — Phase 4g: IRED reasons via contrastive+stationarity (not DSM); flips the 4e caveat
 **Decision:** the working IRED objective is **IREM-style contrastive + stationarity + decode**, not
 denoising score matching. DSM was stuck (score net never fit under the double-grad objective).

docs/EXPERIMENTS.md

@@ -144,20 +144,25 @@ random `z0` by a margin (reachability); the **stationarity** term drives `‖∇
 lacked (low-energy but not stationary, so descent never settled). With annealed descent, ID accuracy
 rises from chance to $0.72$–$0.90$.
 
-**Result (5 seeds, learned IRED landscape; `sweeps/ired_seeds.toml`).**
-
-| reasoner | ΔAURC vs scalar energy | ΔAURC vs best **softmax** baseline |
-|----------|------------------------|------------------------------------|
-| basin-center (Phase-1) | $+0.085$, geometry wins $5/5$ | $-0.011$, geometry wins **$0/5$** |
-| **IRED (learned)** | $+0.159\pm0.051$, wins $5/5$ | $\mathbf{+0.007\pm0.005}$, wins **$4/5$** |
-
-Under a genuinely learned multi-basin landscape, **geometry beats the best softmax-confidence
-baseline (MSP / temperature / entropy) on $4/5$ seeds** (only seed~3 ties), where the basin-center
-reasoner lost $0/5$. The margin is small but consistent, and it **flips the Phase-4e caveat**: the
-limiter was the simple landscape, not the geometry idea---exactly the hypothesis. LTT abstention
-remains valid under IRED (e.g. seed~0: $81\%$ coverage at selective risk $0.052\le\alpha$). Default
-`basin_center` is unchanged; this is opt-in and additive. Next: multi-seed IRED on graph, and a
-larger stat-weight / capacity sweep to widen the margin.
+**Result (5 seeds/task; `sweeps/ired_seeds.toml`, `sweeps/graph_ired_seeds.toml`).** ΔAURC vs the
+best of all baselines (energy + softmax); "wins" = seeds whose 95% CI clears 0.
+
+| task | reasoner | ΔAURC vs scalar energy | ΔAURC vs best **softmax** baseline |
+|------|----------|------------------------|------------------------------------|
+| arithmetic | basin-center | $+0.085$, wins $5/5$ | $-0.011$, wins **$0/5$** |
+| arithmetic | **IRED (learned)** | $+0.159\pm0.051$, $5/5$ | $\mathbf{+0.007\pm0.005}$, wins **$4/5$** |
+| graph | basin-center | $+0.111$, wins $5/5$ | $-0.034$, wins **$0/5$** |
+| graph | **IRED (learned)** | $+0.064$, $5/5$ | $+0.004\pm0.014$, wins **$1/5$** (1 win, 1 loss, 3 ties) |
+
+**Verdict — the flip is real but task-dependent.** On **arithmetic**, a genuinely learned multi-basin
+landscape makes geometry beat the best softmax-confidence baseline (MSP/temp/entropy) on $4/5$ seeds,
+reversing the Phase-4e $0/5$ caveat and confirming the hypothesis that the simple landscape (not the
+geometry idea) was the limiter. On **graph**, where the learned reasoner is weaker and
+higher-variance (ID $0.55$–$0.82$), geometry still beats scalar energy $5/5$ but only **ties** softmax
+($1$ win / $1$ loss / $3$ ties). So the flip is demonstrated on arithmetic, not yet on graph — we
+claim it narrowly. LTT abstention stays valid under IRED. Default `basin_center` unchanged; opt-in +
+additive; IRED rows are `objective`-tagged so they never pollute the basin-center T1 (aggregation is
+now objective-aware). Next: a stronger IRED graph reasoner + more seeds to widen the small margin.
 
 **Likely cause of the softmax tie + the lever.** The reasoner is the Phase-1 **supervised
 basin-center** model with a

docs/SESSION_HANDOFF.md

@@ -72,12 +72,14 @@
   (`train/losses.ired_loss_fn`) + opt-in **annealed Langevin** sampler
   (`optimizer.annealed_langevin`, `[inference] sampler="annealed"`). The stationarity term
   (`||∇_z E(μ_y)||→0`, weight `ired_stat_weight`) was the key — it makes anchors genuine attractors,
-  which the earlier DSM attempts lacked. **5-seed result:** the IRED reasoner reasons (ID ~0.72–0.90)
-  and, under the learned landscape, **geometry beats softmax confidence on 4/5 seeds** (ΔAURC
-  +0.007±0.005 vs the best baseline, vs 0/5 under basin_center) — confirming the Phase-4e hypothesis
-  that the simple landscape was the limiter. `configs/experiments/arithmetic_ired.toml`,
-  `sweeps/ired_seeds.toml`. Default `basin_center` unchanged (92 tests green).
-- **Deferred: stubbed.** More tasks (E3/E4), Modal, multi-seed IRED on graph, scale-up.
+  which the earlier DSM attempts lacked. **5-seed result (task-dependent flip):** on **arithmetic**,
+  geometry beats softmax **4/5 seeds** (ΔAURC +0.007±0.005 vs best baseline, vs 0/5 under
+  basin_center) — flips the Phase-4e caveat; on **graph** (IRED weaker, ID 0.55–0.82) it only **ties**
+  (1 win/1 loss/3 ties, +0.004±0.014). So the flip is demonstrated on arithmetic, not yet graph.
+  `configs/experiments/{arithmetic,graph}_ired.toml`, `sweeps/{ired,graph_ired}_seeds.toml`.
+  Aggregation is now **objective-aware** (`aggregate.objective_of`, `headline_cell(objective=…)`) so
+  IRED rows never pollute the basin-center T1. Default `basin_center` unchanged (92 tests green).
+- **Deferred: stubbed.** Stronger IRED graph reasoner + more seeds (widen margin), E3/E4, Modal, scale-up.
 
 ## What is real vs stub
 

paper/main.tex

@@ -35,8 +35,8 @@
 agreement, and halting saves ${\sim}58\%$ of inference compute at a guaranteed error budget. Against a stronger, standard softmax-confidence baseline (maximum softmax probability, temperature
 scaling, entropy), geometry only \emph{ties} (arithmetic) or \emph{loses} (graph) under a supervised
 basin-center reasoner---but when the reasoner is a genuinely learned, multi-basin landscape,
-geometry \emph{beats} softmax confidence on $4/5$ seeds, indicating that the simple landscape, not
-the geometry idea, was the limiter. The certificates are valid in-distribution and, by design, break
+geometry \emph{beats} softmax confidence on arithmetic ($4/5$ seeds; on graph it ties), indicating
+that the landscape, not the geometry idea, gates the effect. The certificates are valid in-distribution and, by design, break
 under distribution shift, which is the argument for treating an abstention as a routing signal in
 critical systems.
 \end{abstract}

paper/sections/conclusion.tex

@@ -14,10 +14,12 @@ \section{Conclusion}
 supervised basin-center reasoner, geometry beats scalar energy but only ties/loses to a well-%
 calibrated softmax. That dependence is itself informative---and encouraging---because replacing the
 landscape with a genuinely learned, multi-basin energy (per-class attractors trained by a
-contrastive-plus-stationarity objective, annealed sampling) makes geometry beat softmax confidence
-($4/5$ seeds), confirming that the simple landscape was the limiter, not the geometry idea.
+contrastive-plus-stationarity objective, annealed sampling) makes geometry beat softmax confidence on
+arithmetic ($4/5$ seeds), evidence that the landscape---not the geometry idea---gates the effect. On
+graph, where that learned reasoner is weaker, the same recipe only ties softmax, so we claim the flip
+narrowly: demonstrated on one task, task-dependent, with a small margin.
 
-\paragraph{Future work.} The learned-landscape result is on arithmetic and the margin over softmax is
-small; widening it---larger models and richer landscapes (annealed score matching), the graph and
-further tasks, more seeds---is the immediate next step. Beyond that: scaling to language-scale EBTs
-and richer geometry (full spectra, mode connectivity).
+\paragraph{Future work.} Making the learned-landscape flip robust and general is the immediate next
+step: stronger learned reasoners (especially on graph), richer landscapes (annealed score matching),
+and more seeds to widen the small margin. Beyond that: scaling to language-scale EBTs and richer
+geometry (full spectra, mode connectivity).

paper/sections/experiments.tex

@@ -72,13 +72,17 @@ \section{Experiments}
 multi-basin landscape---per-class latent anchors trained as attractors by a contrastive-plus-%
 stationarity objective (pushing the correct anchor below competitors and driving
 $\|\nabla_z E(\mu_y)\|\!\to\!0$ so it is a local minimum), with annealed Langevin descent at
-inference. This reasoner reasons (ID $0.72$--$0.90$), and under it geometry \emph{does} beat the best
-softmax-confidence baseline: over $5$ seeds, $\Delta\mathrm{AURC} = +0.007\pm0.005$ against the best
-overall baseline ($4/5$ seeds' CI excluding zero), and $+0.159\pm0.051$ against scalar energy---%
-compared to $-0.011$ ($0/5$) under the basin-center reasoner. The margin over softmax is small but
-consistent, and it flips the caveat: inference-time landscape geometry can beat a well-calibrated
-softmax \emph{when the landscape is genuinely multi-basin}. We report this on arithmetic; widening
-the margin (larger models, graph, more seeds) is ongoing.
+inference. This reasoner reasons (arithmetic ID $0.72$--$0.90$), and on \emph{arithmetic} geometry \emph{does}
+beat the best softmax-confidence baseline: over $5$ seeds, $\Delta\mathrm{AURC} = +0.007\pm0.005$
+against the best overall baseline ($4/5$ seeds' CI excluding zero) and $+0.159\pm0.051$ against
+scalar energy---compared to $-0.011$ ($0/5$) under the basin-center reasoner. So a genuinely
+multi-basin landscape can flip the caveat: inference-time geometry can beat a well-calibrated softmax
+when the landscape supports it. The effect is, however, \emph{task-dependent}: on \emph{graph}, where
+the learned reasoner is weaker and higher-variance (ID $0.55$--$0.82$), geometry still beats scalar
+energy $5/5$ but only ties softmax ($\Delta\mathrm{AURC}=+0.004\pm0.014$, $1$ win / $1$ loss / $3$
+ties). We therefore claim the flip narrowly and honestly---demonstrated on arithmetic, not yet on
+graph---and read it as evidence that the landscape, not the geometry idea, gates the effect. Widening
+it (stronger learned reasoners, richer landscapes, more seeds) is ongoing.
 
 \input{tables/T1_main}
 \input{tables/T2b_feature_ablation}

src/edc/eval/evaluate.py

@@ -213,6 +213,8 @@ def scores_for(fold: dict) -> dict:
         "n_calib": int(cal["correct"].shape[0]),
         "n_test": int(test["correct"].shape[0]),
         "k_restarts": int(cfg.inference.k_restarts),
+        "objective": getattr(cfg.train, "objective", "basin_center"),
+        "sampler": getattr(cfg.inference, "sampler", "langevin"),
         "accuracy_id": test["accuracy"],
         "base_error": float((~correct).mean()),
         "final_train_loss": float(history["epochs"][-1]["loss"]),