Direction backprop

README.md

@@ -38,6 +38,7 @@ These are load-bearing. They show up in every prompt and every config default.
 - **Grounding over generation.** Research directives can't invent URLs or tool names. Every citation must come from an allowlist built from the source register entry; every tool call must exact-match an allowlist. A verifier scans the output for fabrication; anything dubious ships with a `⚠ FLAGGED ISSUES` block.
 - **Cross-domain is a first-class move.** On surprising findings, an analog probe asks the engine which *distant* fields have structural analogs. On unsurprising-but-confirmed findings, an assumption probe surfaces premises the field takes for granted. A negative-space scan maps which `(method, problem)` combinations *aren't* in the journal yet.
 - **Audit trails, not overwrites.** Re-verification appends to a log instead of mutating the original verdict — both old and new are preserved so you can see what changed under updated rules.
+- **Lessons carry forward as edges, not beliefs.** Every few cycles the engine clusters related investigations (connected components of the knowledge graph) and distills each cluster into a *frontier edge* — the unexplored synthesis or contradiction at its boundary — which is injected into the next introspection as something to push *past*, never to confirm. The distillation runs on the cross-family verifier model so the generator never authors and then consumes its own prior. (Direction backpropagation — see [Self-evolution](#self-evolution-phases-6-9-borrowed-from-comparable-public-systems).)
 - **Domain-agnostic by construction.** Prompts use shape, not content. The same engine works identically for any structured field.
 
 ---
@@ -266,6 +267,7 @@ Once the verifier side stabilized, we audited public research-agent systems for
 - **Phase 7** (persona-conditioned introspection, planned) — borrows multi-perspective question generation from **[Stanford STORM / Co-STORM](https://github.com/stanford-oval/storm)**. Each persona (skeptic / outsider / historian / contrarian / practitioner) surfaces blind spots the single-voice introspection misses.
 - **Phase 8** (idea evolution from downgraded extensions, planned) — borrows the mutation loop from **[Sakana AI's "AI Scientist"](https://github.com/SakanaAI/AI-Scientist)** and the Evolution agent from Co-Scientist. Internally aligns with `r-3c792e21` ("typed supervision from false positives via retrospective unification") — we treat verifier downgrades as typed supervision signal for generator-side mutation.
 - **Phase 9** (hypothesis variants in investigation, planned) — borrows branching exploration from **[Tree of Thoughts](https://arxiv.org/abs/2305.10601)**. The explorer persona generates N divergent priors; the most-distant-from-majority-literature variant drives the investigation.
+- **Direction insight backpropagation** — borrows the *carry-lessons-across-time* mechanism from **Arbor / Hypothesis Tree Refinement ([arXiv:2606.11926](https://arxiv.org/abs/2606.11926))**, whose ablation showed that propagating distilled insights matters more than the tree structure that holds them (removing propagation hurt more than removing the tree). CE adapts it to a *novelty* objective rather than Arbor's scalar one: the carried signal is a frontier edge to **exceed**, not a belief to confirm, because CE's value is divergence (OOD synthesis) where Arbor's is convergence to a measured optimum. Deliberately omits Arbor's frontier *pruning* — for a novelty engine there is no clean falsification signal, and pruning on a weak proxy (repeated low surprise) would kill exactly the directions that pay off late.
 
 ### General agentic patterns CE builds on
 
@@ -279,6 +281,7 @@ Once the verifier side stabilized, we audited public research-agent systems for
 - **Pareto-dominance admission** (Phase 4) is multi-objective optimization theory applied to a register-admission gate.
 - **Negative-space mapping** (the `(method × problem)` matrix) is a long-standing literature-review discipline; CE just instruments it.
 - **Falsifiable predictions with target dates** is descended from prediction-market and forecasting-literature practice (Tetlock, Good Judgement Project, Metaculus).
+- **The adversarial verifier is CE's held-out gate.** Arbor ([arXiv:2606.11926](https://arxiv.org/abs/2606.11926)) admits an improvement only if it beats the current best on a *held-out* eval — separating "looked good on the exploration signal" from verified progress, which is what keeps it from overfitting. CE's adversarial prior-art search plays the same *role*: apparent novelty measured against the journal (the dev signal) is admitted only after surviving search against the literature (the held-out set). Local novelty that collapses under prior-art search is CE's "overfit." This is an analogy of role, not identity of mechanism — CE's gate is structural prior-art search, not a scalar metric comparison. CE deliberately does **not** adopt a numeric dev/held-out gate: a scalar admission target would push the engine toward optimizing a novelty *proxy*, violating "novelty is structural, not vibes."
 
 ### Where CE is genuinely novel
 
@@ -497,6 +500,12 @@ python curiosity_engine.py --backfill-canonical-forms --backfill-force   # re-ca
 python curiosity_engine.py --synth-orphaned-xrefs                 # recover from mid-run crashes
 python curiosity_engine.py --scan-gaps                            # negative-space scan
 
+# Direction insight backpropagation (Arbor / HTR) — runs automatically every
+# [engine].direction_abstract_every_n_cycles cycles; these force/inspect it
+python curiosity_engine.py --abstract-directions                  # force a distillation run now
+python curiosity_engine.py --show-directions                      # show the injected frontier edges
+python curiosity_engine.py --suppress-direction dir-abc12345      # kill a bad prior (stops injection)
+
 # Research directives
 python curiosity_engine.py --export-directive r-cd730b6d          # per-record
 python curiosity_engine.py --export-directives-bundle             # all qualifying

config.py

@@ -247,13 +247,62 @@ class EngineSettings:
     # Set to a non-zero value (e.g. 0.70) on mature journals where you
     # specifically want to prune low-priority noise.
     question_priority_floor: float = 0.0
+    # ── Direction insight backpropagation (Arbor / HTR, arXiv:2606.11926) ──
+    # Abstract clusters of related investigations into direction-level priors
+    # and inject them into introspection as FRONTIER EDGES to push past — the
+    # carry-forward layer CE was missing. The framing is deliberately
+    # divergent ("exceed these edges", not "confirm these beliefs"); abstraction
+    # runs on the cross-family verifier model, never the primary, to avoid the
+    # self-flattery loop where the generator authors and then consumes its own
+    # prior. See docs/superpowers/specs/2026-06-18-direction-backprop-design.md.
+    direction_backprop_enabled: bool = True
+    # In-loop trigger cadence: abstract directions every N cycles, before
+    # introspection, so fresh priors are available to the same cycle.
+    direction_abstract_every_n_cycles: int = 5
+    # Journal-size floor — abstraction stays silent below this many entries
+    # (small journals have no stable directions yet, like negative_space).
+    direction_min_entries: int = 8
+    # Minimum connected-component size (entries) to qualify as a direction.
+    direction_min_cluster_size: int = 4
+    # Max directions abstracted per run (largest components first) — bounds cost.
+    direction_max_count: int = 5
+    # Max frontier edges injected into introspection — the convergence dial.
+    # Lower = safer (priors can't dominate the prompt / homogenize personas).
+    direction_max_injected: int = 4
     # Parallel fan-out — how many investigations / xref-synth+verify pipelines
     # run concurrently per cycle. Default 1 = fully serial (zero behavior change).
     # Rate limiters are shared across threads so raising these does not burst
     # public APIs. Sensible ceilings are 3–4 investigations and 2–3 xref
     # pipelines; beyond that most time is spent waiting on rate limiters anyway.
     parallel_investigations: int = 1
     parallel_xref_pipeline: int = 1
+    # ── Selective verification pipeline (A2 plumbing for Phases B/C/D) ──
+    # All defaults are NO-OPS — A2 alone changes no behavior. Each knob
+    # turns on when the corresponding phase ships.
+    #
+    # Phase B (alias-gap routing). Candidates with alias_gap below the reject
+    # threshold are short-circuited to the rejection_log without an LLM
+    # verification call. Candidates above the fast-track threshold skip
+    # deep prior-art search and take the cheap path. The middle band runs
+    # the existing full pipeline. 0.0 / 1.0 = no-op (no gating, every
+    # candidate runs full pipeline).
+    alias_gap_reject_threshold: float = 0.0
+    alias_gap_fasttrack_threshold: float = 1.0
+    # Phase B audit-back: fraction of below-reject-threshold candidates that
+    # still get full LLM verification, tagged in the rejection_log as audit
+    # samples. Detects discriminator drift — if audit-back items keep
+    # validating, the reject threshold is too aggressive. 0.0 = never
+    # sample-back (no-op).
+    rejection_audit_back_rate: float = 0.0
+    # Phase C (paraphrase-perturbation verifier stability). Run verification
+    # against N paraphrase variants of the prompt, compute verdict variance
+    # as a paraphrase_inconsistency_score. 1 = single pass (no-op,
+    # pre-Phase-C behavior). 3 = recommended once C ships.
+    paraphrase_variant_count: int = 1
+    # Phase D (committee escalation). When paraphrase_inconsistency_score
+    # exceeds this threshold (or committee verdicts disagree), escalate
+    # to a second cross-family verifier. 1.0 = never escalate (no-op).
+    committee_dissent_threshold: float = 1.0
 
 CONFIG_DIR = Path.home() / ".CuriosityEngine"
 CONFIG_PATH = CONFIG_DIR / "engine.toml"
@@ -403,6 +452,24 @@ def load(cls, path: Path = CONFIG_PATH) -> CuriosityEngineConfig:
             question_priority_floor=float(
                 eng_section.get("question_priority_floor", 0.70)
             ),
+            direction_backprop_enabled=bool(
+                eng_section.get("direction_backprop_enabled", True)
+            ),
+            direction_abstract_every_n_cycles=max(
+                1, int(eng_section.get("direction_abstract_every_n_cycles", 5))
+            ),
+            direction_min_entries=max(
+                1, int(eng_section.get("direction_min_entries", 8))
+            ),
+            direction_min_cluster_size=max(
+                2, int(eng_section.get("direction_min_cluster_size", 4))
+            ),
+            direction_max_count=max(
+                1, int(eng_section.get("direction_max_count", 5))
+            ),
+            direction_max_injected=max(
+                0, int(eng_section.get("direction_max_injected", 4))
+            ),
             register_admission_mode=str(
                 eng_section.get("register_admission_mode", "scalar")
             ).strip().lower() or "scalar",
@@ -440,6 +507,21 @@ def load(cls, path: Path = CONFIG_PATH) -> CuriosityEngineConfig:
             investigation_assessor_role=str(eng_section.get("investigation_assessor_role", "")).strip(),
             parallel_investigations=int(eng_section.get("parallel_investigations", 1)),
             parallel_xref_pipeline=int(eng_section.get("parallel_xref_pipeline", 1)),
+            alias_gap_reject_threshold=float(
+                eng_section.get("alias_gap_reject_threshold", 0.0)
+            ),
+            alias_gap_fasttrack_threshold=float(
+                eng_section.get("alias_gap_fasttrack_threshold", 1.0)
+            ),
+            rejection_audit_back_rate=float(
+                eng_section.get("rejection_audit_back_rate", 0.0)
+            ),
+            paraphrase_variant_count=max(
+                1, int(eng_section.get("paraphrase_variant_count", 1)),
+            ),
+            committee_dissent_threshold=float(
+                eng_section.get("committee_dissent_threshold", 1.0)
+            ),
         )
 
         # Resolve cross_ref profile:
@@ -791,6 +873,19 @@ def _build_toml(
 # before the journal could build context). Set to a non-zero value only on
 # mature journals where you specifically want to prune low-priority noise.
 question_priority_floor = {eng.question_priority_floor}
+# Direction insight backpropagation (Arbor / HTR, arXiv:2606.11926). Every N
+# cycles, cluster related investigations and distill each into a FRONTIER EDGE
+# injected into introspection as something to push PAST (never confirm).
+# Abstraction runs on the verifier model to avoid a self-flattery loop.
+# Set direction_backprop_enabled = false to disable entirely.
+direction_backprop_enabled = {str(eng.direction_backprop_enabled).lower()}
+direction_abstract_every_n_cycles = {eng.direction_abstract_every_n_cycles}
+direction_min_entries = {eng.direction_min_entries}
+direction_min_cluster_size = {eng.direction_min_cluster_size}
+direction_max_count = {eng.direction_max_count}
+# Max frontier edges injected per introspection — the convergence dial; lower
+# is safer (priors can't dominate the prompt / homogenize the personas).
+direction_max_injected = {eng.direction_max_injected}
 # Register admission mode. "scalar" = single confidence floor + status checks
 # (default; backward compatible). "pareto" = ALSO require the new entry to be
 # non-dominated by any existing active entry across the 4-axis Pareto set
@@ -889,6 +984,24 @@ def _build_toml(
 # 2–3 xref pipelines before rate-limit waits dominate anyway.
 parallel_investigations = {eng.parallel_investigations}
 parallel_xref_pipeline = {eng.parallel_xref_pipeline}
+# ── Selective verification pipeline (A2 plumbing for Phases B/C/D) ──
+# All defaults below are NO-OPS until the corresponding phase ships.
+# Phase B — alias-gap routing. Candidates below reject_threshold short-
+# circuit to rejection_log without LLM verification; above
+# fasttrack_threshold skip deep prior-art search. Middle band runs the
+# full pipeline. 0.0 / 1.0 = no gating (current behavior).
+alias_gap_reject_threshold = {eng.alias_gap_reject_threshold}
+alias_gap_fasttrack_threshold = {eng.alias_gap_fasttrack_threshold}
+# Phase B audit-back — fraction of below-reject-threshold candidates that
+# still get full LLM verification, tagged as audit samples. Detects
+# discriminator drift. 0.0 = no audit-back (current behavior).
+rejection_audit_back_rate = {eng.rejection_audit_back_rate}
+# Phase C — paraphrase-perturbation. N variants per verification, verdict
+# variance scored as paraphrase_inconsistency_score. 1 = single pass.
+paraphrase_variant_count = {eng.paraphrase_variant_count}
+# Phase D — committee escalation when paraphrase_inconsistency_score
+# exceeds threshold (or verdicts disagree). 1.0 = never escalate.
+committee_dissent_threshold = {eng.committee_dissent_threshold}
 """
     )
     return header + "\n".join(sections)

curiosity_engine.py

@@ -119,6 +119,12 @@ def main():
                         help="Synthesize + verify every cross-reference that doesn't yet have a matching insight (e.g. after a mid-run failure between cross-ref and synthesis).")
     parser.add_argument("--scan-gaps", action="store_true",
                         help="Run a negative-space scan: build (method × problem) matrix from journal entries, classify empty cells, verify underexplored gaps via academic_search, enqueue questions for verified gaps. Gated by [engine].negative_space_min_entries.")
+    parser.add_argument("--abstract-directions", action="store_true",
+                        help="Direction insight backpropagation: cluster related investigations (connected components of the entry graph), abstract each into a frontier-edge prior on the verifier model, and persist it for injection into introspection. Runs automatically every [engine].direction_abstract_every_n_cycles cycles; this forces a run now. Gated by [engine].direction_min_entries.")
+    parser.add_argument("--show-directions", action="store_true",
+                        help="Print the current (latest, non-suppressed) direction insights — the frontier edges injected into introspection.")
+    parser.add_argument("--suppress-direction", type=str, default=None, metavar="DIR_ID",
+                        help="Suppress a direction insight by id (e.g. dir-abc12345) so its lineage stops being injected into introspection. The anti-pollution lever for a bad prior.")
     parser.add_argument("--backfill-canonical-forms", action="store_true",
                         help="Populate canonical_form on existing register entries that lack one. One-shot maintenance pass — safe to interrupt and re-run.")
     parser.add_argument("--backfill-force", action="store_true",
@@ -174,6 +180,9 @@ def main():
         or args.reverify_insight is not None
         or args.synth_orphaned_xrefs
         or args.scan_gaps
+        or args.abstract_directions
+        or args.show_directions
+        or args.suppress_direction is not None
     )
     if args.domain is None:
         if read_only:
@@ -389,6 +398,33 @@ def _override(cli_val, toml_val):
         engine.test_pareto_admission(args.pareto_admission_test)
     elif args.scan_gaps:
         engine.scan_gaps()
+    elif args.abstract_directions:
+        engine.abstract_directions()
+    elif args.show_directions:
+        directions = engine.journal.latest_direction_insights()
+        if not directions:
+            print("No direction insights yet. Run --abstract-directions or let the cycle loop produce them.")
+        else:
+            ranked = sorted(
+                directions,
+                key=lambda r: (float(r.get("confidence") or 0.0), r.get("generated_at", "")),
+                reverse=True,
+            )
+            print(f"{len(ranked)} direction insight(s) (latest, non-suppressed):\n")
+            for r in ranked:
+                print(f"  {r.get('id')}  [conf={float(r.get('confidence') or 0.0):.2f}]  {r.get('label','')}")
+                settled = r.get("settled") or []
+                if settled:
+                    print(f"    settled:   {settled[0]}")
+                print(f"    OPEN EDGE: {r.get('open_edge','')}")
+                print(f"    members:   {len(r.get('member_entry_ids') or [])} entries\n")
+    elif args.suppress_direction is not None:
+        ok = engine.journal.suppress_direction_insight(args.suppress_direction.strip())
+        print(
+            f"Suppressed direction {args.suppress_direction.strip()}."
+            if ok else
+            f"No direction insight with id {args.suppress_direction.strip()!r}."
+        )
     elif args.export_directive:
         engine.export_directive_for(args.export_directive.strip())
     elif args.export_directives_bundle: