Sealed code generation cannot seat: graph activation loops on 'code graph database conflict' every retry, no generation ever serves

[ScriptedAlchemy]

Observed on the operator daemon (proj_a5b3d7e3ebe14ca7), 2026-08-31, PR #707 tip binary

After the #762 fix handoff (pin adb623efecb7, binary built from PR branch + #764 shutdown lane), the operator daemon runs memory-stable (HWM 29.9 GiB, RSS returning to ~14 GiB after each phase, no OOM in 2+ hours — the #762 shape is gone). But the code index never converges to a serving generation:

• The startup active generation is refused: active generation is incompatible with the production owner configuration (expected after the binary upgrade — sanitizer/chunker/policy revisions moved).
• The scheduler stages and seals a fresh generation (staging finished ~07:46, sealed artifact tracedecay.sealed/278bea7a…).
• Every graph activation attempt fails with code-index graph projection failed: code graph database conflict (GraphDbError::Conflict), including attempts made after the fresh seal (07:48, 07:58, 08:11, spaced by the 600 s seat backoff). Each attempt burns ~30–70 s before failing, consistent with a full replay/verification pass.
• event=code_index_graph_activation_retry_scheduled repeats forever; retention passes log no code generation is currently serving for this project and graph_replay_release_failed error=DeadlineExceeded every ~100 s.

Journal state at the time (read-only sqlite, project db)

• graph_publication_replay_v1: newest code rows are sequences 10 and 11 (two code-scope namespaces); newest overall is 14 (project-memory). No new journal row was ever written for the freshly sealed generation — the publish conflicts before journaling.
• graph_publication_replay_tombstones_v1: empty.
• graph_verified_heads_v1: heads exist at replay sequences 6, 7, 8, 10, 11 (five code-scope namespaces).

So the wedge is deterministic and pre-journal: GraphPublicationOperations prepare/replay returns Conflict on every attempt for both the crashed incarnation's sealed generation and a freshly sealed one. Candidate sites are the expected-prior-head mismatch arm (registry/publication.rs, the return Err(GraphDbError::Conflict) after the historical-seat guard) and the sealed-digest mismatch checks; the mapped Conflict is fieldless, so the log line carries no discriminating context.

Why this matters

• The daemon serves stale text (serving_branch: master) indefinitely; graph queries degrade; retention semantic passes stay degraded.
• The retry loop treats a deterministic verdict as retryable — the same shape as Corrupt durable graph store is never quarantined: graph activation retries GRAFEO-X002 forever, keeping the project runtime unadmitted #763 but on the Conflict path rather than corruption: retry Replace session memory with holographic fact store #2 with an identical verdict should be terminal for these inputs (discard + reseal, or a typed refusal), never an infinite 600 s loop.
• GraphDbError::Conflict being fieldless makes the operator log undiagnosable; the conflict site should carry context (which check, which head/sequence pair).

Repro pointers

• Live daemon PID 2408079 journal, 06:29–08:15 UTC window (journalctl --user -u tracedecay.service), plus RSS curves in /fast/tmp/td-sem-verify/rss5.ndjson.
• The wedge survived a clean daemon restart (same conflicts on the prior incarnation from 06:36) and a fresh seal, so it reproduces from the durable journal + store state.

[HashemKhalifa]

@ScriptedAlchemy can I help with some of these issues?

[ScriptedAlchemy]

Narrowed substantially with read-only evidence from the live profile (daemon pid 2408079, still wedged at 08:51 after 2h24m up, memory healthy).

The unseatable generation is an exact replay of an already-verified publication

• The generation sealed at 07:30 in this incarnation (tracedecay.sealed/ab2afc09…, form replay, physical namespace generation:ab2afc09…, 652,290 entities / 743,657 relations) is byte-identical in identity to journal row seq 10: namespace code-scope:sha256:e9b966f4…, generation code-graph:sha256:1a7ccac5…, recovered digest sha256:f715c233….
• The prior incarnation's sealed generation (105e1fcb…, sealed 05:23) is likewise exactly journal row seq 11 (namespace d0f29cd7…, generation 0acb8014…, digest 23b538ea…).
• Both rows already have verified heads at their own sequence, and for all five code-generation heads graph_publication_replay_v1.expected_recovered_digest == graph_verified_heads_v1.recovered_digest. No tombstones. The relational journal is fully coherent and exact-replay-consistent.

So the loop is: the scheduler reseals the same content-addressed generation, the publish resolves to the exact-verified-replay / historical-seat arm in registry/publication.rs (current != expected_prior_head, is_current_head == true), and that arm fails with the fieldless Conflict — meaning the failure is on the durable-graph side of the seat, not journal divergence. Each attempt burns ~30–70 s first, consistent with re-staging/verifying the ~1.4M rows from the sealed replay source before the conflict.

Prime suspect: retention collected the head generations while unseated

At 05:03:18 and 05:05:29 (previous incarnation, pid 330187), retention logged:

event=retention_code_generations store=code-index-v1 mode=offline bytes_reclaimed=5535621245 generations_collected=1 text_artifacts_collected=1
event=retention_code_generations store=code-index-v1 mode=offline bytes_reclaimed=1398837411 generations_collected=1 text_artifacts_collected=0

Two code generations (6.9 GB) were collected during the exact window when no generation was serving and seq 10/11 were the newest journaled publications. If that collection removed durable-graph rows (or replay/cleanup state) that the still-current verified heads reference, every subsequent exact-replay seat fails deterministically — matching the observed wedge across a restart and a fresh reseal.

Note: generation retirement only recently started completing — the per-row GQL deletes previously blew every deadline (#762 lane, fixed by session-id deletes). A retention-selection coherence bug (collecting a generation that is still the verified head for its namespace) may have been latent behind that DeadlineExceeded and is now exposed.

First fix steps

1. Give GraphDbError::Conflict context (site + head/sequence/digest involved) — the operator log is currently undiagnosable across ~13 conflict sites.
2. Make retention refuse to collect a generation that is the current verified head (or any head's dependency closure member) for its namespace.
3. Treat a deterministic exact-replay seat failure as terminal for those inputs (repair/reseal under a new identity), not an infinite 600 s retry.

[ScriptedAlchemy]

New clean-room evidence that rules retention collection out as a required precondition — the "retention collected the head generations" suspect above doesn't survive it, though the three fix steps all still stand.

1. Virgin-store reproduction (no retention, no history). On a clean-room profile the first activation exceeds its operation deadline, and the interrupted publication then yields the same deterministic fieldless Conflict on every subsequent retry. No retention pass ever collected anything; there were no prior generations to collect.

2. Independent minimal repro tonight (tip binary 0.1.0-beta.37+b9181367ba82, fresh TRACEDECAY_DATA_DIR, one-file repo). tracedecay init on a brand-new single-file repository, plus a project open for its linked worktree, wedges the same way ~40 s after init — on a store with zero journal history:

WARN … graph activation failed retryably; the sealed generation stays unseated until the scheduled retry
     event="code_index_graph_activation_retry_scheduled" retry_delay_micros=30000000
     error=code-index graph projection failed: code graph database conflict

then identical failures at the 60 s and 120 s retries, retention_maintenance_tick … reclaimed_bytes=0 throughout. In this repro two publishers plausibly raced for the same store (the init-triggered reconcile on the primary checkout and the hook-triggered open of the linked worktree), consistent with an interrupted/contended first publication being the trigger rather than any collected head. Repro artifacts: /fast/tmp/td-470-repro.B7lzNP/daemon.log on the operator host.

Additional blast radius: while wedged, the project's lifecycle stays busy, so tracedecay wipe refuses indefinitely with cannot start wipe: another lifecycle operation is already active; retry after it finishes — the wedge blocks the escape hatch too.

Net: the deterministic-conflict-on-retry shape reproduces from a contended/interrupted first publication on an empty store, so the fix lanes should not assume verified-head loss as the entry condition. Fix steps 1 (conflict context) and 3 (terminal verdict for deterministic seat failure, not an infinite retry) cover both entry paths; step 2 (retention head-coherence guard) remains worth having but is not the gate for this wedge.

[ScriptedAlchemy]

Integrated the recovered #765 stack into PR #707 by no-force fast-forward at ada9dbc08.

Salvaged/rebased commits:

• 46a7211b4 structured GraphDbError::Conflict guard-site/evidence context
• 39093be6d terminal handling for a repeated identical conflict verdict
• 97fae18c9 CAS-shaped interrupted-publication discard/republication recovery
• 7eabeff5f hotpath coverage for the discard path
• ada9dbc08 keeps the new const conflict constructor compatible with the target branch's blanket hotpath instrumentation

No duplicate of 06b720f190, 0e6ddb62df, or 99941aaa4f was introduced.

Evidence before the final target drift:

• 8/8 exact Sealed code generation cannot seat: graph activation loops on 'code graph database conflict' every retry, no generation ever serves #765 acceptance tests passed (structured context, interrupted discard + fresh republish, restart reseat, interrupted convergence, first-conflict retry, repeated-identical terminal verdict, and mount-transition wait).
• full tracedecay-graph-db package suite: 288 passed, 0 failed; rusqlite graph-publication module: 22 passed, 0 failed.
• default affected-package, hotpath, and workspace --all-features checks passed.

The rapidly moving PR branch then introduced an unrelated test-only compile failure in tracedecay-code-index-runtime/src/code_index_scheduler/tests.rs:6898 (E0616: private reconcile_in_progress, from the incompatible-generation lane). After rebasing, the graph/publication acceptance tests still passed, but Cargo cannot currently compile any code-index-runtime lib test far enough to rerun the two scheduler assertions. Leaving #765 open until that owner fixes the target regression and the current-tip scheduler smoke is rerun; the production compile configurations are green.

[ScriptedAlchemy]

The remaining bounded-recovery pieces are now consolidated in PR #707:

• 46a7211b44b1a025e2e1307fcaea7a40479502cb adds structured GraphDbError::Conflict guard-site context, including expected/actual evidence at compare-and-swap guards.
• 39093be6d3881ce5287bfcd2647ce02a64d1e519 makes a consecutive identical conflict verdict for the same sealed generation terminal instead of scheduling unbounded replay attempts; exact/lexical serving remains available and graph status becomes typed unavailable.
• Interrupted publication recovery stays with the already-landed Grafeo resume/discard authority rather than adding the superseded parallel implementation. The journaled-publication resume test passed under the coordinated Cargo lane; the orphaned-pending-replay recovery test also passes in isolation.
• ccc00c9a2 strengthens the verified-generation contract to assert the exact production guard site (replay.validate_publication_manifest_identity).

I did not duplicate the old interrupted-publication commits (048c5acebb, ab454f3a58) because current Grafeo owns that recovery path. Leaving this issue open until the operator-store reproduction confirms convergence on the shipped PR tip.

[ScriptedAlchemy]

Controlled forward-only operator recovery evidence (2026-08-31)

Deployed:

• Commit 146fae37d4fe572ea3645b99ed54dac13d57ac50 from PR feat: TraceDecay V2 delivery (PR8–PR13) + daemon performance, memory, and reliability overhaul #707 target branch.
• Includes activation fixes 06b720f190, 0e6ddb62df, 99941aaa4f, session projection fix 895d8c1cf2, retention/pool/callers/LCM fixes, and interrupted-publication recovery 97fae18c9.
• Shipping version: 0.1.0-beta.37+146fae37d4fe572ea3645b99ed54dac13d57ac50.
• SHA-256: 68f97ba8a05e3f03d0aed4459a9961175dad9eb194b228944a6c3e1157e8adc5.
• CLI and daemon runtime report the same version.

Reset scope:

• Stopped the daemon and cleared only the target project’s derived grafeo/sealed/replay publication state and five graph journal tables, then repaired the empty semantic-vector staging schema from the canonical schema.
• Memory/session/evidence authorities were not cleared. Post-recovery counts remain memory_facts=13, memory_current=13.
• Forward-only warning retained: do not run beta.37-or-older binaries against the v4 compact store.

Activation evidence:

• Stage completed in one pass: 4,857/4,857 files, 2,955 pages, 623,420,194 payload bytes; staging elapsed 568.688 s.
• The manufactured stage-boundary deadline did not recur.
• Initial activation still produced a real code_graph_manifest.bind conflict at 20:52:56Z, scheduled one 30 s retry, then classified the identical conflict terminal at 20:54:06Z. Therefore the one-pass/no-retry acceptance criterion was not met.
• Peak RSS during the first publication was 41.57 GB on a 94.8 GB host; memory PSI remained low and the service was not OOM-killed.

Restart/reopen evidence on the final binary:

• Final restart requested 21:42:32Z; PID changed 2088399 -> 3884950, socket ready 21:43:23Z.
• Graph reopened from partial_unverified_restore/pending to ready at 21:46:55Z (3m32s after process start; 4m23s after restart request), with no watchdog restart.
• Canonical bare-ID traversal works after reopen: callers for symbol.v1.sha256:38ac89e1b0fef8d4f839370999ca17c6949dcf60f6a19b77df79253f2871517c returned 7 edges; callees for symbol.v1.sha256:4d3cccf885e47db0f6e6fb73f162c0c0c2000c120f5af9a3ecf26caac4b9550c returned the expected scheduler edge.
• Status truthfully reports stale-serving graph ready while a newer live-worktree rebuild runs.
• Final service: active, PID 3884950, zero watchdog restarts, cgroup RSS 14.65 GB, post-restart peak 21.41 GB, about 3.1 CPU cores during rebuild.

Session/runtime evidence and residual failures:

• Scoped Cursor message_search succeeded with application elapsed 2.116 s.
• Explicit Cursor catch-up/projection is still unavailable under typed HistoricalConvergence; logs also show cursor_conflict and durable observation identity collisions. The live 114x projection benchmark was not reproduced, so no throughput success claim is made.
• doctor confirms the daemon is installed/enabled/running and version-matched, but still reports active indexing and durable refused-ingest records.

Evidence retained under /home/zack/.tracedecay/operator-recovery-765-20260831T1330Z/, including build logs, activation/status timelines, canonical traversal outputs, restart proof, runtime snapshots, journal evidence, and the earlier hotpath report hotpath-activation-ee0204d1.json.

Disposition: leaving #765 open. Service is restored and traversal survives restart, but the graph-bind retry and session historical-convergence failures do not satisfy the issue’s full acceptance criteria.