Skip to content

fix(memory-ingest): stop two silent transcript-ingest failures - #2699

Open
rayers wants to merge 1 commit into
garrytan:mainfrom
rayers:fix/transcript-ingest-silent-failures
Open

fix(memory-ingest): stop two silent transcript-ingest failures#2699
rayers wants to merge 1 commit into
garrytan:mainfrom
rayers:fix/transcript-ingest-silent-failures

Conversation

@rayers

@rayers rayers commented Aug 26, 2026

Copy link
Copy Markdown
20260826-083142-gstack ## Why (in your own words)

Two independent bugs in bin/gstack-memory-ingest.ts make transcript pages
silently fail to reach the brain, so a user's Claude Code / Codex sessions never
become searchable and the transcripts source can get stuck failing every run.

1. Frontmatter fence gluing. buildTranscriptPage() builds the closing
--- with no trailing newline, and a session body always starts with ##
(never a newline), so the rendered page ends ...---## User. gbrain's
frontmatter matcher (/^---\r?\n([\s\S]*?)\r?\n---(\r?\n|$)/ in
src/core/markdown.ts) requires the closing --- to end its own line. It skips
the glued fence, latches onto the next standalone --- in the transcript body
(a markdown horizontal rule), parses the prose between as YAML, and drops the
whole page with Invalid YAML frontmatter. Transcripts with no later --- fall
back to body-only, silently losing their frontmatter (title/type/tags/session
metadata). This has been present since the V1 transcript-ingest feature.

2. Slug collisions. Two source files can map to one path-derived slug
(transcripts/<agent>/<repo>/<date>-<session_id[:12]>): a session resumed under
the same id on one day, or two session ids that share a 12-char prefix.
writeStaged() names each file ${slug}.md, so the second overwrites the first.
gbrain import then collects N-1 of N staged files and the staged-vs-collected
reconciliation guard (correctly) fails the whole batch. It repeats every run
until the offending inputs age out ("accounted for N-1 of N staged ... Refusing
to advance state").

Live evidence

Before. In a real hourly auto-ingest, the transcripts source failed every
run with the reconciliation guard, and individual pages were dropped:

[memory-ingest] ERR: gbrain import accounted for 103 of 104 staged page(s) ...
  Refusing to advance state ...
Skipped transcripts/.../<session>.md: Invalid YAML frontmatter: can not read a
  block mapping entry; a multiline key may not be an implicit key at line 20

After. With both fixes, the same run reconciles fully and the previously
dropped pages import cleanly:

[import.files] 9/9 (100%) imported=9 skipped=0 errors=0
[memory-ingest] gbrain import: 9 imported, 0 unchanged, 0 failed

The staged page now closes its frontmatter on its own line:

  - date:2026-08-25
---

## User

Regression tests (added in this PR), run with bun test:

$ bun test test/regression-transcript-frontmatter-fence.test.ts \
           test/regression-transcript-slug-collision.test.ts
 4 pass
 0 fail
Ran 4 tests across 2 files.

The fence test embeds gbrain's exact matcher and asserts the frontmatter closes
at the real fence (body prose stays out of YAML); it fails if the fix is
reverted. The slug tests assert deterministic disambiguation and that
non-colliding slugs are untouched. The existing gstack-memory-ingest.test.ts
suite still passes (42/42) with the three new exports.

Scope

  • Changed: bin/gstack-memory-ingest.ts (fence newline in
    buildTranscriptPage; new disambiguateSlugs() called in preparePages;
    export on buildTranscriptPage, renderPageBody, disambiguateSlugs).
  • Added: two regression test files.
  • Verified live by: production auto-ingest run before/after, plus the new
    unit tests and the existing memory-ingest suite.
  • Did NOT test: end-to-end against every gbrain version; remote-http MCP
    staging path (unchanged by this diff).
  • No VERSION / CHANGELOG bump (left to the maintainer's squash-merge). No
    ETHOS.md / voice / promotional changes.
  • This is a fork PR, so only the secretless free-tests check runs; the
    eval/E2E jobs need base-repo secrets and will show empty-env auth failures
    (expected for forks).

Liveness proof (required)

Checklist

  • Liveness screenshot attached: GSTACK PR typed live into a real surface (not edited onto the image)
  • This is not a generated-file-only diff (edited source + added tests)
  • No ETHOS.md edits, and no changes to voice / founder perspective / YC references
  • New public command / external service / host adapter has an accepted issue linked (or N/A) — N/A (bug fix)
  • Linked issue or reproduction: reproduction in "Live evidence" above + regression tests (no separate issue filed)

@trunk-io

trunk-io Bot commented Aug 26, 2026

Copy link
Copy Markdown

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

@rayers
rayers marked this pull request as ready for review August 26, 2026 13:39
Two independent bugs made transcript pages silently fail to reach the brain.

1. Frontmatter fence gluing. buildTranscriptPage() built the closing "---"
   with no trailing newline, and session bodies always start with "## ", so
   the rendered page ended "...---## User". gbrain's frontmatter matcher
   (/^---\r?\n([\s\S]*?)\r?\n---(\r?\n|$)/ in src/core/markdown.ts) requires
   the closing "---" to end its own line, so it skipped the glued fence,
   latched onto the next standalone "---" in the transcript body, parsed the
   prose between as YAML, and dropped the page with "Invalid YAML frontmatter".
   Transcripts with no later "---" fell back to body-only, silently losing
   their frontmatter. Fix: emit the fence on its own line with a blank
   separator, matching renderPageBody()'s artifact branch.

2. Slug collisions. Two source files can map to one path-derived slug (a
   session resumed under the same id on one day, or two ids sharing a 12-char
   prefix). writeStaged() names each file "${slug}.md", so the second
   overwrote the first; gbrain collected N-1 of N staged files and the
   reconciliation guard failed the whole batch every run. Fix:
   disambiguateSlugs() keeps the first occurrence and gives each later collider
   a stable "-<sha8(source_path)>" suffix (deterministic, and slug + page_slug
   move together so writeStaged, the failure mapping, and state recording agree).

Exports buildTranscriptPage, renderPageBody, and disambiguateSlugs for tests.
Adds regression tests for both failures.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@rayers
rayers force-pushed the fix/transcript-ingest-silent-failures branch from 8957f30 to 9563469 Compare August 26, 2026 13:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant