feat(transcript): externalize images (Claude Code, Codex) + capture Cursor sidecar images#1589
Open
suhaanthayyil wants to merge 16 commits into
Open
feat(transcript): externalize images (Claude Code, Codex) + capture Cursor sidecar images#1589suhaanthayyil wants to merge 16 commits into
suhaanthayyil wants to merge 16 commits into
Conversation
Add a per-agent image codec that lifts inline base64 images out of a session transcript into externalized assets and re-injects them byte-for-byte on restore. Only Claude Code (which embeds images as base64) registers a codec today; every other agent resolves to nil and its transcript flows through untouched. The codec swaps only the base64 value in place (never re-marshalling the JSON) and refuses to externalize any image whose raw bytes don't re-encode to the exact original string, so ReinjectImages(ExtractImages(x)) == x. A low-entropy, path-bearing placeholder (entire-asset:assets/img-<id>.<ext>) is left in the log so it survives redaction and still reads meaningfully to an agent summarizing the transcript. Values shorter than the 32-char placeholder id are left inline and replacements run longest-first, so a value can never corrupt a placeholder or another value. Also add the shared surface consumed by the storage layer: the TranscriptAsset model and AssetsManifest pointer in the checkpoint metadata contract, assets path constants, and the opt-in redaction.externalize_images setting (off by default; also togglable via ENTIRE_EXTERNALIZE_IMAGES).
Wire the image codec into the checkpoint store/restore paths behind the opt-in flag. During condensation, externalize inline images BEFORE redaction (base64 is high-entropy and would otherwise be flagged), then persist the extracted assets as raw blobs under the session's assets/ folder plus an assets/manifest.json index, in the same atomic commit. git content-addresses the blobs, so identical images dedupe across checkpoints. On read, ReadSessionContent re-injects images whenever the stored transcript carries placeholders, gated on placeholder presence rather than the config flag, so a checkpoint written with the flag on always restores correctly. Preserve the assets/ subtree verbatim through the pre-push OPF rewrite: the raw image blobs are opaque binary with no redactable text, so both the collect and rebuild passes skip them. This avoids corrupting the images and avoids the fail-closed rebuild aborting the push.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an opt-in pipeline to externalize Claude Code inline base64 images from checkpoint transcripts into a per-session assets/ store (plus assets/manifest.json), replacing them with stable placeholders on write and re-injecting the original bytes on read. This reduces transcript bloat while preserving byte-exact restore behavior and ensures OPF rewrite preserves the binary assets subtree.
Changes:
- Introduces
transcript/imageextractwith a Claude Code codec to extract/reinject base64 image payloads via placeholders. - Wires extraction into manual-commit condensation before redaction and persists extracted assets into checkpoint trees; restores by reinjecting assets on read when placeholders exist.
- Updates OPF rewrite tree-walkers to skip/preserve
assets/verbatim; adds settings and tests covering the new behavior end-to-end.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| cmd/entire/cli/transcript/imageextract/imageextract.go | New codec for extracting/reinjecting inline base64 images using placeholders. |
| cmd/entire/cli/transcript/imageextract/imageextract_test.go | Unit tests for codec round-trip, dedupe, substring handling, and entropy constraints. |
| cmd/entire/cli/strategy/manual_commit_opf_rewrite.go | Skips assets/ during OPF blob collection and preserves it verbatim during rebuild. |
| cmd/entire/cli/strategy/manual_commit_opf_rewrite_test.go | Adds regression coverage asserting OPF rewrite preserves assets/ byte-identically. |
| cmd/entire/cli/strategy/manual_commit_condensation.go | Runs image externalization pre-redaction and plumbs extracted assets into checkpoint write options. |
| cmd/entire/cli/strategy/condense_images_test.go | Tests the externalization flag behavior and extract→redact→reinject ordering. |
| cmd/entire/cli/settings/settings.go | Adds redaction.externalize_images setting and env override helper. |
| cmd/entire/cli/settings/settings_images_test.go | Tests default/setting/env behaviors for enabling image externalization. |
| cmd/entire/cli/paths/paths.go | Adds assets/ path constants for consistent tree/file references. |
| cmd/entire/cli/checkpoint/persistent.go | Writes assets/ blobs + manifest into checkpoint trees and reinjects assets on read. |
| cmd/entire/cli/checkpoint/persistent_assets_test.go | End-to-end persistent-store test for placeholder storage + byte-exact reinjection. |
| cmd/entire/cli/checkpoint/aliases.go | Exposes TranscriptAsset API type via CLI aliases. |
| api/checkpoint/metadata.go | Adds TranscriptAsset type, WriteOptions.Assets, and session assets_manifest metadata pointer. |
…erge mergeRedaction handled pii, custom_redactions, and openai_privacy_filter but not externalize_images, so setting it in settings.local.json (the natural place to opt into a rollout flag) was silently dropped — and a local value of false could not turn off a base value of true. Handle the key in the merge so both the per-machine enable and the per-machine kill switch work.
Address review feedback and audit findings on the image codec: - Surface crypto/rand failures from newAssetID instead of swallowing them, so a broken entropy source can't silently yield an all-zero, collision-prone id. - Guarantee distinct asset names within a transcript (uniqueImageName), so the round trip holds even if the id source degenerates. - Accept array-rooted JSONL lines, matching the recursive collector's contract. - Skip base64 values below the placeholder id length, and only record an asset when the value swap actually replaced something (never orphan an asset). The value swap replaces the base64 itself (not a "data":"<b64>" wrapper): Claude Code serializes the image block both compactly and with a space after the colon depending on version, so the bare value is the only format-agnostic anchor. A copy of the same base64 in a text field is swapped too, but the round trip stays byte-exact because reinjection restores every occurrence identically. Verified byte-exact extract/reinject on both a spaced real fixture and a compact 17MB real Claude Code transcript with inline images.
At session stop, finalizeAllTurnCheckpoints rewrote each turn checkpoint's full.jsonl with the raw live transcript, running redaction but not externalization. That re-inlined the base64 images condensation had lifted out and orphaned the assets/ blobs, so the feature never persisted for a session with mid-turn commits. - finalize now externalizes the full transcript before redaction (mirroring CondenseSession) and passes the assets through UpdateOptions. - The transcript backfill writes those assets (clearing any stale ones) and reconciles the root assets_manifest pointer, but only when replaceTranscript actually rewrote the transcript: on a content-hash short-circuit the stored transcript and assets are unchanged, so touching assets would strip the blobs a still-present placeholder depends on (a dangling placeholder on restore). - Externalization no longer mutates the caller's transcript: CondenseResult and the CheckpointTranscriptSize growth baseline stay based on the raw inline transcript, which is what the shadow-branch blob is compared against (mutating it reported spurious growth on every later commit).
…flow Drive the real entire hook binary on a Claude Code session with an inline base64 image and externalization enabled via settings.local.json: mid-turn commit triggers post-commit condensation, Stop triggers finalize. Assert the actual entire/checkpoints/v1 ref stores a placeholder (not raw base64) that survives finalize, writes the asset blob + manifest.json, and that the restore path (ReadSessionContent) reinjects the image. Exercises the finalize re-externalization and local-settings-merge fixes end to end.
Codex rollout JSONL embeds images as base64 data-URIs (data:image/…;base64,…) in input_image image_url values, user messages, and function_call_output — and inlines multi-MB generated images the same way, which bloats the transcript badly (real sessions here carry 100+ MB of inline image base64). Externalize them into checkpoint assets like Claude Code. - Refactor the codec into a shared extraction engine + reinject routine that take a per-agent collector, so codecs differ only in how they find images. - Add codexCodec keyed on the data-URI marker (not a specific field), covering input, generated, and tool-output images uniformly. The bare base64 value is swapped (the tiny data:…;base64, prefix stays inline), so the swap is format-agnostic and byte-exact. - Detect the media type from the decoded magic bytes (declared type is the fallback): real Codex data-URIs mislabel some PNGs as image/jpeg, and the asset filename/manifest should reflect the actual content. Metadata only — the transcript's declared type is untouched, so the round trip stays byte-exact. Verified byte-exact extract/reinject over 176 images across real Codex rollouts (valid PNG/JPEG magic on every asset), saving up to ~119 MB in one session.
Drive the real Codex hook binary (user-prompt-submit → apply_patch post-tool-use → mid-turn commit condensation → stop finalize) on a rollout transcript with an inline data-URI image, externalization enabled via settings.local.json. Assert the actual entire/checkpoints/v1 full.jsonl stores a placeholder (base64 gone, inside the preserved data-URI) that survives Codex's SanitizePortableTranscript, writes the asset blob + manifest, and that ReadSessionContent reinjects the image.
suhaanthayyil
changed the title
suhaanthayyil
changed the title
suhaanthayyil
force-pushed
the
feat-image-externalize
branch
from
cbf3e98 to
fd15cc2
Compare
suhaanthayyil
changed the title
suhaanthayyil
force-pushed
the
feat-image-externalize
branch
from
fd15cc2 to
7d51237
Compare
Cursor keeps pasted/generated images in a per-session SQLite blob store (~/.cursor/chats/<workspace>/<session>/store.db), not the JSONL transcript Entire condenses, so the transcript image codec used for Claude and Codex cannot reach them. Add a SidecarImageProvider capability that the strategy layer invokes during condensation and finalize: the Cursor agent locates the store for the session, shells out to sqlite3 to read image blobs (filtered by magic bytes), and returns them as checkpoint assets. Preserve/view-only: the images are stored as assets with a manifest but there is no transcript placeholder and no reinject on restore (Cursor reads its own store). Gated on the same redaction.externalize_images opt-in; best-effort, so a missing store, missing sqlite3, or unexpected schema is a silent no-op.
Drive the real Cursor hook flow (session-start -> before-submit-prompt -> stop -> commit condensation) with a store.db holding an image and externalization enabled, asserting the checkpoint captures the image as an asset (blob + manifest, byte-exact) while the transcript stays untouched (no placeholder). Propagate TestEnv.ExtraEnv to git-hook subprocesses so condensation inherits per-test agent store/project overrides (empty for tests that don't set it).
suhaanthayyil
changed the title
Address findings from an adversarial review of the sidecar path: - Bound the sqlite3 shell-out with a 30s timeout so a locked or malformed store.db can never hang the commit/stop hook it runs inside. - Skip stores larger than 64MB and drop any single image over the 50MB blob cap, bounding memory and avoiding an unpushable git object. - Treat a session id as a literal path component (glob only the workspace level, then stat) so glob metacharacters in the id can't match another session's store; reject '.'/'..' ids. - Read every store.db matching the session and union + dedup the images instead of silently taking the first glob match. - Treat an unrecognized store schema (no such table/column) as a silent no-op rather than an error that would warn on every checkpoint, matching the documented best-effort contract. - Capture sidecar images only after the post-redaction skip check, so the sqlite3 work is avoided when the checkpoint is discarded. - Drop the redundant finalize-path capture: sidecar images never change the transcript, so finalize (which only writes assets when the transcript is rewritten) would gate them out; condensation is the single capture point and always runs first for a turn's checkpoints. Add unit coverage for the WEBP magic-byte filter end-to-end and the unrecognized-schema no-op.
Contributor
Author
Self-review pass on the Cursor sidecar pathRan an adversarial multi-lens review over the new Cursor sidecar image-capture diff (the part not covered by the earlier Copilot pass, which was on the Claude/Codex transcript codec). Fixed the confirmed findings in
Added unit coverage for the WEBP magic-byte filter end-to-end and the unrecognized-schema no-op. Verified: |
An earlier hardening commit removed the sidecar-image capture from finalizeAllTurnCheckpoints on the mistaken assumption that finalize never writes sidecar assets. It does: when a mid-turn commit's condensation captures a Cursor image and a later stop finalizes that checkpoint with a grown transcript, replaceTranscript reports rewrote==true and writeAssets clears the entire assets/ folder before re-writing only finalizeAssets — permanently dropping the captured image (for Cursor, finalizeAssets was empty, so the whole folder was cleared). Re-capture sidecar images into finalizeAssets so a rewriting finalize re-writes them too. Content-hash asset names make this idempotent with condensation's write; when the transcript is unchanged, writeAssets is not called and condensation's assets are left intact. Add an integration regression test that drives an active-session mid-turn commit (condensation captures the image) followed by a stop with a grown transcript (finalize rewrites): it fails on the pre-fix code with the image wiped and passes with the fix.
Finalize re-captures sidecar images (best-effort) and, when the transcript is rewritten (rewrote=true), writeAssets clears the whole assets/ folder before re-writing only the fresh set. If the finalize-time capture transiently yields nothing (sqlite3 locked/timed out, temp-dir error) after CondenseSession already captured and stored the images, the clear permanently drops them. Add UpdateOptions.PreserveAssetsWhenEmpty: when set and the update carries no assets, the backfill keeps the already-stored assets/ subtree instead of clearing it. Finalize sets it for sidecar-capable agents (Cursor), where an empty set means a best-effort miss rather than "the transcript has no images" — codec agents (Claude/Codex) leave it false so stale blobs are still cleared. Add an integration regression test that captures at condensation, removes the store.db so the finalize re-capture finds nothing, and asserts the image survives the rewrite (fails pre-fix with the image wiped, passes post-fix).
Image externalization enforced only a minimum size, so an inline image over agent.MaxChunkSize was lifted into a single asset blob that could become an unpushable git object. The transcript itself is chunked under MaxChunkSize (always pushable), so cap externalization at the same limit and leave larger images inline, mirroring the Cursor sidecar guard. Addresses trail #716 review finding. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # cmd/entire/cli/checkpoint/persistent.go
For codec agents, an empty finalize asset set was treated as 'the transcript has no images' and cleared the stored assets/ folder on rewrite. That assumption breaks when externalization was active at condensation (placeholders + assets stored) but inactive at finalize — an ENTIRE_EXTERNALIZE_IMAGES env override not inherited by the hook process, or redaction.externalize_images toggled off mid-session. The finalize then re-inlines the transcript, redaction destroys the inline base64, and clearing the assets loses the images permanently. Track whether extraction actually RAN at finalize: when it did not (flag off, or extraction errored), the empty set means 'didn't run', so PreserveAssetsWhenEmpty keeps the previously-stored blobs. Integration test drives condense-with-flag-on then stop-with-flag-off and asserts the manifest and blobs survive (fails pre-fix with the assets cleared).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Trail: https://entire.io/gh/entireio/cli/trails/716
What: Opt-in externalization/capture of images from checkpoint sessions, for Claude Code, Codex, and Cursor. Each captured image is stored in a per-session
assets/folder (raw blob +assets/manifest.json). Every other agent flows through unchanged (no codec, no sidecar → no-op).Two mechanisms, one storage layout:
entire-asset:assets/img-<id>.<ext>); on restore the exact bytes are re-injected in place. Contract:ReinjectImages(ExtractImages(x)) == x, byte-for-byte.~/.cursor/chats/<workspace>/<session>/store.db), not the JSONL transcript Entire condenses — so the transcript codec can't reach them. A newSidecarImageProvidercapability locates that store, shells out tosqlite3to read image blobs (filtered by magic bytes), and stores them as checkpoint assets. There is no transcript placeholder and no reinject: Cursor reads its own store on restore, so the checkpoint assets are for preservation/viewing.Why / how it helps: Agents inline images as huge base64 in the transcript, which bloats
full.jsonl, makes it expensive to read/summarize, and pushes large blobs to the platform. Externalizing shrinks transcripts, lets git content-address and dedupe identical images across checkpoints, and keeps restore lossless. Codex is an especially big win: real sessions carry 100+ MB of inline image base64 (OpenAI's own issue #22603); one real session in testing saved ~119 MB. Cursor's images never reached the checkpoint at all before — now they're preserved with the session.How:
transcript/imageextractpackage: a shared extraction engine + reinject routine that take a per-agent collector, so codecs differ only in how they find images. Claude Code codec:{"type":"image","source":{"type":"base64",…}}blocks. Codex codec: thedata:image/…;base64,…data-URI marker (input, generated, and tool-output images). Byte-exact via swapping only the bare base64 value in place (never re-marshalling JSON), refusing any image whose bytes don't re-encode to the exact original, longest-first ordering, and a no-orphan guard.agent.SidecarImageProvideroptional capability +agent.AsSidecarImageProviderhelper; the strategy layer calls it during condensation and finalize. The Cursor agent implements it via a best-effortsqlite3shell-out (copies the store to a temp path first so a live session can't lock/mutate it and any WAL is applied). Asset names are content-hashed (img-<sha256>.<ext>), so identical images dedupe.image/jpeg. Metadata only; the transcript's declared type is untouched, so the round trip stays byte-exact.assets/subtree verbatim.redaction.externalize_images(project or local settings) orENTIRE_EXTERNALIZE_IMAGES=1. Off by default. Reinject is gated on placeholder presence, not the flag.Agents intentionally NOT supported (verified against real sessions):
attachmentsalways empty). No-op.Testing:
mise run lint— 0 issues;mise run test:ci— unit + integration + Vogon canary green on the committed tree.ReadSessionContentbyte-exact (Claude/Codex); finalize/backfill re-externalizes and replaces assets; identical-transcript backfill keeps assets; OPF preservesassets/.store.dbextracted and verified.entire/checkpoints/v1placeholder + asset + manifest → restore reinjects byte-exact. Cursor — real hook flow with an image instore.db→ condensation captures it as a checkpoint asset (blob + manifest, byte-exact) while the transcript stays untouched.Risk / rollback: Off by default; no behavior change unless enabled; reinject is a no-op without placeholders; Cursor capture is best-effort (silent no-op if
sqlite3/store absent). Revert by reverting the commits on this branch.Follow-ups: live-confirm Gemini post-migration; platform-side web rendering of the stored assets (the CLI stores them; the entire.io session viewer would need to fetch/render
assets/by placeholder id or manifest).