Kitty Graphics Protocol Discussion

[anthonykim1]

Kitty Graphics Protocol — Feature Inventory

Spec: https://sw.kovidgoyal.net/kitty/graphics-protocol/

Coming from: #5619

Key files:

• Handler: KittyGraphicsHandler.ts
• Types: KittyGraphicsTypes.ts
• KittyImageStorage: KittyImageStorage.ts
• ImageStorage: ImageStorage.ts
• ImageRenderer: ImageRenderer.ts
• Integration tests: KittyGraphics.test.ts

flowchart TD
 A["Shell sends Kitty escape sequence<br/>ESC _ G a=T,f=32,s=2,v=2,i=1;<br/>base64... ESC \\"]
 A --> B

 B["ApcParser matches 'G'<br/>→ KittyGraphicsHandler<br/>registered in ImageAddon.ts"]
 B --> C

 C["start — reset state"]
 C --> D

 D["put — two-phase parsing<br/>1. Scan for semicolon,<br/>buffer keys<br/>2. parseKittyCommand<br/>extracts keys<br/>3. Stream payload into<br/>Base64Decoder"]
 D --> E

 E["end — finalize"]

 E --> F{"More chunks?<br/>m=1"}
 F -->|"Yes"| G["Park decoder in<br/>_pendingTransmissions<br/>keeps decoder alive<br/>across sequences"]
 G -.->|"Next sequence"| C
 F -->|"No"| H{"Action?"}

 H -->|"a=t transmit"| I["_handleTransmit<br/>KittyImageStorage.storeImage<br/>Blob copies bytes off JS heap"]
 H -->|"a=T transmit+display"| J["_handleTransmitDisplay<br/>storeImage as Blob<br/>then render"]
 H -->|"a=q query"| K["_handleQuery<br/>validate format/size<br/>respond OK or EINVAL"]
 H -->|"a=d delete"| L

 J --> M["_decodeAndDisplay"]
 M --> M1{"Image format?"}
 M1 -->|"f=100 PNG"| M1a["Wrap as image/png Blob<br/>browser decodes natively"]
 M1 -->|"f=32 RGBA"| M1b["Raw bytes → ImageData<br/>already has alpha"]
 M1 -->|"f=24 RGB"| M1c["uint32 block interleave<br/>RGB → RGBA<br/>adds alpha=0xFF000000"]
 M1a --> BMP
 M1b --> BMP
 M1c --> BMP
 BMP["createImageBitmap<br/>GPU-backed render-ready<br/>object for efficient<br/>canvas drawing"]
 BMP --> M2["Resize via<br/>createImageBitmap<br/>if c/r columns/rows<br/>specified"]
 M2 --> M3["KittyImageStorage.addImage<br/>→ ImageStorage.addImage<br/>Tiles bitmap into cells,<br/>marks each with<br/>image+tile ID.<br/>Adds scroll marker<br/>for cleanup."]
 M3 --> M4["KittyImageStorage<br/>bridges Kitty ID<br/>→ internal storage ID<br/>via _kittyIdToStorageId<br/>+ _storageIdToKittyId"]
 M4 --> M5{"Cursor<br/>movement<br/>C key"}
 M5 -->|"C=0 default"| M6["Advance cursor<br/>past image"]
 M5 -->|"C=1"| M7["Restore cursor<br/>with scroll correction"]

 subgraph LAYER["Layer Selection"]
   Z1{"z key?"}
   Z1 -->|"z < 0"| Z2["bottom layer<br/>(behind text)"]
   Z1 -->|"z >= 0 or absent"| Z3["top layer<br/>(on top of text)"]
 end
 M2 --> Z1
 Z2 --> M3
 Z3 --> M3

 subgraph DEL["_handleDelete — selector dispatch"]
   L1{"d key selector?"}
   L1 -->|"d=a / d=A"| L2["Abort ALL in-flight decoders<br/>KittyImageStorage.deleteAll()"]
   L1 -->|"d=i / d=I"| L3["Abort targeted decoder<br/>KittyImageStorage.deleteById()"]
   L1 -->|"default (no d key)"| L2
   L1 -->|"other (c,n,p,…)"| L4["Ignored (not yet supported)"]
   L2 --> L5["Delete Blob from _images<br/>Delete bitmap from ImageStorage<br/>Clear kittyId↔storageId maps"]
   L3 --> L5
 end
 L --> L1

 subgraph EVICT["Eviction & Cleanup (KittyImageStorage)"]
   EV1["onImageDeleted callback<br/>fires when ImageStorage<br/>evicts/deletes an image"]
   EV1 --> EV2["_handleStorageImageDeleted<br/>cleans _kittyIdToStorageId<br/>_storageIdToKittyId<br/>_images"]
   EV3["storeImage with same ID<br/>deletes old placement<br/>from ImageStorage first"]
   EV4["_evictUndisplayedImages<br/>when _images exceeds 256<br/>preferentially removes<br/>images without placements"]
 end

 I --> RESP
 M5 --> RESP
 K --> RESP

 RESP["_sendResponse<br/>OK or EINVAL to client<br/>q=1 suppresses OK<br/>q=2 suppresses errors"]
 RESP --> DONE["Client receives response"]

Loading

Legend: ✅ Done | ⚠️ Partial | ❌ TODO | 🚫 Blocked (renderer/architecture limitation)

---

1. Escape Code Format (APC)

#	Feature	Status	MVP	Notes
1.1	APC ESC_G<control>;<payload>ESC\	✅	Yes	registerApcHandler(0x47, ...) in ImageAddon.ts
1.2	Comma-separated key=value control data parsing	✅	Yes	parseKittyCommand() in KittyGraphicsTypes.ts
1.3	Base64-encoded payload with streaming decode	✅	Yes	Wasm Base64Decoder in KittyGraphicsHandler.ts

2. Actions (a key)

#	Feature	Key	Status	MVP	Notes
2.1	Transmit	a=t	✅	Yes	Stores raw data as Blob via KittyImageStorage.storeImage(). Sends Gi=<id>;OK when i provided
2.2	Transmit + Display	a=T	✅	Yes	Transmit then display at cursor. Sends OK after async display completes
2.3	Query	a=q	⚠️	Yes	Validates base64 and raw pixel dimensions, responds OK/error. PNG queries return OK without validating PNG decode
2.4	Put/Display previously transmitted	a=p	❌	No	Requires transmit/display separation. Post-MVP
2.5	Delete	a=d	✅	Yes	Action a=d dispatches to _handleDelete. d key (delete selector) is fully parsed — d=a/d=A deletes all, d=i/d=I deletes by image ID. Default selector is 'a' per spec. Cascades to ImageStorage via KittyImageStorage. 7 unit tests for d/p key parsing + 14 integration tests for delete dispatch
2.6	Animation frame	a=f	❌	No	Entire animation subsystem not implemented
2.7	Animation control	a=a	❌	No
2.8	Compose animation frames	a=c	❌	No

3. Image Formats (f key)

#	Feature	Key	Status	MVP	Notes
3.1	RGBA 32-bit (default)	f=32	✅	Yes	Works correctly. Default is KittyFormat.RGBA per spec
3.2	RGB 24-bit	f=24	✅	Yes	uint32 block interleave: 3 uint32 reads + 4 uint32 writes per 4 pixels, ~6× faster than per-pixel loop. Remainder pixels handled byte-by-byte
3.3	PNG	f=100	✅	Yes	Blob(type:'image/png') → createImageBitmap

4. Transmission Medium (t key)

#	Feature	Key	Status	MVP	Notes
4.1	Direct (inline base64, default)	t=d	✅	Yes	t key parsed as transmission. Explicitly checks and accepts t=d. 13 integration tests for medium rejection
4.2	File path	t=f	❌	?	Requires filesystem access; N/A in browsers. Rejected with EINVAL. Need VS Code API hook for Electron
4.3	Temp file	t=t	❌	?	Same as above, plus must delete after reading. Rejected with EINVAL
4.4	Shared memory	t=s	❌	No	POSIX shm; N/A in browsers. Rejected with EINVAL

5. Compression (o key)

#	Feature	Key	Status	MVP	Notes
5.1	Zlib/deflate compression	o=z	✅	Yes	DecompressionStream('deflate') with fallback to 'deflate-raw'. Applied after base64 decode, before pixel interpretation
5.2	S key validation for PNG + compression	—	❌	No	Spec: "if you use both PNG and compression, then you must provide the S key." Functionally works without it but conformance gap

6. Chunked Transmission (m key)

#	Feature	Status	MVP	Notes
6.1	Multi-chunk transfer (m=1 / m=0)	✅	Yes	_pendingTransmissions map accumulates chunks via shared decoder
6.2	Size limit enforcement across chunks	✅	Yes	Cumulative encoded size tracked in IPendingTransmission.totalEncodedSize
6.3	First chunk carries full control data, subsequent only m+q	✅	Yes	pending.cmd stores first chunk's command. Subsequent chunks' extra keys are silently ignored. _lastPendingKey tracks most recent pending upload for chunks without i=
6.4	Animation frame chunks must include a=f	❌	No	N/A until animation is implemented

7. Image Display Layout

#	Feature	Key	Status	MVP	Notes
7.1	Display at cursor position	—	✅	Yes	Via KittyImageStorage.addImage() → ImageStorage.addImage()
7.2	Scale to columns/rows	c, r	✅	Yes	Resizes bitmap via createImageBitmap; stretches to fill placement rectangle per spec
7.3	Source rectangle crop	x,y,w,h	❌	No	x,y parsed but unused; w,h not parsed. Needs createImageBitmap crop path
7.4	Cell-offset within first cell	X, Y	❌	No	Keys not parsed; sub-cell positioning
7.5	Cursor movement after placement	C	✅	Yes	C key parsed. Default (C=0): cursor at (originX + cols, last row) per spec. C=1 restores cursor with scroll correction. 10 integration tests
7.6	z-index stacking	z	⚠️	Yes	z key parsed. Negative z → bottom layer (behind text), z≥0 → top layer. Within same layer, z-index stored but render order is insertion order (no per-image z-sort within a canvas). 9 integration tests
7.7	Scroll margin clipping	—	❌	No	Spec: images must be clipped when scrolling outside page area

8. Image IDs & Placement IDs

#	Feature	Key	Status	MVP	Notes
8.1	Image ID	i	✅	Yes	Parsed and used as KittyImageStorage._images map key
8.2	Auto-assigned IDs when i not specified	—	✅	Yes	_nextImageId++ in KittyImageStorage
8.3	Image number (terminal assigns ID)	I	⚠️	?	Parsed, i+I conflict validated. But Gi=<id>,I=<num>;OK response not sent, and I-only commands don't do image number lookup
8.4	i+I conflict → EINVAL	—	✅	Yes	Validated in both put() and _handleNoPayloadCommand()
8.5	Placement ID	p	⚠️	No	Parsed in parseKittyCommand() as placementId. Not yet used for multiple placements of same image
8.6	Multiple placements of same image	—	❌	No	ImageStorage assumes 1 image = 1 placement
8.7	outsideId → storageId mapping	—	✅	Yes	KittyImageStorage manages bidirectional _kittyIdToStorageId / _storageIdToKittyId maps. onImageDeleted callback keeps maps in sync when ImageStorage evicts images. Used by delete, re-transmit, and eviction flows
8.8	Re-transmit same ID cleans up old placements	—	✅	Yes	KittyImageStorage.storeImage() checks for existing _kittyIdToStorageId entry and calls _storage.deleteImage() on old placement before storing new data. 2 integration tests (re-transmit with a=T, re-transmit with a=t then a=T)

9. Delete Commands (a=d, d key)

#	Feature	Key	Status	MVP	Notes
9.1	Delete all visible	a / A	✅	Yes	d key parsed via deleteSelector. Default selector is 'a' per spec. KittyImageStorage.deleteAll() clears _images, iterates _kittyIdToStorageId to delete from ImageStorage, clears maps. Also aborts all in-flight uploads. 4 integration tests (d=a, d=A, storage cleanup, pixel cleanup)
9.2	Delete by image ID	i / I	✅	Yes	d=i/d=I dispatched via switch/case. KittyImageStorage.deleteById() removes from _images, cascades to ImageStorage. Aborts only the targeted in-flight upload. 6 integration tests (d=i, d=I, storage cleanup, pixel cleanup, no-id edge case, targeted abort)
9.3	Delete by image ID + placement ID	i/I + p	❌	No	p key parsed but placement IDs not fully implemented
9.4	Delete by image number	n / N	❌	No
9.5	Delete at cursor position	c / C	❌	No
9.6	Delete at specific cell	p / P	❌	No
9.7	Delete by z-index at cell	q / Q	❌	No	Requires z-index support
9.8	Delete by ID range	r / R	❌	No
9.9	Delete by column	x / X	❌	No
9.10	Delete by row	y / Y	❌	No
9.11	Delete by z-index	z / Z	❌	No
9.12	Delete animation frames	f / F	❌	No
9.13	Lowercase = keep data, uppercase = free data	—	❌	?	Behavioral distinction not implemented. TODOs in KittyGraphicsHandler.ts and KittyGraphics.test.ts
9.14	Abort in-flight chunked upload on any delete	—	✅	Yes	_handleDelete releases decoders and clears _pendingTransmissions before deleting images. d=a/d=A aborts all uploads; d=i/d=I aborts only the targeted upload. 4 integration tests

10. Response / Acknowledgement System

#	Feature	Status	MVP	Notes
10.1	OK response for queries (ESC_Gi=<id>;OK ESC\)	✅	Yes	Works for a=q
10.2	OK response for transmit/display	✅	Yes	a=t sends OK synchronously after storing blob. a=T sends OK after async display via .then(). Only responds when cmd.id provided. Respects q=1/q=2
10.3	Error responses (EINVAL, etc.)	✅	Yes	EINVAL for invalid data, i+I conflict, missing dimensions, unsupported transmission medium, unsupported action
10.4	Quiet mode q=1 (suppress OK)	✅	Yes
10.5	Quiet mode q=2 (suppress errors)	✅	Yes
10.6	ENOENT response for missing image on a=p	❌	No	Depends on a=p
10.7	Placement ID in response (Gi=<id>,p=<pid>;OK)	❌	No	Placement IDs not fully implemented
10.8	Image number in response (Gi=<id>,I=<num>;OK)	❌	No	Depends on §8.3

11. Unicode Placeholders

#	Feature	Status	MVP	Notes
11.1	U+10EEEE placeholder character	❌	No	Requires parser + renderer support
11.2	Virtual placement (U=1)	❌	No
11.3	Row/column diacritics	❌	No
11.4	Image ID encoding via foreground color	❌	No
11.5	Placement ID via underline color	❌	No
11.6	Diacritic inheritance from left cell	❌	No

12. Relative Placements

#	Feature	Key	Status	MVP	Notes
12.1	Parent image/placement reference	P, Q	❌	No	Not parsed
12.2	Horizontal/vertical offset	H, V	❌	No	Not parsed
12.3	Chain depth limit (≥8)	—	❌	No
12.4	Cycle detection (ECYCLE error)	—	❌	No
12.5	Missing parent (ENOPARENT error)	—	❌	No

13. Animation

#	Feature	Key	Status	MVP	Notes
13.1	Transfer frame data	a=f	❌	No	Entire animation subsystem
13.2	Frame composition from previous frames	c	❌	No
13.3	Partial frame updates	x,y,s,v	❌	No
13.4	Frame gap / timing	z	❌	No
13.5	Background color for frames	Y (RGBA)	❌	No
13.6	Alpha blend vs overwrite composition	X=1	❌	No
13.7	Animation control — stop/run/loop	a=a, s, v	❌	No
13.8	Client-driven frame switching	a=a,c=<frame>	❌	No
13.9	Frame composition command	a=c	❌	No
13.10	Gapless frames	z=<neg>	❌	No

14. Terminal Interaction

#	Feature	Status	MVP	Notes
14.1	Clear images on terminal reset (RIS)	✅	Yes	reset() clears KittyImageStorage maps and _pendingTransmissions. Wired via DECSTR, RIS, onRequestReset
14.2	Clean up image data on ESC[2J	✅	Yes	ImageStorage's marker disposal fires onImageDeleted callback → KittyImageStorage._handleStorageImageDeleted cascades cleanup to _images, _kittyIdToStorageId, _storageIdToKittyId. No stale entries
14.3	Clear images on alt buffer switch (1049)	✅	Yes	ImageStorage's wipeAlternate() calls _delImg() which fires onImageDeleted → KittyImageStorage._handleStorageImageDeleted cleans kitty maps. Fully cascaded
14.4	Images scroll with text	✅	Yes	ImageStorage marker-based scrolling
14.5	Erase text commands don't affect images	✅	Yes	ImageStorage is separate from text buffer
14.6	Window size query (TIOCGWINSZ / ESC[14t / ESC[16t)	✅	Yes	Pre-existing xterm.js support

15. Memory & Storage

#	Feature	Status	MVP	Notes
15.1	Storage quota / eviction for displayed images	✅	Yes	ImageStorage has pixel-based eviction. KittyImageStorage enforces _maxStoredImages = 256 cap. onImageDeleted callback keeps kitty maps in sync when ImageStorage evicts. 2 integration tests (memory limit eviction, scrollback eviction)
15.2	Raw data offloaded to Blob (off JS heap)	✅	Yes	IKittyImageData.data is Blob
15.3	Per-image size limit	✅	Yes	kittySizeLimit option enforced during streaming decode
15.4	Preferentially delete images without placements	✅	Yes	KittyImageStorage._evictUndisplayedImages() preferentially removes images that have no _kittyIdToStorageId entry (i.e., not displayed)
15.5	Eliminate double storage	❌	No	Raw Blob lives in KittyImageStorage._images AND rendered ImageBitmap lives in ImageStorage. Two separate stores but now coordinated via onImageDeleted callback. Still two copies of data — post-MVP to unify

16. Control Data Keys — Parse Status

Click to expand full key table

Key	Name	Parsed	Used	Notes
a	Action	✅	✅	t, T, q, d handled. p, f, a, c not handled
f	Format	✅	✅	24, 32, 100. Defaults to RGBA (f=32) per spec
t	Transmission medium	✅	✅	Parsed as transmission. t=d accepted; t=f, t=t, t=s rejected with EINVAL. 13 integration tests
s	Source width	✅	✅
v	Source height	✅	✅
S	Data size	❌	—	Not parsed. Needed for PNG+compression validation
o	Compression	✅	✅	z (zlib/deflate)
m	More chunks	✅	✅	Chunked transmission
i	Image ID	✅	✅
I	Image number	✅	⚠️	Parsed, conflict validated, not used for lookup
p	Placement ID	✅	❌	Parsed as placementId in parseKittyCommand(). Not yet used for multiple placements
q	Quiet mode	✅	✅	1=suppress OK, 2=suppress errors
x	Source X offset	✅	❌	Parsed, not used in display
y	Source Y offset	✅	❌	Parsed, not used in display
w	Source width (crop)	❌	—	Not parsed
h	Source height (crop)	❌	—	Not parsed
X	Cell X offset	❌	—	Not parsed
Y	Cell Y offset	❌	—	Not parsed
c	Display columns	✅	✅
r	Display rows	✅	✅
C	Cursor movement policy	✅	✅	C=0 advance cursor, C=1 no movement
z	Z-index	✅	✅	Parsed as zIndex. Negative → bottom layer, ≥0 → top layer. Stored in ImageSpec. 9 integration tests
d	Delete selector	✅	✅	Parsed as deleteSelector. Dispatches a/A (all), i/I (by ID). Default 'a' per spec
P	Parent image ID	❌	—	Relative placements
Q	Parent placement ID	❌	—	Relative placements
H	Horizontal offset	❌	—	Relative placements
V	Vertical offset	❌	—	Relative placements
U	Unicode virtual placement	❌	—	Unicode placeholders

---

MVP Checklist (Feb 9 – 20)

Items from the chart above that need work before initial merge. Ordered by priority.

P0 — Clients break without these

• §7.5 — Cursor movement — C key parsed. Default (C=0): cursor at (originX + cols, lastRow). C=1: cursor restored. Fixed proportional resize to spec-correct stretch. 10 integration tests (5 unskipped + 5 updated). 2 unit tests for C key parsing.

P1 — Important for correctness

• §9.13 — Strict malformed sequence handling — Abort on invalid keys, unexpected payload. Document policy. (PR feedback)
• Share Base64Decoder with IIP — Both KittyGraphicsHandler and IIPHandler create independent wasm Base64Decoder instances. Avoid duplicate wasm memory. (PR feedback)

Done

• §8.7 — outsideId → storageId map — KittyImageStorage manages bidirectional _kittyIdToStorageId / _storageIdToKittyId. onImageDeleted callback keeps maps in sync when ImageStorage evicts. Delete cascades to ImageStorage.
• §8.8 — Re-transmit same ID cleanup — KittyImageStorage.storeImage() checks for existing _kittyIdToStorageId entry and calls _storage.deleteImage() on old placement before storing new data. 2 integration tests.
• §9.1-9.2 — Parse d key + delete dispatch — DELETE_SELECTOR and PLACEMENT_ID added to KittyKey. deleteSelector parsed as string, placementId parsed as number. _handleDelete dispatches via switch/case: d=a/d=A → KittyImageStorage.deleteAll(), d=i/d=I → KittyImageStorage.deleteById(). Default selector is 'a' per spec. d=a aborts all in-flight uploads; d=i aborts only targeted upload. 7 unit tests for d/p key parsing. 14 integration tests for delete dispatch including pixel-level verification. TODOs for lowercase/uppercase distinction.
• §9.2 — Delete by ID cascades to ImageStorage — KittyImageStorage.deleteById() calls _storage.deleteImage() for displayed images.
• deleteImage() public API on ImageStorage — Thin wrapper around _delImg + marker dispose.
• f=24/f=32 integration tests — 27 tests with pixel verification, storage, dimensions.
• §10.2 — Send OK for a=t / a=T — a=t sends OK in dispatcher after _handleTransmit. a=T sends OK after async display via .then(). Only when cmd.id provided. Respects q=1/q=2.
• §3.1 — Fix default format — _handleTransmit now defaults to KittyFormat.RGBA (f=32) per spec instead of PNG.
• §3.2 — Bit-shift RGB→RGBA — uint32 block interleave: 3 uint32 reads + 4 uint32 writes per 4 pixels vs 28 byte reads/writes — ~6× faster. Remainder pixels handled byte-by-byte.
• §9.14 — Abort in-flight chunks on delete — _handleDelete releases decoders and clears _pendingTransmissions. d=a/d=A aborts all; d=i/d=I aborts only targeted. 4 integration tests.
• §7.5 — Cursor movement — C key parsed. Default (C=0): cursor at (originX + cols, lastRow). C=1: cursor restored. Fixed proportional resize to spec-correct stretch. 10 integration tests (5 unskipped + 5 updated). 2 unit tests for C key parsing.
• §14.2 — Clean up Kitty blobs on ESC[2J — ImageStorage's marker disposal fires onImageDeleted callback → KittyImageStorage._handleStorageImageDeleted cascades cleanup to _images, _kittyIdToStorageId, _storageIdToKittyId. No more stale entries.
• §14.3 — Clean up Kitty blobs on alt buffer switch — ImageStorage's wipeAlternate() calls _delImg() which fires onImageDeleted → KittyImageStorage._handleStorageImageDeleted cleans kitty maps. Fully cascaded.
• §15.1 — Storage eviction keeps kitty maps clean — KittyImageStorage enforces _maxStoredImages = 256 cap with _evictUndisplayedImages(). onImageDeleted callback keeps kitty maps in sync when ImageStorage evicts by pixel limit or scrollback. 2 integration tests (memory limit eviction, scrollback eviction).
• §15.4 — Preferentially delete images without placements — KittyImageStorage._evictUndisplayedImages() preferentially removes images that have no _kittyIdToStorageId entry.
• §7.6 — z-index layer selection — z key parsed. Negative z → bottom layer (behind text), z≥0 → top layer. ImageStorage.addImage() accepts layer and zIndex params. 9 integration tests including DOM order verification.
• §4.1 — t key parsing + rejection — t key parsed as transmission via KittyKey.TRANSMISSION. t=d accepted; t=f/t=t/t=s rejected with EINVAL for both transmit and query actions. 13 integration tests.