Skip to content

[python] Avoid redundant blob stream opens on index cache hits - #9974

Open
XiaoHongbo-Hope wants to merge 1 commit into
apache:masterfrom
XiaoHongbo-Hope:codex/avoid-blob-cache-open
Open

XiaoHongbo-Hope wants to merge 1 commit into
apache:masterfrom
XiaoHongbo-Hope:codex/avoid-blob-cache-open

Conversation

@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor

Purpose

Avoid opening a BLOB file when its cached index is sufficient to return scalar descriptors. Open streams on index misses and when payloads, nested layouts, or video metadata require bytes. Keep the cache, public API, and file format unchanged.

Validation

  • Real local files with known sizes: cold + warm descriptor reads reduce opens from 2 to 1, with identical results.
  • 259 related tests passed, covering payload reads, ARRAY/MAP BLOB, row selection, reader isolation, error cleanup, deferred resolution, and video.
  • flake8, license checks, and git diff --check passed.

Measurements count Python FileIO calls; OSS HEAD/GET traffic and Rust native reads were not measured.

@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review September 19, 2026 03:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant