Skip to content

[python] Add block-based manifest Avro reader - #9826

Merged
JingsongLi merged 3 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/python-manifest-selective-reader
Sep 15, 2026
Merged

JingsongLi merged 3 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/python-manifest-selective-reader

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Purpose

Speed up PyPaimon manifest reads by decoding Avro object-container blocks directly and applying partition and bucket pruning before materializing nested DataFileMeta rows. This aligns the Python path with the Java reader introduced by #9160.

Changes

  • Read standard-order manifest entries by Avro block.
  • Decode the partition and bucket prefix first and skip _FILE for rejected entries.
  • Preserve the generic fastavro.reader() path when no early filter is present.
  • Fall back to the generic reader for historical manifests with reordered top-level fields.
  • Support legacy and current fastavro named-schema layouts.
  • Avoid executor overhead for single-manifest reads.

Benchmark

In a fixed-snapshot planning benchmark on a large partitioned table, this reduced planning time by 33.5% (1.50x) while producing an identical scan plan.

Tests

  • Manifest selective decoding, reordered fields, and named-type references
  • Partition predicate regression tests
  • Python 3.6 / fastavro 1.4.7 compatibility
  • fastavro 1.11.1 and 1.12.2 named-schema compatibility
  • BTree Java-to-Python E2E on Python 3.6.15
  • Flake8 and git diff --check

@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Skip pruned manifest file metadata decoding [python] Add block-based manifest Avro reader Sep 15, 2026
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review September 15, 2026 04:22
@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit bf73a19 into apache:master Sep 15, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants