[python] Preserve declared primary-key order in merge reads - #9801
Conversation
There was a problem hiding this comment.
🟢 Approval recommended
No unresolved blocking issues were identified.
Pull request overview
Fixes merge-read deduplication by preserving declared composite primary-key order.
Changes:
- Builds merge keys from ordered trimmed primary-key fields.
- Adds coverage for projections, partitions, reordered keys, and schema changes.
File summaries
| File | Summary |
|---|---|
paimon-python/pypaimon/tests/schema_evolution_pk_read_test.py |
Tests primary-key position changes across schemas. |
paimon-python/pypaimon/tests/reader_primary_key_test.py |
Tests composite-key ordering and projections. |
paimon-python/pypaimon/read/split_read.py |
Aligns merge-key construction with writer sort order. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: CLEAN.
Reviewed e674f81862d0. Merge keys must follow the declared primary-key order used to sort the data files. Using trimmed_primary_keys_fields fixes duplicate/stale logical rows when schema field order differs, while preserving key IDs and types and excluding partition keys.
Validation: all 70 primary-key reader, schema-evolution, projection/predicate and sequence-field tests passed with PyArrow 19.0.1. Current head CI is green.
No actionable implementation regression found in this review.
Purpose
Fixes #9800
When a composite primary key is declared in a different order from the table columns, PyPaimon can return both the old and updated row. For example, four orders with
PRIMARY KEY (order_id, customer_id)produce five rows after one update whencustomer_idprecedesorder_idin the table schema.Build merge-read key fields from the table's ordered trimmed primary-key fields, matching the writer's sort order. This preserves field IDs, types and the storage format. Regression coverage checks complete merged results, value-only projections, partition-key trimming and a primary-key column position change across schema versions.
Tests
Validation: the new regression tests fail before the fix; all 70 tests in
reader_primary_key_test,schema_evolution_pk_read_test,projection_predicate_index_testandtest_sequence_field_readpass after it. Flake8 on changed files andgit diff --checkpass. Local validation used Python 3.12.6 and PyArrow 19.0.1.