Skip to content

[core][python] Reject dynamic write-cols optimization changes - #9912

Closed
XiaoHongbo-Hope wants to merge 1 commit into
apache:masterfrom
XiaoHongbo-Hope:codex/fix-dynamic-write-cols-persisted
Closed

XiaoHongbo-Hope wants to merge 1 commit into
apache:masterfrom
XiaoHongbo-Hope:codex/fix-dynamic-write-cols-persisted

Conversation

@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor

Problem

Enabling data-evolution.write-cols-optimization.enabled through FileStoreTable.copy() changes how normal Data Evolution files encode writeCols, but the option is not persisted in the table schema. A later reader resolves the file schema without that option and can misinterpret omitted writeCols as the full schema. With a dedicated BLOB column, a Java-written fixture returned a null BLOB in both Java and PyPaimon instead of the stored bytes.

Change

Reject a dynamic override when it changes the effective value of this option in both Java and PyPaimon. Same-value overrides remain allowed. Persisted ALTER TABLE changes remain allowed; the option is deliberately not marked immutable.

Validation

  • Targeted Java tests for the dynamic guard and persisted optimized BLOB reads passed on JDK 8.
  • PyPaimon DedicatedFormatWriterTest: 76 passed, 1 skipped.
  • Java Spotless check passed.

This prevents new ambiguous files. It does not rewrite files already produced with an unpersisted dynamic override.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant