[python] Consolidate composite writer lifecycle and file output

[wangzhigang1999]

Purpose

Closes #9947.

DataVectorWriter and DedicatedFormatWriter duplicate normal-file output, buffering, rolling checks, and prepare/close handling. Consolidate the shared lifecycle and state in CompositeDataWriter, keeping column splitting, consistency checks, child-writer management, and blob/video-specific rules in the concrete writers. Share physical format dispatch through DataWriter.

Prepare and validate all sidecar metadata before publishing it, then transfer abort cleanup responsibility through _release_prepared_files(). Preserve metadata ordering, lower-level writer retry behavior, externally owned Blob files, and the cleanup from #9917. No public API change is intended.

Tests

• Base and candidate each passed 421 related tests on Linux / Python 3.12.8 / PyArrow 19.0.1, with no skips. Lance and Vortex dependencies were installed. Reuse existing tests, extending the metadata-order case to check a vector preparation failure followed by retry; no new test files.
• Eight real-file failure probes passed across both versions and both composite writers: no partial metadata publication, retry without rewriting prepared files, and removal of owned files on abort. The existing one-shot BatchTableWrite.prepare_commit() contract is unchanged.
• Five workloads derived from the complete 60,000-row MNIST training split passed exact readback, logical metadata comparison, and cross-version reads. Seven alternating runs per workload showed median write-to-close times within 0.9% of the base; Blob + Vector had a +1.17% median paired change, while Lance showed wider variation. These measurements use a warm local filesystem and do not establish OSS or concurrent-writer performance.
• Separate perf and cProfile runs found the main costs in Parquet encoding and per-row Blob processing. Shared lifecycle methods were not a major hotspot. Flake8 and whitespace checks passed.

Validation baseline: d0d2dec0ae4c568fd1b1f92d5b77d8376d286196. Benchmark and profiling scripts are outside this patch. No performance improvement is claimed.

[Copilot]

🟢 Approval recommended

No unresolved review issues remain.

Pull request overview

Consolidates composite writer lifecycle, metadata preparation, cleanup, and physical format dispatch while preserving retry and file-ownership behavior.

Changes:

• Adds shared CompositeDataWriter lifecycle handling.
• Centralizes format dispatch and metadata handoff.
• Updates writers and tests for retry and metadata ordering.

File summaries

File	Description
paimon-python/pypaimon/write/writer/dedicated_format_writer.py	Uses the consolidated lifecycle and sidecar preparation.
paimon-python/pypaimon/write/writer/data_writer.py	Provides shared format dispatch and metadata handoff.
paimon-python/pypaimon/write/writer/data_vector_writer.py	Uses the consolidated lifecycle.
paimon-python/pypaimon/write/writer/composite_data_writer.py	Implements shared composite writer lifecycle and cleanup.
paimon-python/pypaimon/tests/write/write_buffer_test.py	Extends retry and metadata-order coverage.

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[JingsongLi]

+1