Search before asking
Motivation
DataVectorWriter and DedicatedFormatWriter both coordinate normal data files with separate blob/vector files. They duplicate normal-file output, rolling checks, prepare/close handling, and parts of the metadata handoff and abort cleanup logic.
Maintainers need to keep these paths consistent when changing failure handling or file metadata. Some composite-only state, such as _pending_normal_meta, also lives in DataWriter, although ordinary append and primary-key writers do not use it.
Solution
Introduce CompositeDataWriter for the shared lifecycle and its state. Keep column splitting, child-writer management, consistency checks, and blob/video-specific rules in the concrete writers.
Reuse physical format dispatch through DataWriter. Preserve existing metadata ordering, retry behavior, and file ownership during abort, including the cleanup from #9917.
Are you willing to submit a PR?
Search before asking
Motivation
DataVectorWriterandDedicatedFormatWriterboth coordinate normal data files with separate blob/vector files. They duplicate normal-file output, rolling checks, prepare/close handling, and parts of the metadata handoff and abort cleanup logic.Maintainers need to keep these paths consistent when changing failure handling or file metadata. Some composite-only state, such as
_pending_normal_meta, also lives inDataWriter, although ordinary append and primary-key writers do not use it.Solution
Introduce
CompositeDataWriterfor the shared lifecycle and its state. Keep column splitting, child-writer management, consistency checks, and blob/video-specific rules in the concrete writers.Reuse physical format dispatch through
DataWriter. Preserve existing metadata ordering, retry behavior, and file ownership during abort, including the cleanup from #9917.Are you willing to submit a PR?