[refactor](be) Support recursive parquet complex readers#64357
Open
suxiaogang223 wants to merge 26 commits into
Open
[refactor](be) Support recursive parquet complex readers#64357suxiaogang223 wants to merge 26 commits into
suxiaogang223 wants to merge 26 commits into
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
suxiaogang223
force-pushed
the
codex/parquet-complex-type-contract
branch
from
a0f9236 to
43f3a10
Compare
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Add a single contract document for the final new parquet reader complex type implementation model. The new document defines reader boundaries, nested path semantics, shape/value separation, schema evolution, pruning safety rules, lazy materialization, and phased rollout. It also removes the older struct primitive predicate proposal document that is superseded by the complete contract. ### Release note None ### Check List (For Author) - Test: No need to test (documentation-only change) - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Clarify that the current nested parquet predicate pushdown is a transitional mapper-side extension for STRUCT and nested STRUCT primitive leaves only. Document that LIST/MAP/repeated predicate pushdown should wait for a ColumnPredicate or nested filter target refactor, and add an inline comment near the mapper extraction logic to prevent extending the transitional path beyond struct semantics. ### Release note None ### Check List (For Author) - Test: No need to test (documentation and comment-only change) - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Refine the complex type contract so the transitional nested predicate pushdown model is STRUCT-only and DuckDB StructFilter-like. Remove LIST/MAP predicate pushdown and repeated pruning planning from the contract, while keeping LIST/MAP reader coverage as part of complex type reading. ### Release note None ### Check List (For Author) - Test: No need to test (documentation-only change) - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Clarify that unsupported file-layer pruning does not mean unsupported row-level filtering. Complex predicates that cannot produce pruning hints must still be read through predicate projection and evaluated by localized VExprContext, especially during lazy materialization predicate phase. ### Release note None ### Check List (For Author) - Test: No need to test (documentation-only change) - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Refocus the complex type contract on nested complex type reading and a unified Arrow-style Dremel shape/value abstraction. Clarify that row-level filtering, file-layer pruning, and lazy materialization are consumers of the nested shape model, and add shape builder and testing expectations. ### Release note None ### Check List (For Author) - Test: No need to test (documentation-only change) - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: Add Phase 0 safety coverage for the new parquet reader complex type contract. The tests lock down that nested file-layer pruning remains struct-only during the transition, while list/map row-level predicates still force predicate projection and keep filter-only children out of final output mapping. ### Release note None ### Check List (For Author) - Test: No need to test (per request; verification will be run at a later implementation stage) - Behavior changed: No - Does this need documentation: No
Issue Number: close #xxx Related PR: #xxx Problem Summary: Nullable STRUCT reading previously depended on scalar child level streams to construct parent null shape. When the projected children were all complex, such as LIST/MAP children or nested STRUCT children that only contain complex descendants, the reader could not derive the STRUCT validity while preserving the top-level complex column layout. This change adds a transitional nested shape channel so complex readers can materialize values and expose ancestor null shapes from the same Dremel stream, validates sibling shapes, and covers nullable struct shape sources from complex descendants. None - Test: No need to test (not run per request; verification deferred) - Behavior changed: Yes (nullable STRUCT with only complex projected children can derive parent shape) - Does this need documentation: No
### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: The parquet nested shape reader channel collected parent shape null maps in a vector of NullMap values. StructColumnReader later attempted to copy-assign one NullMap into a local variable before applying the parent shape. NullMap is backed by PODArray and its copy assignment operator is deleted, so the BE UT build failed before running NewParquetReaderTest. This change uses a const reference to the selected parent shape map instead of copying it. ### Release note None ### Check List (For Author) - Test: Remote BE UT failed at compile before this fix; rerun pending. - Behavior changed: No - Does this need documentation: No
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: New parquet complex readers handled nested LIST/MAP combinations with type-specific state machines, so combinations such as array<map<...>> and map<K,map<...>> were rejected or could not be materialized recursively. This change adds a recursive nested batch/build channel backed by Doris schema level information, lets LIST and MAP readers delegate to child readers instead of enumerating combinations, and adds unit coverage for LIST of MAP and MAP of MAP.
Support additional nested Parquet complex type combinations in the new reader.
- Test: Unit Test
- Attempted ./run-be-ut.sh --run --filter=ParquetColumnReaderTest.ReadSupportedComplexTypes -j 4 locally, but macOS toolchain configuration failed before compilation with ld: library 'c++' not found. Remote BE UT will be run on fedora.
- Behavior changed: Yes. New parquet reader can construct recursive LIST/MAP complex type combinations.
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: The new Parquet map-of-map unit test data only appended four top-level rows while the test fixture requires five rows. This fixes the test builder by appending the final empty map row so the generated Arrow table is valid.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- ParquetColumnReaderTest.* will be rerun on fedora.
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Recursive complex reader support changed top-level LIST and MAP reads to always use the nested batch build path. Existing scalar, struct, and list/map combinations already had overflow-aware read_internal paths, and bypassing them regressed chunk-boundary, skip, selected read, and projected complex column cases. Keep the legacy top-level path for combinations it already supports, while using the new recursive build path only for recursive-only LIST<MAP> and MAP<MAP> shapes.
### Release note
None
### Check List (For Author)
- Test: Manual test
- Ran build-support/clang-format.sh and git diff --check locally. Fedora ParquetColumnReaderTest rerun will be triggered after pushing.
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Recursive LIST<MAP> materialization treated container-only map shape levels as nullable scalar value slots. This inserted an extra default value before real map scalar values and shifted nullable string payloads. Skip nested scalar levels below the scalar materialization slot, while still preserving real nullable scalar null slots.
### Release note
None
### Check List (For Author)
- Test: Manual test
- Ran build-support/clang-format.sh and git diff --check locally. Fedora ParquetColumnReaderTest rerun will be triggered after pushing.
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Recursive nested MAP materialization built keys from MAP entry levels but built scalar values independently from the scalar level stream. In LIST<MAP> and nested MAP shapes, scalar value streams can contain shape-only levels before the next real map entry, shifting nullable scalar values. Track the MAP entry level indices while assembling offsets and append scalar MAP values from those same indices.
### Release note
None
### Check List (For Author)
- Test: Manual test
- Ran build-support/clang-format.sh and git diff --check locally. Fedora ParquetColumnReaderTest rerun will be triggered after pushing.
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Nested MAP key and scalar value level streams can diverge around value-only shape slots such as null nested containers. Appending scalar values by the key level index can therefore consume a shape/null slot before the real entry value. While appending nested MAP scalar values, scan the value stream and match the key entry repetition level, skipping non-entry value shape slots.
### Release note
None
### Check List (For Author)
- Test: Manual test
- Ran build-support/clang-format.sh and git diff --check locally. Fedora ParquetColumnReaderTest rerun will be triggered after pushing.
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Matching only MAP entry levels still allowed scalar value streams to consume value-only shape slots around null nested containers. Track all processed MAP key shape levels and consume the scalar value stream for each shape event, appending a value only when the key shape represents a real MAP entry.
### Release note
None
### Check List (For Author)
- Test: Manual test
- Ran build-support/clang-format.sh and git diff --check locally. Fedora ParquetColumnReaderTest rerun will be triggered after pushing.
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: Recursive nested MAP build aligned scalar value levels to key levels by scanning for the next equal repetition level. Under LIST<MAP<...>>, outer list shape slots can share a repetition level with later map entries, so the scalar value stream could shift and materialize the wrong string value. This change treats MAP key and scalar value as fields of the same repeated entry struct: their definition levels may differ, but their shape slot index and repetition level must align. ### Release note None ### Check List (For Author) - Test: Pending remote BE UT rerun for ParquetColumnReaderTest.ReadSupportedComplexTypes. - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: Nested MAP build still misaligned scalar values for LIST<MAP<...>> when the parent tried to match value slots against key slots. The scalar value stream can include its own shape slots around nullable or empty containers, so parent-side slot matching is ambiguous. This change follows the recursive nested reader contract: MAP builds entry shape and keys, then asks the value reader to build exactly the number of materialized entry values from its own def/rep stream. ### Release note None ### Check List (For Author) - Test: Pending remote BE UT rerun for ParquetColumnReaderTest.ReadSupportedComplexTypes. - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: Recursive nested MAP build let scalar value readers materialize the first N value-shaped slots directly. For LIST<MAP<...>>, shape-only slots from empty or null MAP elements can have the same nullable scalar definition level as a real null value, which shifts later scalar values. This change keeps MAP entry ownership in the key stream and consumes scalar value shape slots in lockstep with key shape slots, only appending a value when the key slot represents a real MAP entry. ### Release note None ### Check List (For Author) - Test: Pending remote BE UT for ParquetColumnReaderTest.ReadSupportedComplexTypes. - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: Recursive nested scalar reads passed the max definition level as the value slot threshold. For nullable scalar leaves, the parquet record reader materializes null placeholders in the payload column, so value indices must advance for the nullable slot level as well as non-null values. Otherwise LIST<MAP<..., nullable scalar>> shifts later values after the first null. This change makes nested scalar batch loading use the materialized scalar slot definition level and makes column build consume the same threshold. ### Release note None ### Check List (For Author) - Test: Pending remote BE UT for ParquetColumnReaderTest.ReadSupportedComplexTypes. - Behavior changed: No - Does this need documentation: No
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Recursive parquet complex-type reads could shift scalar children after a null STRUCT element or MAP value. The reader inserted a default child slot for the null STRUCT parent, but the nested leaf value index stream did not account for Arrow RecordReader placeholders emitted for null ancestors. LIST<STRUCT<...>> and MAP<..., STRUCT<...>> could therefore materialize the next real scalar value one slot late. This change keeps recursive complex readers on the nested load/build path, tracks STRUCT parent level positions for null parent alignment, and makes nested leaf value index generation advance across null-ancestor placeholders while only exposing materialized value slots.
None
- Test: Unit Test
- On fedora: ./run-be-ut.sh -j 8 --run --filter="ParquetColumnReaderTest.ReadSupportedComplexTypes"
- On fedora: ./run-be-ut.sh -j 8 --run --filter="ParquetColumnReaderTest.*"
- On fedora: ./run-be-ut.sh -j 8 --run --filter="NewParquetReaderTest.*"
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: The new parquet complex type reader now has a recursive shape/value build path for LIST, MAP, and STRUCT. Keeping the old per-type read_internal state machines left two implementations for the same nested type semantics and made future fixes risky. This change removes the legacy complex reader paths and old overflow/helper abstractions, routes top-level and nested complex reads through load_nested_batch/build_nested_column, and keeps STRUCT ancestor shape exposure on the recursive level stream.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- Remote fedora: ./run-be-ut.sh -j 8 --run --filter="ParquetColumnReaderTest.*"
- Remote fedora: ./run-be-ut.sh -j 8 --run --filter="NewParquetReaderTest.*"
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The recursive parquet complex reader handles nested LIST, MAP, and STRUCT combinations through one shape/value path. Existing tests covered common shallow combinations, but did not stress deeper nesting with null and empty shapes at multiple levels. This change adds deep LIST/STRUCT/MAP/LIST and MAP/LIST/MAP fixtures, then validates normal reads, skipped reads, selected reads, and chunked reads across those shapes.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- Remote fedora: ./run-be-ut.sh -j 8 --run --filter="ParquetColumnReaderTest.*"
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: New parquet reader complex projection needs to keep the physical file layout separate from table output layout when nested schema evolution is involved. Without this, nested complex columns with renamed, reordered, missing, or predicate-only struct children can build file reader types in table order and then read the wrong child ordinal from the file-local block. This change records projected file children in file schema order, sorts nested scan projections by file-local id, reconstructs table output columns from the projected file layout, and keeps nested STRUCT predicate pushdown working for predicate-only siblings while continuing to reject LIST/MAP predicate pushdown by contract.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- ./run-be-ut.sh -j 8 --run --filter="TableReaderTest.*:TableColumnMapper*:LocalColumnIndexTest.*"
- ./run-be-ut.sh -j 8 --run --filter="ParquetColumnReaderTest.*:NewParquetReaderTest.*"
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: New parquet reader nested pruning was still represented by a raw file_child_id_path beside primitive ColumnPredicate objects. That made the STRUCT-only contract implicit and left file-layer pruning code tied to a transitional path vector. This change introduces a struct-only FileNestedPredicateTarget, makes TableColumnMapper emit it for nested STRUCT primitive leaf predicates, keeps ColumnPredicate as a primitive pruning predicate, and makes Parquet reader/statistics pruning consume the target through compatibility accessors while old direct filter construction still works during migration.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- build-support/clang-format.sh be/src/format_v2/file_reader.h be/src/format_v2/file_reader.cpp be/src/format_v2/column_mapper.cpp be/src/format_v2/parquet/parquet_reader.cpp be/src/format_v2/parquet/parquet_statistics.cpp be/test/format_v2/parquet/parquet_reader_test.cpp
- git diff --check
- ./run-be-ut.sh -j 8 --run --filter="TableReaderTest.*:TableColumnMapper*:LocalColumnIndexTest.*:ParquetColumnReaderTest.*:NewParquetReaderTest.*"
- Behavior changed: No
- Does this need documentation: Yes
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: New parquet reader nested file-layer pruning only recognized comparison and IN predicates on STRUCT leaf paths. This left nested IS NULL / IS NOT NULL predicates and safe widening cast wrappers as row-level filters only, even though Parquet leaf statistics can prune those cases safely for STRUCT / nested STRUCT primitive leaves. This change adds mapper extraction for nested null predicates and conservative order-preserving cast stripping for pruning targets while keeping row-level filter evaluation unchanged.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- Added TableColumnMapperTest coverage for nested IS NULL / IS NOT NULL, safe cast pruning, and unsafe cast rejection.
- Attempted ./run-be-ut.sh -j 8 --run --filter="TableColumnMapperTest.*" locally, but CMake failed before compiling Doris because the local macOS clang linker could not find libc++.
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: The temporary new parquet reader complex type contract has been used to drive the current scoped implementation. Phase 6 decoder-level lazy materialization is explicitly deferred, and nested runtime filter pushdown remains outside the current transition scope. The remaining implemented contract is now covered by code and focused tests, so the planning document is removed to avoid stale guidance. ### Release note None ### Check List (For Author) - Test: No need to test (documentation cleanup only) - Behavior changed: No - Does this need documentation: No
suxiaogang223
force-pushed
the
codex/parquet-complex-type-contract
branch
from
11a0f4f to
b57b439
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The new parquet reader needs one complete path for nested complex types instead of separate legacy top-level LIST/MAP handling. This PR defines the nested shape/value contract, keeps nested predicate pushdown scoped to STRUCT/nested STRUCT, adds recursive reader composition for LIST, MAP, STRUCT, and scalar leaves, removes the old complex reader fallback paths, and adds unit coverage for deep nested shapes with null and empty levels.
Release note
None
Check List (For Author)