[fix](be) Avoid materializing complex columns for external count#65560
Closed
xylaaaaa wants to merge 1 commit into
Closed
[fix](be) Avoid materializing complex columns for external count#65560xylaaaaa wants to merge 1 commit into
xylaaaaa wants to merge 1 commit into
Conversation
xylaaaaa
requested review from
924060929,
Gabriel39,
englefly,
morrySnow,
starocean999 and
yiguolei
as code owners
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
xylaaaaa
force-pushed
the
codex/fix-v2-count-complex-memory-pr
branch
from
7a26fef to
994746b
Compare
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29626 ms |
Contributor
TPC-DS: Total hot run time: 179358 ms |
Contributor
ClickBench: Total hot run time: 25.49 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
|
/review |
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: COUNT(nullable complex column) over Parquet or ORC file TVFs cannot use COUNT(*) row metadata because top-level NULL values must be excluded. With FileScannerV2, the query therefore decoded the full complex column through Arrow and then materialized Doris columns, retaining decoded binary chunks and causing excessive peak memory or intermittent OOM. This change introduces an explicit COUNT_NON_NULL operation and target slot from FE to BE. Parquet counts top-level shape from bounded definition/repetition-level batches without Arrow builders or Doris value columns, while ORC uses non-null stripe statistics. It also emits valid non-NULL synthetic target rows and releases file-block ownership before mutable output conversion to avoid unnecessary COW copies.
### Release note
Reduce memory usage for COUNT on nullable columns read from Parquet and ORC file TVFs with FileScannerV2.
### Check List (For Author)
- Test: Unit Test
- FE PhysicalStorageLayerAggregateTest: 4 tests passed
- BE targeted Parquet, ORC, and TableReader tests: 14 tests passed
- HDFS TVF regression assertion added but not run locally because this worktree has no Doris cluster or external HDFS test environment
- Behavior changed: Yes. Eligible nullable file-column counts use explicit COUNT_NON_NULL pushdown instead of materializing the full column.
- Does this need documentation: No
xylaaaaa
force-pushed
the
codex/fix-v2-count-complex-memory-pr
branch
from
994746b to
ef14917
Compare
Contributor
Author
|
run buildall |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
I found one correctness issue in the new COUNT_NON_NULL pushdown path. The PR head moved after the review snapshot, so I fetched and rechecked live head ef14917; the issue still exists at the submitted anchor.
Checkpoint summary:
- Correctness: COUNT_NON_NULL currently fails valid count(col) queries when the per-split mapping or file reader cannot serve the optimization, instead of falling back to a normal scan.
- Lifecycle/concurrency: runtime-filter, conjunct, delete, condition-cache, close, and finalize ownership paths were checked; no separate issue remained.
- Compatibility/config: TVF format gating, session-variable gates, thrift propagation, and SlotId-to-GlobalIndex mapping were checked.
- Tests: existing tests cover successful direct Parquet/ORC paths, but not fallback for missing/defaulted split columns or unsupported reader metadata/layouts. No tests were run in this review environment.
| FileAggregateResult file_result; | ||
| const auto status = _data_reader.reader->get_aggregate_result(file_request, &file_result); | ||
| if (status.is<ErrorCode::NOT_IMPLEMENTED_ERROR>()) { | ||
| if (_push_down_agg_type == TPushAggOp::type::COUNT_NON_NULL) { |
Contributor
There was a problem hiding this comment.
This makes an optional pushdown miss fail the query. Status::NotSupported is NOT_IMPLEMENTED_ERROR, and the Parquet/ORC aggregate readers still return it for cases they cannot answer from metadata or levels, such as unsupported Parquet nested layouts or missing ORC stripe/column statistics. The per-split mapper can also reject the target when a TVF glob contains a later file where the column is missing/defaulted; the normal scan path can synthesize those values. Since the scan still has the real count column projected, COUNT_NON_NULL needs the same fallback behavior as COUNT/MINMAX: leave pushed_down=false and scan rows normally instead of returning an error for valid count(col) queries.
Contributor
|
Codex automated review failed and did not complete. Error: Codex completed, but no new pull request review was submitted for the current head SHA. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 29139 ms |
Contributor
TPC-DS: Total hot run time: 177574 ms |
Contributor
ClickBench: Total hot run time: 25.54 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: COUNT(nullable complex column) over Parquet or ORC file TVFs cannot use COUNT(*) row metadata because top-level NULL values must be excluded. With FileScannerV2, the query therefore decoded the full complex column through Arrow and then materialized Doris columns, retaining decoded binary chunks and causing excessive peak memory or intermittent OOM. This change introduces an explicit COUNT_NON_NULL operation and target slot from FE to BE. Parquet counts top-level shape from bounded definition/repetition-level batches without Arrow builders or Doris value columns, while ORC uses non-null stripe statistics. It also emits valid non-NULL synthetic target rows and releases file-block ownership before mutable output conversion to avoid unnecessary COW copies.
Release note
Reduce memory usage for COUNT on nullable columns read from Parquet and ORC file TVFs with FileScannerV2.
Check List (For Author)