feat(pruning): prune containers through CASE predicates - #24238
Draft
adriangb wants to merge 2 commits into
Draft
Conversation
`build_predicate_expression` had no arm for `CaseExpr`, so any predicate
whose top level (or a conjunct of it) is a `CASE` was handed to the
unhandled hook and rewritten to `true`, contributing nothing to pruning.
This shows up for dynamic filters pushed down from hash partitioned
joins: the filter arrives at the scan wrapped in a partition switch, one
arm per partition, e.g.
CASE hash(l_orderkey) % 4
WHEN 0 THEN l_orderkey >= 165 AND l_orderkey <= 5999783 AND ...
WHEN 1 THEN ...
ELSE false
END
A container (row group, file, ...) may hold rows from any partition, so
the container level predicate is the disjunction of the arms' `THEN`
predicates. The `WHEN` values are not expressible in terms of container
statistics and are dropped, which only weakens the predicate and is
therefore sound.
Arms that can never be `true` (`false` or `NULL` literals, and the
implicit `NULL` of a missing `ELSE`) drop out of the disjunction; an arm
that can not be rewritten becomes `true` via the unhandled hook and so
makes the whole `CASE` `true` (no pruning), never `false`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Covers both `CASE` shapes (with and without a base expression), a missing/NULL `ELSE`, `false` arms dropping out of the disjunction, an all-`false` `CASE` pruning everything, an unhandled arm degrading to `true`, and an end to end prune with the partitioned-ranges shape a dynamic filter from a hash partitioned join produces. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Author
|
run benchmarks |
Contributor
Author
|
show benchmark queue |
|
Hi @adriangb, you asked to view the benchmark queue (#24238 (comment)).
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing prototype/pruning-case-or (db3fd52) to 33ad1cc (merge-base) diff Run configurationrun benchmark tpchResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing prototype/pruning-case-or (db3fd52) to 33ad1cc (merge-base) diff Run configurationrun benchmark clickbench_partitionedResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing prototype/pruning-case-or (db3fd52) to 33ad1cc (merge-base) diff Run configurationrun benchmark tpcdsResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing prototype/pruning-case-or (db3fd52) to 33ad1cc (merge-base) diff Run configurationrun benchmark tpchCPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing prototype/pruning-case-or (db3fd52) to 33ad1cc (merge-base) diff Run configurationrun benchmark tpcdsCPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing prototype/pruning-case-or (db3fd52) to 33ad1cc (merge-base) diff Run configurationrun benchmark clickbench_partitionedCPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24238 +/- ##
========================================
Coverage 80.99% 80.99%
========================================
Files 1106 1106
Lines 383352 383667 +315
Branches 383352 383667 +315
========================================
+ Hits 310488 310752 +264
- Misses 54544 54580 +36
- Partials 18320 18335 +15 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Related: #20195 (dynamic filtering on partitioned data from a file source), #24095 (A/B benchmark for dynamic filtering on range-partitioned hash joins).
Rationale for this change
datafusion/pruninghas no handling forCaseExpr. When a predicate is (or contains) aCASE,build_predicate_expressionfalls through toConstantUnhandledPredicateHook, which rewrites it tolit(true)— i.e. the predicate contributes nothing to container pruning.That matters because dynamic filters pushed down from a hash-partitioned join arrive at the scan wrapped in a partition switch, one arm per partition. This is a real plan from TPC-H q18 at
target_partitions=12(dfbench --debug,ordersscan):On
mainthis whole thing is invisible to pruning: theordersscan has nopruning_predicateat all.What changes are included in this PR?
A
CaseExprarm inbuild_predicate_expression(datafusion/pruning/src/pruning_predicate.rs) that rewrites aCASEused as a predicate into the disjunction of its arms:The
WHENvalues are deliberately dropped. A container (row group, file, ...) can only be pruned if no row in it can pass, and a container may hold rows from any partition, so the arms must be OR'd. Dropping theWHENs only weakens the predicate — container pruning is inherently a relaxation, so this is sound.Details:
ELSEis an implicitNULL, which never passes a filter, so it contributes nothing (it must not becometrue).false(empty partitions are emitted aslit(false)) drops out.truevia the unhandled hook, making the whole disjunctiontrue— no pruning, never a wrong prune.falseand all containers are pruned.Are these changes tested?
Yes — 7 new unit tests in
datafusion/pruning/src/pruning_predicate.rscovering bothCASEshapes (with and without a base expression), a missing/NULLELSE,falsearms dropping out, an all-falseCASEpruning everything, an unhandled arm degrading totrue, and an end-to-end prune with the partitioned-ranges shape a hash-partitioned dynamic filter actually produces.cargo test -p datafusion-pruning --lib→ 96 passed. Fullsqllogictestsuite → 502/502 files, no.sltchanges needed.cargo fmt --all --checkand clippy with the CI feature set are clean.Are there any user-facing changes?
No API change. The only user-visible effect is that scans behind a
CASEpredicate may now prune containers. Please read the measurements below before judging this worth merging — the honest summary is that the win is narrow and there are costs.Measurements
TPC-H SF=1, parquet, 12 physical cores,
target_partitions=12. Both binaries built from the same worktree and the same target dir (build change → copy binary aside → revert the file → rebuild → copy aside; the two binaries were asserted to differ). 12 rounds ×--iterations 5, median per round, ordering counterbalanced within rounds (change-first on odd rounds, baseline-first on even). q1 and q6 are controls: they have no joins and no dynamic filters, so this change cannot affect them, and their measured delta is the noise floor.Positive % = the change is slower.
The control floor is ±3.6%. Note that control q1 shows a sign-consistent +2.26% across both orderings for a change that provably cannot affect it — so there is a systematic build/code-layout bias of about +2% on top of run-to-run variance. Nothing in the table below ~3.6% should be read as an effect of this change, and the earlier numbers I had on this branch (a claimed +1–5% spread of regressions on q7/q9/q12/q19/q3/q5/q21) do not survive this protocol.
What survives:
Mechanical evidence for q18 (this is the part timing noise cannot fake)
dfbench --debug, TPC-H q18, theordersscan:pruning_predicatebytes_scannedA row group is pruned and 6.5 MB less is read. That is the whole win.
I also diffed
bytes_scannedand row-group counts for every scan in all 13 queries above:So for everything except q18 this change is pure cost with zero benefit, and q20's +5.35% is that cost showing up above the noise.
Why the effect is nil for most hash-partitioned joins
The q18
ordersscan carries two dynamic filters. The one ono_custkeyis useless, and it shows exactly why:Hash partitioning scatters keys, so on a dense key column every partition's min/max converges on the full domain, and the OR of 12 near-full ranges is the full domain. This is not a weakness of the rewrite — no min/max-based rewrite can do better on this input.
q18 wins only because its other dynamic filter comes from a semi-join against
... group by l_orderkey having sum(l_quantity) > 300, whose build side is tiny and sparse. There each arm is genuinely narrow and their union still excludes a row group.The realistic conclusion: this helps selective semi-joins / small build sides, and does nothing for hash-partitioned equi-joins on dense keys.
Known follow-up (the lever, if the cost needs fixing)
The generated predicate repeats the null-count guard twice per arm:
That is ~4 redundant subterms per arm, and the arm count scales with
target_partitions— so the predicate this builds grows linearly with core count while its selectivity does not. That redundancy, not the OR structure itself, is the lever if the q20-style regression needs to be addressed; deduplicating the guards (or hoisting them out of the disjunction) should remove most of the added evaluation cost without changing what gets pruned.Filed as a draft because the cost/benefit above is genuinely marginal and I would like input on whether it is worth carrying.