Skip to content

feat: Prune ListingTable file groups for range-key filters - #25819

Open
goutamadwant wants to merge 1 commit into
apache:mainfrom
goutamadwant:feat/range-static-pruning-25437
Open

goutamadwant wants to merge 1 commit into
apache:mainfrom
goutamadwant:feat/range-static-pruning-25437

Conversation

@goutamadwant

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

A ListingTable scan with declared Range partitioning currently retains every file group even when a range-key filter proves that some groups cannot contain matching rows. For example, range_key < 10 still planned all four groups for split points (10), (20), (30).

What changes are included in this PR?

  • Prune file groups that are provably disjoint from simple literal comparisons on a single integer range key.
  • Leave pruned groups empty at their original partition indexes so range-partitioned joins retain alignment.
  • Recompute scan statistics after pruning. Unsupported predicates, types, and uncertain bounds retain their file groups.

This does not yet compact partition counts or remove all downstream work for empty partitions.

What is the testing strategy for this PR?

  • SQLLogic plan tests verify range_key < 10 and equality at the 10 split point retain only the expected file group; a join test verifies the resulting rows.
  • Unit tests cover ascending and descending boundaries, reversed comparisons, compound predicates, NULL split points, and conservative fallbacks.
  • cargo fmt --all --check, cargo clippy --all-targets --all-features -- -D warnings, focused tests, and the extended workspace test suite passed locally.

Are there any user-facing changes?

Yes. Supported range-key filters can avoid reading file groups known not to match. There is no public API change.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) catalog Related to the catalog crate labels Sep 27, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.11765% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.52%. Comparing base (871058c) to head (d982ffd).

Files with missing lines Patch % Lines
datafusion/catalog-listing/src/range_pruning.rs 94.54% 9 Missing ⚠️
datafusion/catalog-listing/src/table.rs 90.90% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main   #25819    +/-   ##
========================================
  Coverage   82.51%   82.52%            
========================================
  Files        1141     1142     +1     
  Lines      439801   439984   +183     
  Branches   439801   439984   +183     
========================================
+ Hits       362916   363078   +162     
- Misses      54943    54959    +16     
- Partials    21942    21947     +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

catalog Related to the catalog crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants