Skip to content

fix: preserve window inputs during TopK aggregation - #25066

Draft
discord9 wants to merge 1 commit into
apache:mainfrom
discord9:fix/topk-aggregation-window-boundary
Draft

fix: preserve window inputs during TopK aggregation#25066
discord9 wants to merge 1 commit into
apache:mainfrom
discord9:fix/topk-aggregation-window-boundary

Conversation

@discord9

@discord9 discord9 commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

No linked issue. This is an independently reproduced correctness fix.

Rationale for this change

TopK aggregation can discard groups that are needed to compute window values. With two groups g = 1, 2, the default SQL pipeline returns (1, 1) instead of (1, 2) for:

SELECT g, COUNT(*) OVER () AS n
FROM t
GROUP BY g
ORDER BY g ASC
LIMIT 1;

The window preserves the number of input rows, but its output values can depend on groups that an ancestor TopK would discard. Cardinality preservation alone does not make this propagation valid.

What changes are included in this PR?

Stop this ancestor TopK's aggregate-limit propagation at WindowAggExec and BoundedWindowAggExec, using the existing traversal flag. Keep aggregate/projection handling and independent optimization of eligible sorts within window inputs unchanged.

This change is independent of #23800 and #25065 and does not modify their limit-pushdown or distribution-enforcement paths.

What is the testing strategy for this PR?

Added SQLLogicTest coverage in aggregates_topk.slt for:

  • COUNT(*) OVER (), with TopK aggregation enabled and disabled;
  • a bounded preceding-row window ordered descending under an ascending outer TopK, ensuring the fetched outer sort remains present;
  • a no-outer-limit control and generated EXPLAIN plans for both window implementations.

Both window guards were independently removed during regression verification: each corresponding enabled query changed from (1, 2) to (1, 1). Both guards were restored, and the full test file passed.

Verified locally:

  • cargo check -p datafusion-physical-optimizer
  • cargo test -p datafusion-sqllogictest --test sqllogictests -- aggregates_topk
  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets --all-features -- -D warnings

The full workspace runtime test suite was not run.

Are there any user-facing changes?

Correct window results for affected grouped TopK queries. No public API or configuration changes. This conservatively prevents an invalid optimization across windows rather than changing their execution semantics.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
@github-actions github-actions Bot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) labels Sep 8, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.74%. Comparing base (b39cb4a) to head (3559556).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25066      +/-   ##
==========================================
- Coverage   81.74%   81.74%   -0.01%     
==========================================
  Files        1128     1128              
  Lines      416644   416650       +6     
  Branches   416644   416650       +6     
==========================================
- Hits       340585   340581       -4     
- Misses      55999    56009      +10     
  Partials    20060    20060              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants