refactor(hash-aggr): treat soft limit as hint

[2010YOUY01]

Which issue does this PR close?

Part of #22710
Depends on #24259

Only the last commit belongs to this PR. I’ll rebase onto main after the dependent PR is merged.

Rationale for this change

For the background of 'soft limit optimization' in aggregation, see:

datafusion/datafusion/physical-plan/src/aggregates/hash_stream.rs

Line 81 in 8cb7c84

/// # Optimization: DISTINCT LIMIT Soft Limit

'soft' in the term means it's only a hint for the operator, it can try to produce outputs less than the 'soft limit' for efficiency, if it ignores this soft limit, there will be no correctness issue, there will be a downstream LimitExec for the final processing.

The original implementation only targets partial/final mode for this optimization: #8038, but somehow this soft limit hint can be pushed to other aggregation modes (like single aggregation)

For simplicity, this PR allows soft limit in all aggregation modes, since

• no correctness issue
• Should have no performance regression, as previously this optimization is only applicable to partial/final aggregation.
• [] TODO for myself: double check the benchmark

Later we can try to apply this optimization more widely, or trying to implement the related optimizer rule more defensively.

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

[codecov-commenter]

Codecov Report

❌ Patch coverage is 86.91099% with 75 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.00%. Comparing base (634f0b3) to head (0fc3e59).
⚠️ Report is 42 commits behind head on main.

Files with missing lines	Patch %	Lines
...sical-plan/src/aggregates/ordered_single_stream.rs	86.74%	49 Missing and 10 partials ⚠️
datafusion/physical-plan/src/aggregates/mod.rs	84.81%	4 Missing and 8 partials ⚠️
.../aggregates/aggregate_hash_table/common_ordered.rs	60.00%	0 Missing and 2 partials ⚠️
...gates/aggregate_hash_table/ordered_single_table.rs	94.44%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24260      +/-   ##
==========================================
- Coverage   81.06%   81.00%   -0.06%     
==========================================
  Files        1107     1108       +1     
  Lines      382191   384589    +2398     
  Branches   382191   384589    +2398     
==========================================
+ Hits       309805   311531    +1726     
- Misses      54083    54700     +617     
- Partials    18303    18358      +55     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.