[EPIC] Sort Pushdown: remaining improvements

[zhuqi-lucas]

Follow-up to #23036, which tracked the broader Sort Pushdown effort across DataFusion v52 → v55. The bulk of that epic is now complete — the optimizer-rule framework and Exact / Inexact / Unsupported classifier, reverse iteration, statistics-based file/RG reorder, multi-partition BufferExec, and the runtime RG-level early stop (#22450, merged) have all landed. This issue tracks the remaining open follow-ups so #23036 can be closed as substantially done.

See #23036 for the full architecture, phase history, and benchmark results.

Remaining open items

Statistics-based reorder follow-ups

• Follow-ups for stats-based RG / file reorder (#21956): multi-column, function-wrapped, compound exprs #22198 — Follow-ups for stats-based RG / file reorder: multi-column, function-wrapped sort expressions, compound ORDER BY
• Bug: EnforceDistribution optimizer loses fetch (LIMIT) from CoalescePartitionsExec and SortPreservingMergeExec #21169 — Bug: EnforceDistribution loses fetch (LIMIT) from CoalescePartitionsExec / SortPreservingMergeExec
• Add more ExecutionPlan to support sort pushdown #19394 — Add more ExecutionPlan nodes to support sort pushdown

Multi-partition / BufferExec follow-ups

• Parallel merge in SortPreservingMergeExec after sort elimination #21381 — Parallel merge in SortPreservingMergeExec after sort elimination
• Dynamic BufferExec sizing: row limit + memory cap for sort pushdown #21440 — Dynamic BufferExec sizing: row limit + memory cap
• refactor: make DynamicFilterPhysicalExpr.fetch a shared reference to SortExec.fetch #21780 — Refactor: make DynamicFilterPhysicalExpr.fetch a shared reference to SortExec.fetch

Runtime RG-level early stop follow-ups (Phase 4 landed via #22450)

• [Follow-up #22450] Per-RG fully_matched RowFilter skip · needs arrow-rs#10158 (peek_next_row_group) #23067 — Per-RG fully_matched RowFilter skip on top of feat(parquet): intra-file early stopping via statistics + dynamic filters #22450 (PR perf(parquet): skip RowFilter on statically fully-matched row groups #23696, in review)
• PhysicalExprSimplifier folds DynamicFilterPhysicalExpr wrapper, breaking live link to TopK updates #22949 — Bug: PhysicalExprSimplifier folds DynamicFilterPhysicalExpr wrapper, breaking the live link to TopK updates

Future — Exact reverse path (blocked on arrow-rs page-level reverse)

• Support reverse page for sort pushdown phase 1 #19486 — Support reverse page for sort pushdown (phase 1 follow-up)
• Follow up improvement for reverse parquet when row group size is huge, so it may cause OOM. #19048 — Reverse parquet OOM mitigation for huge row groups
• Cross-repo: apache/arrow-rs#9937 — POC ReverseSerializedPageReader (unlocks strictly-DESC output → Exact sort pushdown)