Describe the bug
Queries using ORDER BY ... LIMIT can return incorrect rows when the physical plan contains a fetched SortPreservingMergeExec over an ordered Parquet scan.
For a plan shaped like:
SortPreservingMergeExec: [key@0 ASC], fetch=5
DataSourceExec: output_ordering=[key@0 ASC]
LimitPushdown pushes the fetch into the scan but does not recognize that the fetch on SortPreservingMergeExec is order-sensitive. It consequently updates the scan with:
limit=Some(5)
preserve_order=false
This overrides the scan's existing order-preservation requirement.
For Parquet scans, preserve_order=false enables limit-based row-group pruning. That optimization may discard an earlier partially matched row group in favor of a later fully matched row group, changing which rows are returned by the ordered limit.
To Reproduce
Consider a Parquet file sorted by key, scanned into multiple ordered partitions and combined by the fetched SortPreservingMergeExec above:
| Row group |
Values |
Predicate classification |
| RG0 |
0..99 |
Partially matches key >= 1 |
| RG1 |
100..199 |
Fully matches key >= 1 |
Run:
SELECT key
FROM t
WHERE key >= 1
ORDER BY key ASC
LIMIT 5;
Expected result:
The faulty plan can discard RG0 during limit-based pruning and return:
A minimal optimizer-level reproduction is:
- Construct an ordered Parquet
DataSourceExec.
- Wrap it in a
SortPreservingMergeExec with fetch=5.
- Run
LimitPushdown.
- Inspect the resulting
FileScanConfig.
On main at 66677feea, the resulting scan has limit=Some(5) but preserve_order=false.
Expected behavior
A fetch on SortPreservingMergeExec selects the leading rows according to its ordering. When that fetch is pushed into the underlying scan, LimitPushdown should propagate preserve_order=true.
The resulting FileScanConfig should contain:
limit=Some(5)
preserve_order=true
This prevents order-insensitive Parquet limit pruning from changing which rows are eligible for the ordered limit.
Additional context
The issue occurs because pushdown_limit_helper updates its global requirements when it encounters an operator with a fetch, but does not infer order sensitivity from a fetched SortPreservingMergeExec.
This is related to #24215, but is a distinct path. #24215 concerns order requirements lost when limit nodes are reconstructed or optimized again. This issue occurs in the standard single-pass optimizer flow after a fetched SortPreservingMergeExec has already been created.
Describe the bug
Queries using
ORDER BY ... LIMITcan return incorrect rows when the physical plan contains a fetchedSortPreservingMergeExecover an ordered Parquet scan.For a plan shaped like:
LimitPushdownpushes the fetch into the scan but does not recognize that the fetch onSortPreservingMergeExecis order-sensitive. It consequently updates the scan with:This overrides the scan's existing order-preservation requirement.
For Parquet scans,
preserve_order=falseenables limit-based row-group pruning. That optimization may discard an earlier partially matched row group in favor of a later fully matched row group, changing which rows are returned by the ordered limit.To Reproduce
Consider a Parquet file sorted by
key, scanned into multiple ordered partitions and combined by the fetchedSortPreservingMergeExecabove:0..99key >= 1100..199key >= 1Run:
Expected result:
The faulty plan can discard RG0 during limit-based pruning and return:
A minimal optimizer-level reproduction is:
DataSourceExec.SortPreservingMergeExecwithfetch=5.LimitPushdown.FileScanConfig.On
mainat66677feea, the resulting scan haslimit=Some(5)butpreserve_order=false.Expected behavior
A fetch on
SortPreservingMergeExecselects the leading rows according to its ordering. When that fetch is pushed into the underlying scan,LimitPushdownshould propagatepreserve_order=true.The resulting
FileScanConfigshould contain:This prevents order-insensitive Parquet limit pruning from changing which rows are eligible for the ordered limit.
Additional context
The issue occurs because
pushdown_limit_helperupdates its global requirements when it encounters an operator with a fetch, but does not infer order sensitivity from a fetchedSortPreservingMergeExec.This is related to #24215, but is a distinct path. #24215 concerns order requirements lost when limit nodes are reconstructed or optimized again. This issue occurs in the standard single-pass optimizer flow after a fetched
SortPreservingMergeExechas already been created.