Skip to content

Core: Don't prune equality delete manifest entries by non-key column stats - #18338

Open
findinpath wants to merge 1 commit into
apache:mainfrom
findinpath:findinpath/equality-deletes
Open

findinpath wants to merge 1 commit into
apache:mainfrom
findinpath:findinpath/equality-deletes

Conversation

@findinpath

@findinpath findinpath commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

An equality delete's match condition depends only on its equality_ids
columns. Per spec, the file may legitimately carry additional columns of
the deleted row, but their stats describe values that play no part in
the match condition. InclusiveMetricsEvaluator and StrictMetricsEvaluator
evaluated the scan's row filter against all of a delete file's column
stats, so a predicate on a non-key column could incorrectly prune a
delete manifest entry (inclusive) or cause it to be dropped from the
manifest entirely (strict), leaving a row that should have been deleted
in the query result. V4ManifestReader's stats-based filtering has the
same issue for v4 manifests.

Narrow stats to a file's equality_ids columns before metrics evaluation
for equality delete files.

Additional context

https://iceberg.apache.org/spec/#equality-delete-files

Equality delete files identify deleted rows in a collection of data files by one or more column values, and may optionally contain additional columns of the deleted row.

Is there a specific writer that does this (spark? flink? something else?)

Oracle GoldenGate

Issue found through trinodb/trino trinodb/trino#31399

@github-actions github-actions Bot added the core label Oct 1, 2026
@findinpath
findinpath force-pushed the findinpath/equality-deletes branch 2 times, most recently from 2ec8a70 to adcf116 Compare October 1, 2026 11:53
Comment thread core/src/main/java/org/apache/iceberg/ManifestReader.java Outdated
@findinpath
findinpath force-pushed the findinpath/equality-deletes branch 2 times, most recently from f2cc7ae to efa28c2 Compare October 1, 2026 13:29
Comment thread core/src/main/java/org/apache/iceberg/ManifestReader.java Outdated
Comment thread core/src/main/java/org/apache/iceberg/ManifestReader.java Outdated
Comment thread core/src/main/java/org/apache/iceberg/ManifestReader.java Outdated
Comment thread core/src/test/java/org/apache/iceberg/DeleteFileIndexTestBase.java Outdated
@pvary

pvary commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

I'm not entirely sure how much effort do we want to put into fixing this, as there is no current use-case for this. Especially considering that we would like to get rid of the equality deletes in the long run.
If we fix this then we should think about fixing this on the commit path as well (newDelete().deleteFromRowFilter), and this is where I'm really getting unconvinced.

@findinpath
findinpath force-pushed the findinpath/equality-deletes branch from efa28c2 to 0f942a8 Compare October 7, 2026 04:36
@github-actions github-actions Bot added the API label Oct 7, 2026
@findinpath

findinpath commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor Author

I'm not entirely sure how much effort do we want to put into fixing this

@pvary This represents a silent correctness issue.
We definitely should want to put effort into fixing such issues. 🙏

Comment thread api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java Outdated
@pvary

pvary commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

The more I think about this, we should read stats for non-eq columns.
Maybe even a spec change/clarification is warranted.

…stats

An equality delete's match condition depends only on its equality_ids
columns. Per spec, the file may legitimately carry additional columns of
the deleted row, but their stats describe values that play no part in
the match condition. InclusiveMetricsEvaluator and StrictMetricsEvaluator
evaluated the scan's row filter against all of a delete file's column
stats, so a predicate on a non-key column could incorrectly prune a
delete manifest entry (inclusive) or cause it to be dropped from the
manifest entirely (strict), leaving a row that should have been deleted
in the query result. V4ManifestReader's stats-based filtering has the
same issue for v4 manifests.

Narrow stats to a file's equality_ids columns before metrics evaluation
for equality delete files.
@findinpath
findinpath force-pushed the findinpath/equality-deletes branch from f415394 to 3e2b47c Compare October 9, 2026 12:02
@findinpath
findinpath requested a review from pvary October 9, 2026 12:03

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants