Skip to content

[core] Optimize tag reads during snapshot expiration - #9967

Merged
JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/optimize-tag-snapshot-expiration
Sep 19, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/optimize-tag-snapshot-expiration

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Purpose

Snapshot expiration previously built a full data-file index for every referenced tag. On large tables, concurrent tag reads retained memory proportional to the number of tags multiplied by all table files and could cause OOM.

This change:

  • plans deletion candidates in bounded expiration batches;
  • prunes tag manifests by candidate bucket ranges;
  • pushes partition and bucket filtering into Avro decoding before nested file metadata;
  • filters file names on a reusable projected entry before materializing ExpireFileEntry;
  • keeps only matching tag entries while preserving changelog-decoupled cleanup behavior.

The filename filter is streaming rather than storage-index based; manifests still need to be scanned within matching partitions and buckets.

Tests

  • Targeted manifest streaming/filter-pushdown and tag expiration regression tests
  • ExpireSnapshotsTest and FileDeletionTest
  • mvn -pl paimon-core -DskipTests spotless:check

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The candidate-scoped tag reads reduce retained state while preserving tag protection and changelog-decoupled cleanup behavior. Repeated scanning across batches is an acceptable trade-off for reducing memory pressure.

Validated with Java 8: 161 existing test cases across ManifestFileTest, ExpireSnapshotsTest, FileDeletionTest, ChangelogExpireTest, and TagTest, plus a local cross-batch tag-retention regression (162 passed in total). No blocking issues found.

@JingsongLi
JingsongLi merged commit 1d368b4 into apache:master Sep 19, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants