Skip to content

[lake/tiering] Support reporting watermark to Paimon snapshot#3420

Open
Shawn-Hx wants to merge 1 commit into
apache:mainfrom
Shawn-Hx:FLUSS-3419
Open

[lake/tiering] Support reporting watermark to Paimon snapshot#3420
Shawn-Hx wants to merge 1 commit into
apache:mainfrom
Shawn-Hx:FLUSS-3419

Conversation

@Shawn-Hx
Copy link
Copy Markdown

@Shawn-Hx Shawn-Hx commented Jun 2, 2026

Purpose

Linked issue: close #3419

This PR supports reporting watermarks to Paimon snapshots during lake tiering.

Brief change log

  • Introduced watermark extraction for Fluss rows in the lake tiering pipeline.
  • Added watermark propagation through lake write results and tiering commit.
  • Aggregated watermarks across tiered buckets before committing to the lake.
  • Reported the aggregated watermark to Paimon snapshots when committing.
  • Updated serializers and tests to support nullable and negative watermark values.
  • Added coverage for watermark extraction, commit aggregation, and Paimon snapshot watermark reporting.

Tests

Add tests in:

  • PaimonWriteResultSerializerTest
  • PaimonTieringTest
  • TieringCommitOperatorTest
  • SimpleWatermarkExtractorTest

API and Format

This change extends the lake tiering API to carry watermark information through write results and committers.

Documentation

No user-facing documentation needs to be added.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[tiering] Support reporting watermark to Paimon snapshot

1 participant