[kv] Supports compacted row as change log #2108
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Pull request overview
This pull request adds support for using compacted row format as a change log in primary key tables. The change enables a new COMPACTED log format option alongside existing ARROW and INDEXED formats, providing a space-optimized row-oriented format that trades CPU for reduced storage.
Key changes:
- Introduces
COMPACTEDas a new log format option for primary key tables withCOMPACTEDkv format - Implements compacted row encoding/decoding infrastructure with variable-length integer encoding (VLQ)
- Refactors existing indexed log builder to extract common functionality into a shared abstract base class
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| fluss-server/src/main/java/org/apache/fluss/server/utils/TableDescriptorValidation.java | Updates validation logic to allow both ARROW and COMPACTED log formats when kv format is COMPACTED |
| fluss-server/src/main/java/org/apache/fluss/server/kv/wal/CompactedWalBuilder.java | New WAL builder implementation for COMPACTED log format with row encoding support |
| fluss-server/src/main/java/org/apache/fluss/server/kv/KvTablet.java | Integrates CompactedWalBuilder into KV tablet for handling COMPACTED log format |
| fluss-common/src/main/java/org/apache/fluss/record/CompactedLogRecord.java | New log record class implementing space-optimized encoding for CompactedRow format |
| fluss-common/src/main/java/org/apache/fluss/record/MemoryLogRecordsCompactedBuilder.java | Builder for creating memory log records in COMPACTED format |
| fluss-common/src/main/java/org/apache/fluss/record/AbstractRowMemoryLogRecordsBuilder.java | New abstract base class extracting common logic from row-based log record builders |
| fluss-common/src/main/java/org/apache/fluss/record/MemoryLogRecordsIndexedBuilder.java | Refactored to extend AbstractRowMemoryLogRecordsBuilder, reducing code duplication |
| fluss-common/src/main/java/org/apache/fluss/record/LogRecordReadContext.java | Adds support for creating read contexts for COMPACTED format with projection validation |
| fluss-common/src/main/java/org/apache/fluss/record/DefaultLogRecordBatch.java | Implements iterator for reading COMPACTED log records from batches |
| fluss-common/src/main/java/org/apache/fluss/metadata/LogFormat.java | Adds COMPACTED enum value and updates documentation |
| fluss-common/src/main/java/org/apache/fluss/config/ConfigOptions.java | Updates configuration documentation to include COMPACTED format |
| fluss-common/src/test/java/org/apache/fluss/record/MemoryLogRecordsCompactedBuilderTest.java | Comprehensive unit tests for CompactedBuilder including append, abort, and offset semantics |
| fluss-common/src/test/java/org/apache/fluss/record/CompactedLogRecordTest.java | Unit tests for CompactedLogRecord serialization/deserialization |
| fluss-client/src/test/java/org/apache/fluss/client/table/FlussTableITCase.java | Integration tests for COMPACTED log format including upsert, delete, and projection validation |
| fluss-client/src/test/java/org/apache/fluss/client/admin/FlussAdminITCase.java | Updates test assertion to match new validation error message |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| import static org.apache.fluss.record.LogRecordBatchFormat.LENGTH_LENGTH; | ||
|
|
||
| /** | ||
| * An immutable log record for @CompactedRow which can be directly persisted. The on-wire schema is |
There was a problem hiding this comment.
The JavaDoc reference should use {@link CompactedRow} instead of @CompactedRow for proper linking.
Suggested change
| * An immutable log record for @CompactedRow which can be directly persisted. The on-wire schema is | |
| * An immutable log record for {@link CompactedRow} which can be directly persisted. The on-wire schema is |
| if (currentRecordNumber > 0) { | ||
| outputView.writeInt(currentRecordNumber - 1); | ||
| } else { | ||
| // If there is no record, we write 0 for filed lastOffsetDelta, see the comments about |
There was a problem hiding this comment.
Typo: "filed" should be "field".
Suggested change
| // If there is no record, we write 0 for filed lastOffsetDelta, see the comments about | |
| // If there is no record, we write 0 for field lastOffsetDelta, see the comments about |
| } | ||
|
|
||
| @Override | ||
| public boolean equals(Object o) { |
There was a problem hiding this comment.
This 'equals()' method does not check argument type.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #2107
Brief change log
CP from #1605 , only keep the code of supporting compacted row as change log
Tests
API and Format
Documentation