feat: introduce snapshot checkpoints #27153
Merged
+3,715
−125
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
waynr
force-pushed
the
cherry-pick/main/parquet-snapshot-checkpointing
branch
from
5d7d610 to
5c46713
Compare
waynr
added a commit
that referenced
this pull request
Jan 26, 2026
This feature is cherry-picked from Enterprise. It is an optional feature (enabled by setting the `--checkpoint-duration <duration>` flag) that is intended to reduce overall server startup time under extremely high load combined with files regularly deleted via retention periods.
waynr
force-pushed
the
cherry-pick/main/parquet-snapshot-checkpointing
branch
from
5c46713 to
cade4d6
Compare
waynr
added a commit
that referenced
this pull request
Jan 26, 2026
This feature is cherry-picked from Enterprise. It is an optional feature (enabled by setting the `--checkpoint-duration <duration>` flag) that is intended to reduce overall server startup time under extremely high load combined with files regularly deleted via retention periods.
Contributor
|
Cherry-pick of: |
hiltontj
approved these changes
Jan 27, 2026
Contributor
hiltontj
left a comment
hiltontj
left a comment
There was a problem hiding this comment.
There are some changes that are slightly different than in the original PR, but I don't see issue with them.
I made a note on a couple, but do want to double check that the changes to CircleCI config were intentional.
Contributor
There was a problem hiding this comment.
There are some discrepancies in the diff for this file compared to https://github.com/influxdata/influxdb_pro/pull/2271 but from what I see it is for code that was added in Enterprise elsewhere.
waynr
force-pushed
the
cherry-pick/main/parquet-snapshot-checkpointing
branch
from
cade4d6 to
ac2b97c
Compare
* chore: add trace logs to help troubleshoot slow snapshot loading * test: add benchmarks for PersistedFiles.get_files * test: add benchmarks for persisted_files::add_persisted_snapshot * chore: use swap_remove instead of remove during snapshot merge
…#2030) * chore: pass compacted data snapshot markers to PersistedFiles constructor * chore: initialize CompactedData for ingester-only mode (drop CompactedData after obtaining snapshot markers in ingest-only mode) * chore: enable setting build profile for manual PR trigger
waynr
force-pushed
the
cherry-pick/main/parquet-snapshot-checkpointing
branch
from
ac2b97c to
08772d6
Compare
* feat: introduce PersistedSnapshotCheckpoint to parquet engine * chore: introduce PersistedSnapshotCheckpoint, SnapshotCheckpointPath, SnapshotCheckpointSequenceNumber * chore: introduce Persister methods for PersistedSnapshotCheckpoint * chore: add checkpoint_interval param to Persister::new * chore: add --checkpoint-interval flag to serve subcommand (disable by default) * feat: implement periodic snapshot checkpoint creation as part of snapshot creation * chore: make use of checkpoints during WriteBufferImpl initialization * chore: retain previous checkpoint in-memory for subsequent snapshot updates * test: improve checkpoint-related Persister coverage * chore: store FileIndex with cached snapshot checkpoint in Persister * fix: use insta redaction to avoid cross-test contamination with id values * chore: warn if checkpoints cannot be loaded during startup * test: add WriteBufferImpl startup integration tests with checkpoint edge cases * chore: register checkpoint-interval flag as non-sensitive * chore: build and persist checkpoints covering lookback period if they don't exist * chore: pass Vec<PersistedSnapshot> around as Arc<Vec<PersistedSnapshot>> to reduce peak memory usage
waynr
force-pushed
the
cherry-pick/main/parquet-snapshot-checkpointing
branch
from
08772d6 to
6467f9a
Compare
waynr
added a commit
that referenced
this pull request
Jan 27, 2026
This feature is cherry-picked from Enterprise. It is an optional feature (enabled by setting the `--checkpoint-duration <duration>` flag) that is intended to reduce overall server startup time under extremely high load combined with files regularly deleted via retention periods.
Contributor
Author
|
Note: this PR now also includes snapshot cleanup behaviors added in https://github.com/influxdata/influxdb_pro/pull/2174 |
waynr
added a commit
that referenced
this pull request
Jan 27, 2026
This feature is cherry-picked from Enterprise. It is an optional feature (enabled by setting the `--checkpoint-duration <duration>` flag) that is intended to reduce overall server startup time under extremely high load combined with files regularly deleted via retention periods. Cherry-picked PRs: * influxdata/influxdb_pro#2018 * influxdata/influxdb_pro#2030 * influxdata/influxdb_pro#2173 * influxdata/influxdb_pro#2174 Based on influxdata/influxdb_pro#2173
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This feature is cherry-picked from Enterprise. It is an optional feature (enabled by setting the
--checkpoint-duration <duration>flag) that is intended to reduce overall server startup time under extremely high load combined with files regularly deleted via retention periods.