Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pruning and WAL capability for SS store #66

Merged
merged 14 commits into from
Jun 26, 2024
Merged

Add pruning and WAL capability for SS store #66

merged 14 commits into from
Jun 26, 2024

Conversation

yzang2019
Copy link
Collaborator

@yzang2019 yzang2019 commented Jun 21, 2024

Describe your changes and provide context

This PR include several changes:

  1. We add a new config for SS store: DedicatedChangelog (Means that we should use a dedicated WAL file)
  2. We add a new interface for SS store: ApplyChangesetAsync (Used for supporting async writes)
  3. We make pruning part of SS and automatically start when we initialize the DB
  4. We add pruning logic for changelog as well
  5. We modify recovery logic to backward search the starting position one by one
  6. If DedicatedChangelog = true, we will perform a recovery first during initialization

Testing performed to validate your change

Added unit test

ss/pebbledb/db.go Fixed Show fixed Hide fixed
}
stream.nextOffset = startIndex + 1
if config.KeepRecent > 0 {
go stream.StartPruning(config.KeepRecent, config.PruneInterval)

Check notice

Code scanning / CodeQL

Spawning a Go routine Note

Spawning a Go routine may be a possible source of non-determinism
},
)
database.streamHandler = streamHandler
go database.writeAsyncInBackground()

Check notice

Code scanning / CodeQL

Spawning a Go routine Note

Spawning a Go routine may be a possible source of non-determinism
@yzang2019 yzang2019 merged commit 498ee09 into main Jun 26, 2024
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants