-
Notifications
You must be signed in to change notification settings - Fork 995
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enhancement: persist commit index in LogStore to accelerate recovery #613
base: main
Are you sure you want to change the base?
Enhancement: persist commit index in LogStore to accelerate recovery #613
Commits on Sep 1, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 2e5a8a0 - Browse repository at this point
Copy the full SHA 2e5a8a0View commit details
Commits on Sep 3, 2024
-
Configuration menu - View commit details
-
Copy full SHA for ffc6b3b - Browse repository at this point
Copy the full SHA ffc6b3bView commit details
Commits on Sep 4, 2024
-
feat(raft): add fast recovery mode for quicker log application
- Introduced a `fastRecovery` flag in the Raft structure and configuration to enable fast recovery mode. - Updated `NewRaft` to initialize `fastRecovery` from the configuration. - Added `persistCommitIndex` function to store the commit index when fast recovery is enabled. - Modified `processLogs` to persist the commit index before updating `lastApplied`. - Documented the `FastRecovery` option in the config.
Configuration menu - View commit details
-
Copy full SHA for 7383d96 - Browse repository at this point
Copy the full SHA 7383d96View commit details -
feat(raft): add recovery from committed logs during startup
- Implemented `recoverFromCommitedLogs` function to recover the Raft node from committed logs. - If `fastRecovery` is enabled and the log store implements `CommitTrackingLogStore`, the commit index is read from the store, avoiding the need to replay logs. - Logs between the last applied and commit index are fed into the FSM for faster recovery.
Configuration menu - View commit details
-
Copy full SHA for f6295e0 - Browse repository at this point
Copy the full SHA f6295e0View commit details
Commits on Sep 6, 2024
-
refactor(store): rename ReadCommitIndex to GetCommitIndex for consist…
…ency - Refactor `ReadCommitIndex` to `GetCommitIndex` across `LogStore` and `InmemCommitTrackingStore`. - Introduce `InmemCommitTrackingStore` to track commit index in memory for testing purposes. - Add locking mechanism to safely read/write commit index in `InmemCommitTrackingStore`.
Configuration menu - View commit details
-
Copy full SHA for f2ae7a9 - Browse repository at this point
Copy the full SHA f2ae7a9View commit details
Commits on Sep 10, 2024
-
Configuration menu - View commit details
-
Copy full SHA for ce1895c - Browse repository at this point
Copy the full SHA ce1895cView commit details -
Configuration menu - View commit details
-
Copy full SHA for ab50a58 - Browse repository at this point
Copy the full SHA ab50a58View commit details
Commits on Sep 13, 2024
-
refactor(inmem-commit-tracking-store): store commit index in memory u…
…ntil the next StoreLogs call and then write it out on disk along with the log data.
Configuration menu - View commit details
-
Copy full SHA for 4e7e04b - Browse repository at this point
Copy the full SHA 4e7e04bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 41df55e - Browse repository at this point
Copy the full SHA 41df55eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 400a27d - Browse repository at this point
Copy the full SHA 400a27dView commit details -
Configuration menu - View commit details
-
Copy full SHA for e2617e8 - Browse repository at this point
Copy the full SHA e2617e8View commit details
Commits on Sep 18, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 6daca47 - Browse repository at this point
Copy the full SHA 6daca47View commit details -
feat(raft): add commit tracking logs and fast recovery tests
- Implemented commit tracking logs in the Raft cluster configuration. - Added tests for restoring snapshots on startup with commit tracking logs. - Introduced a fast recovery test to ensure logs are applied correctly after restart. - Updated `MakeClusterOpts` to include `CommitTrackingLogs` option.
Configuration menu - View commit details
-
Copy full SHA for cc09317 - Browse repository at this point
Copy the full SHA cc09317View commit details
Commits on Sep 19, 2024
-
Configuration menu - View commit details
-
Copy full SHA for fe57b32 - Browse repository at this point
Copy the full SHA fe57b32View commit details -
Configuration menu - View commit details
-
Copy full SHA for 20e8701 - Browse repository at this point
Copy the full SHA 20e8701View commit details -
fix: rename persistCommitIndex to tryPersistCommitIndex
Updated the function name from `persistCommitIndex` to `tryPersistCommitIndex` to better reflect its behavior. This function now updates the commit index in the persistent store only if fast recovery is enabled and if the log store implements `CommitTrackingLogStore`. Adjusted references to this function in `dispatchLogs` and `appendEntries`.
Configuration menu - View commit details
-
Copy full SHA for 6f146e1 - Browse repository at this point
Copy the full SHA 6f146e1View commit details
Commits on Sep 20, 2024
-
Configuration menu - View commit details
-
Copy full SHA for a8438b0 - Browse repository at this point
Copy the full SHA a8438b0View commit details -
refactor(log): introduce StagCommitIndex for optimized atomic persist…
…ence - Rename SetCommitIndex to StagCommitIndex in CommitTrackingLogStore interface - Add detailed documentation for StagCommitIndex method - Update raft.go to use StagCommitIndex instead of SetCommitIndex - Optimize commit index staging in appendEntries using min(lastNewIndex, leaderCommitIndex) This change ensures commit index updates are only persisted atomically with the following StoreLogs call, preventing inconsistencies between the commit index and log entries in case of crashes. It also optimizes the staged commit index to be as up-to-date as possible without exceeding the last new entry index, reducing commit index lag in the CommitTrackingStore. The contract for implementations is clarified, specifying that GetCommitIndex must never return a value higher than the last index in the log.
Configuration menu - View commit details
-
Copy full SHA for 5e6d8a4 - Browse repository at this point
Copy the full SHA 5e6d8a4View commit details
Commits on Sep 24, 2024
-
fix(raft): correct CommitTrackingLogStore implementation
- Rename SetCommitIndex to StageCommitIndex in InmemCommitTrackingStore - Fix typo in CommitTrackingLogStore interface (StagCommitIndex to StageCommitIndex) - Update error message in tryStageCommitIndex for consistency - Ensure CommitTrackingLogStore extends LogStore interface
Configuration menu - View commit details
-
Copy full SHA for e248f00 - Browse repository at this point
Copy the full SHA e248f00View commit details -
feat(raft): improve fast recovery error handling and commit index val…
…idation - Add error logging and panic for critical errors during recovery - Replace lastApplied check with lastIndex comparison - Ensure commitIndex does not exceed lastIndex - Improve code structure and readability
Configuration menu - View commit details
-
Copy full SHA for 2a913ab - Browse repository at this point
Copy the full SHA 2a913abView commit details
Commits on Oct 9, 2024
-
feat: add
CommitTrackingLogStore
interface check and adjust return ……type This commit adds a compile-time interface check for InmemCommitTrackingStore to ensure it implements the CommitTrackingLogStore interface. It also changes the return type of NewInmemCommitTrackingStore to be more specific. - Add interface check: var _ CommitTrackingLogStore = &InmemCommitTrackingStore{} - Change NewInmemCommitTrackingStore return type from CommitTrackingLogStore to *InmemCommitTrackingStore
Configuration menu - View commit details
-
Copy full SHA for 7cd6732 - Browse repository at this point
Copy the full SHA 7cd6732View commit details -
refactor: improve type assertion for log store in TestRaft_FastRecovery
This commit enhances the type assertion for CommitTrackingLogStore in the TestRaft_FastRecovery function, adding explicit error handling for better test reliability and clarity. - Add explicit type assertion check for CommitTrackingLogStore interface - Introduce new variable 'store' to hold the asserted CommitTrackingLogStore - Add error handling to fail the test if the type assertion is unsuccessful
Configuration menu - View commit details
-
Copy full SHA for 92c04a0 - Browse repository at this point
Copy the full SHA 92c04a0View commit details
Commits on Oct 10, 2024
-
feat: add warning log for unsupported fast recovery
This commit adds a warning log message when fast recovery is enabled but the log store does not support the CommitTrackingLogStore interface. This provides better visibility into potential configuration issues. - Add warning log in recoverFromCommittedLogs when log store doesn't implement CommitTrackingLogStore - Include the type of the log store in the warning message for easier debugging
Configuration menu - View commit details
-
Copy full SHA for 8e8ba07 - Browse repository at this point
Copy the full SHA 8e8ba07View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2020cab - Browse repository at this point
Copy the full SHA 2020cabView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2a7d584 - Browse repository at this point
Copy the full SHA 2a7d584View commit details
Commits on Oct 11, 2024
-
refactor: rename FastRecovery and revert the stageCommittedIdx change
This commit renames the FastRecovery feature to RestoreCommittedLogs for better clarity and consistency. This is a breaking change as it modifies public API elements. - Rename Config.FastRecovery to Config.RestoreCommittedLogs - Rename Raft.fastRecovery to Raft.RestoreCommittedLogs - Update all references to fastRecovery in the codebase - Rename TestRaft_FastRecovery to TestRaft_RestoreCommittedLogs - Update comments to reflect the new terminology - Refactor tryStageCommitIndex to accept commitIndex as a parameter BREAKING CHANGE: Config.FastRecovery has been renamed to Config.RestoreCommittedLogs
Configuration menu - View commit details
-
Copy full SHA for bdac45b - Browse repository at this point
Copy the full SHA bdac45bView commit details -
Configuration menu - View commit details
-
Copy full SHA for ed47a25 - Browse repository at this point
Copy the full SHA ed47a25View commit details -
Configuration menu - View commit details
-
Copy full SHA for ad87d86 - Browse repository at this point
Copy the full SHA ad87d86View commit details -
Configuration menu - View commit details
-
Copy full SHA for 30fc43e - Browse repository at this point
Copy the full SHA 30fc43eView commit details -
docs: clarify RestoreCommittedLogs configuration requirement
- Add a notice in the Config struct documentation for RestoreCommittedLogs - Specify that Raft will fail to start with ErrIncompatibleLogStore if the requirement is not met
Configuration menu - View commit details
-
Copy full SHA for e797962 - Browse repository at this point
Copy the full SHA e797962View commit details -
Configuration menu - View commit details
-
Copy full SHA for 500567f - Browse repository at this point
Copy the full SHA 500567fView commit details -
refactor!: update MakeCluster functions to return error
- Update makeCluster to return (*cluster, error) - Modify MakeCluster, MakeClusterNoBootstrap, and MakeClusterCustom to return error - Update all test cases to handle potential errors from cluster creation - Replace t.Fatalf() calls with t.Logf() and error returns in makeCluster BREAKING CHANGE: MakeCluster, MakeClusterNoBootstrap, and MakeClusterCustom now return an additional error value, which needs to be handled in existing tests.
Configuration menu - View commit details
-
Copy full SHA for cfffcb5 - Browse repository at this point
Copy the full SHA cfffcb5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 560c0b9 - Browse repository at this point
Copy the full SHA 560c0b9View commit details
Commits on Oct 12, 2024
-
Revert "refactor!: update MakeCluster functions to return error"
This reverts commit cfffcb5.
Configuration menu - View commit details
-
Copy full SHA for 8c722fa - Browse repository at this point
Copy the full SHA 8c722faView commit details -
refactor: update makeCluster to return errors
- Add PropagateError option to MakeClusterOpts - Update makeCluster to return (*cluster, error) - Modify MakeCluster, MakeClusterNo
Configuration menu - View commit details
-
Copy full SHA for 300a6e7 - Browse repository at this point
Copy the full SHA 300a6e7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8d11a28 - Browse repository at this point
Copy the full SHA 8d11a28View commit details
Commits on Oct 15, 2024
-
docs: clarify GetCommitIndex behavior in CommitTrackingLogStore inter…
…face - Specify that GetCommitIndex should not return a value higher than the last index in the log - Clarify that if a higher value is returned, the last index in the log will be used instead - Add instruction to return (0, nil) when no commit index is found in the log store
Configuration menu - View commit details
-
Copy full SHA for 1bdf161 - Browse repository at this point
Copy the full SHA 1bdf161View commit details