Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhancement: persist commit index in LogStore to accelerate recovery #613

Open
wants to merge 37 commits into
base: main
Choose a base branch
from

Commits on Sep 1, 2024

  1. Configuration menu
    Copy the full SHA
    2e5a8a0 View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. Configuration menu
    Copy the full SHA
    ffc6b3b View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2024

  1. feat(raft): add fast recovery mode for quicker log application

    - Introduced a `fastRecovery` flag in the Raft structure and configuration to enable fast recovery mode.
    - Updated `NewRaft` to initialize `fastRecovery` from the configuration.
    - Added `persistCommitIndex` function to store the commit index when fast recovery is enabled.
    - Modified `processLogs` to persist the commit index before updating `lastApplied`.
    - Documented the `FastRecovery` option in the config.
    peterxcli committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    7383d96 View commit details
    Browse the repository at this point in the history
  2. feat(raft): add recovery from committed logs during startup

    - Implemented `recoverFromCommitedLogs` function to recover the Raft node from committed logs.
    - If `fastRecovery` is enabled and the log store implements `CommitTrackingLogStore`, the commit index is read from the store, avoiding the need to replay logs.
    - Logs between the last applied and commit index are fed into the FSM for faster recovery.
    peterxcli committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    f6295e0 View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. refactor(store): rename ReadCommitIndex to GetCommitIndex for consist…

    …ency
    
    - Refactor `ReadCommitIndex` to `GetCommitIndex` across `LogStore` and `InmemCommitTrackingStore`.
    - Introduce `InmemCommitTrackingStore` to track commit index in memory for testing purposes.
    - Add locking mechanism to safely read/write commit index in `InmemCommitTrackingStore`.
    peterxcli committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    f2ae7a9 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. Configuration menu
    Copy the full SHA
    ce1895c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ab50a58 View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2024

  1. refactor(inmem-commit-tracking-store): store commit index in memory u…

    …ntil the next StoreLogs call and then write it out on disk along with the log data.
    peterxcli committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    4e7e04b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    41df55e View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    400a27d View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    e2617e8 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. Configuration menu
    Copy the full SHA
    6daca47 View commit details
    Browse the repository at this point in the history
  2. feat(raft): add commit tracking logs and fast recovery tests

    - Implemented commit tracking logs in the Raft cluster configuration.
    - Added tests for restoring snapshots on startup with commit tracking logs.
    - Introduced a fast recovery test to ensure logs are applied correctly after restart.
    - Updated `MakeClusterOpts` to include `CommitTrackingLogs` option.
    peterxcli committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    cc09317 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. Configuration menu
    Copy the full SHA
    fe57b32 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    20e8701 View commit details
    Browse the repository at this point in the history
  3. fix: rename persistCommitIndex to tryPersistCommitIndex

    Updated the function name from `persistCommitIndex` to `tryPersistCommitIndex` to better reflect its behavior. This function now updates the commit index in the persistent store only if fast recovery is enabled and if the log store implements `CommitTrackingLogStore`. Adjusted references to this function in `dispatchLogs` and `appendEntries`.
    peterxcli committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    6f146e1 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. Configuration menu
    Copy the full SHA
    a8438b0 View commit details
    Browse the repository at this point in the history
  2. refactor(log): introduce StagCommitIndex for optimized atomic persist…

    …ence
    
    - Rename SetCommitIndex to StagCommitIndex in CommitTrackingLogStore interface
    - Add detailed documentation for StagCommitIndex method
    - Update raft.go to use StagCommitIndex instead of SetCommitIndex
    - Optimize commit index staging in appendEntries using min(lastNewIndex, leaderCommitIndex)
    
    This change ensures commit index updates are only persisted atomically
    with the following StoreLogs call, preventing inconsistencies between the
    commit index and log entries in case of crashes. It also optimizes the
    staged commit index to be as up-to-date as possible without exceeding the
    last new entry index, reducing commit index lag in the CommitTrackingStore.
    The contract for implementations is clarified, specifying that GetCommitIndex
    must never return a value higher than the last index in the log.
    peterxcli committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    5e6d8a4 View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. fix(raft): correct CommitTrackingLogStore implementation

    - Rename SetCommitIndex to StageCommitIndex in InmemCommitTrackingStore
    - Fix typo in CommitTrackingLogStore interface (StagCommitIndex to StageCommitIndex)
    - Update error message in tryStageCommitIndex for consistency
    - Ensure CommitTrackingLogStore extends LogStore interface
    peterxcli committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    e248f00 View commit details
    Browse the repository at this point in the history
  2. feat(raft): improve fast recovery error handling and commit index val…

    …idation
    
    - Add error logging and panic for critical errors during recovery
    - Replace lastApplied check with lastIndex comparison
    - Ensure commitIndex does not exceed lastIndex
    - Improve code structure and readability
    peterxcli committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    2a913ab View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2024

  1. feat: add CommitTrackingLogStore interface check and adjust return …

    …type
    
    This commit adds a compile-time interface check for InmemCommitTrackingStore
    to ensure it implements the CommitTrackingLogStore interface. It also changes
    the return type of NewInmemCommitTrackingStore to be more specific.
    
    - Add interface check: var _ CommitTrackingLogStore = &InmemCommitTrackingStore{}
    - Change NewInmemCommitTrackingStore return type from CommitTrackingLogStore to *InmemCommitTrackingStore
    peterxcli committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    7cd6732 View commit details
    Browse the repository at this point in the history
  2. refactor: improve type assertion for log store in TestRaft_FastRecovery

    This commit enhances the type assertion for CommitTrackingLogStore in the
    TestRaft_FastRecovery function, adding explicit error handling for better
    test reliability and clarity.
    
    - Add explicit type assertion check for CommitTrackingLogStore interface
    - Introduce new variable 'store' to hold the asserted CommitTrackingLogStore
    - Add error handling to fail the test if the type assertion is unsuccessful
    peterxcli committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    92c04a0 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2024

  1. feat: add warning log for unsupported fast recovery

    This commit adds a warning log message when fast recovery is enabled but
    the log store does not support the CommitTrackingLogStore interface. This
    provides better visibility into potential configuration issues.
    
    - Add warning log in recoverFromCommittedLogs when log store doesn't
      implement CommitTrackingLogStore
    - Include the type of the log store in the warning message for easier debugging
    peterxcli committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    8e8ba07 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2020cab View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    2a7d584 View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2024

  1. refactor: rename FastRecovery and revert the stageCommittedIdx change

    This commit renames the FastRecovery feature to RestoreCommittedLogs for
    better clarity and consistency. This is a breaking change as it modifies
    public API elements.
    
    - Rename Config.FastRecovery to Config.RestoreCommittedLogs
    - Rename Raft.fastRecovery to Raft.RestoreCommittedLogs
    - Update all references to fastRecovery in the codebase
    - Rename TestRaft_FastRecovery to TestRaft_RestoreCommittedLogs
    - Update comments to reflect the new terminology
    - Refactor tryStageCommitIndex to accept commitIndex as a parameter
    
    BREAKING CHANGE: Config.FastRecovery has been renamed to Config.RestoreCommittedLogs
    peterxcli committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    bdac45b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ed47a25 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    ad87d86 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    30fc43e View commit details
    Browse the repository at this point in the history
  5. docs: clarify RestoreCommittedLogs configuration requirement

    - Add a notice in the Config struct documentation for RestoreCommittedLogs
    - Specify that Raft will fail to start with ErrIncompatibleLogStore if the
      requirement is not met
    peterxcli committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    e797962 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    500567f View commit details
    Browse the repository at this point in the history
  7. refactor!: update MakeCluster functions to return error

    - Update makeCluster to return (*cluster, error)
    - Modify MakeCluster, MakeClusterNoBootstrap, and MakeClusterCustom to return error
    - Update all test cases to handle potential errors from cluster creation
    - Replace t.Fatalf() calls with t.Logf() and error returns in makeCluster
    
    BREAKING CHANGE: MakeCluster, MakeClusterNoBootstrap, and MakeClusterCustom
    now return an additional error value, which needs to be handled in existing tests.
    peterxcli committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    cfffcb5 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    560c0b9 View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2024

  1. Configuration menu
    Copy the full SHA
    8c722fa View commit details
    Browse the repository at this point in the history
  2. refactor: update makeCluster to return errors

    - Add PropagateError option to MakeClusterOpts
    - Update makeCluster to return (*cluster, error)
    - Modify MakeCluster, MakeClusterNo
    peterxcli committed Oct 12, 2024
    Configuration menu
    Copy the full SHA
    300a6e7 View commit details
    Browse the repository at this point in the history
  3. Use wrapped err

    peterxcli committed Oct 12, 2024
    Configuration menu
    Copy the full SHA
    8d11a28 View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2024

  1. docs: clarify GetCommitIndex behavior in CommitTrackingLogStore inter…

    …face
    
    - Specify that GetCommitIndex should not return a value higher than the last
      index in the log
    - Clarify that if a higher value is returned, the last index in the log will
      be used instead
    - Add instruction to return (0, nil) when no commit index is found in the
      log store
    peterxcli committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    1bdf161 View commit details
    Browse the repository at this point in the history