Skip to content

Fix AOF recovery crash on torn-tail commit record#1910

Open
hamdaankhalid wants to merge 5 commits into
microsoft:mainfrom
hamdaankhalid:hkhalid/fix-aof-recovery
Open

Fix AOF recovery crash on torn-tail commit record#1910
hamdaankhalid wants to merge 5 commits into
microsoft:mainfrom
hamdaankhalid:hkhalid/fix-aof-recovery

Conversation

@hamdaankhalid

Copy link
Copy Markdown
Contributor

An ungraceful shutdown can leave the final AOF commit record half-written (a "torn tail"). During recovery, TsavoriteLogRecoveryInfo.Initialize wrote its fields before validating the embedded checksum, so ScanForwardForCommit's catch left info holding the torn record's UntilAddress. That address flowed into the recovery upper bound and AOF replay then failed its checksum ("Invalid checksum found during commit recovery").

  • Initialize: validate-before-mutate; a failed parse never mutates recovery state, and the checksum/cookie are now computed from parsed locals.
  • ScanForwardForCommit: log a clear warning when a torn tail is dropped.
  • TestUtils.CreateGarnetServer: add failOnRecoveryError so recovery errors can surface as a startup exception in tests.
  • RespAofTornTailTests: reproduce a torn-tail AOF, assert recovery does not crash and replays committed ops up to the last valid commit, plus a post-recovery write + re-recover cycle. Fails without the fix, passes with it.

An ungraceful shutdown can leave the final AOF commit record half-written (a
"torn tail"). During recovery, TsavoriteLogRecoveryInfo.Initialize wrote its
fields before validating the embedded checksum, so ScanForwardForCommit's catch
left `info` holding the torn record's UntilAddress. That address flowed into the
recovery upper bound and AOF replay then failed its checksum
("Invalid checksum found during commit recovery").

- Initialize: validate-before-mutate; a failed parse never mutates recovery
  state, and the checksum/cookie are now computed from parsed locals.
- ScanForwardForCommit: log a clear warning when a torn tail is dropped.
- TestUtils.CreateGarnetServer: add failOnRecoveryError so recovery errors can
  surface as a startup exception in tests.
- RespAofTornTailTests: reproduce a torn-tail AOF, assert recovery does not crash
  and replays committed ops up to the last valid commit, plus a post-recovery
  write + re-recover cycle. Fails without the fix, passes with it.
Copilot AI review requested due to automatic review settings July 3, 2026 03:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a Tsavorite AOF recovery failure mode where an ungraceful shutdown leaves the final commit record half-written (“torn tail”), causing recovery state to be incorrectly mutated and later AOF replay to fail.

Changes:

  • Make TsavoriteLogRecoveryInfo.Initialize(ReadOnlySpan<byte>) validate commit-record integrity (checksum) before mutating recovery fields.
  • Improve TsavoriteLogScanIterator.ScanForwardForCommit handling by warning and stopping cleanly when an invalid/torn record is encountered.
  • Add failOnRecoveryError plumbing to TestUtils.CreateGarnetServer and introduce a regression test that simulates a torn-tail commit record and verifies recovery + post-recovery writes.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
test/standalone/Garnet.test/TestUtils.cs Exposes FailOnRecoveryError via CreateGarnetServer to allow tests to surface recovery failures as startup exceptions.
test/standalone/Garnet.test/RespAofTornTailTests.cs Adds a regression test that corrupts the tail commit record on disk and validates recovery behavior.
libs/storage/Tsavorite/cs/src/core/TsavoriteLog/TsavoriteLogScanIterator.cs Tracks/logs the address of the invalid/torn record and stops scanning safely at the last valid commit.
libs/storage/Tsavorite/cs/src/core/TsavoriteLog/TsavoriteLogRecoveryInfo.cs Parses into locals and only commits state after checksum validation to avoid poisoning recovery bounds.
Comments suppressed due to low confidence (1)

libs/storage/Tsavorite/cs/src/core/TsavoriteLog/TsavoriteLogRecoveryInfo.cs:123

  • Initialize(ReadOnlySpan) reuses the same TsavoriteLogRecoveryInfo instance while scanning multiple commit records (see ScanForwardForCommit). If a prior commit had a cookie and a later commit record encodes cookieLength = -1 (no cookie), the current code leaves Cookie unchanged, so recovery can return the wrong cookie for the recovered commit.
                Cookie = cookie;
        }

        /// <summary>
        /// Reset
        /// </summary>
        public void Reset()

Comment thread test/standalone/Garnet.test/RespAofTornTailTests.cs
Comment thread test/standalone/Garnet.test/RespAofTornTailTests.cs
Comment thread test/standalone/Garnet.test/RespAofTornTailTests.cs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants