Fix AOF recovery crash on torn-tail commit record#1910
Open
hamdaankhalid wants to merge 5 commits into
Open
Conversation
An ungraceful shutdown can leave the final AOF commit record half-written (a
"torn tail"). During recovery, TsavoriteLogRecoveryInfo.Initialize wrote its
fields before validating the embedded checksum, so ScanForwardForCommit's catch
left `info` holding the torn record's UntilAddress. That address flowed into the
recovery upper bound and AOF replay then failed its checksum
("Invalid checksum found during commit recovery").
- Initialize: validate-before-mutate; a failed parse never mutates recovery
state, and the checksum/cookie are now computed from parsed locals.
- ScanForwardForCommit: log a clear warning when a torn tail is dropped.
- TestUtils.CreateGarnetServer: add failOnRecoveryError so recovery errors can
surface as a startup exception in tests.
- RespAofTornTailTests: reproduce a torn-tail AOF, assert recovery does not crash
and replays committed ops up to the last valid commit, plus a post-recovery
write + re-recover cycle. Fails without the fix, passes with it.
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes a Tsavorite AOF recovery failure mode where an ungraceful shutdown leaves the final commit record half-written (“torn tail”), causing recovery state to be incorrectly mutated and later AOF replay to fail.
Changes:
- Make
TsavoriteLogRecoveryInfo.Initialize(ReadOnlySpan<byte>)validate commit-record integrity (checksum) before mutating recovery fields. - Improve
TsavoriteLogScanIterator.ScanForwardForCommithandling by warning and stopping cleanly when an invalid/torn record is encountered. - Add
failOnRecoveryErrorplumbing toTestUtils.CreateGarnetServerand introduce a regression test that simulates a torn-tail commit record and verifies recovery + post-recovery writes.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| test/standalone/Garnet.test/TestUtils.cs | Exposes FailOnRecoveryError via CreateGarnetServer to allow tests to surface recovery failures as startup exceptions. |
| test/standalone/Garnet.test/RespAofTornTailTests.cs | Adds a regression test that corrupts the tail commit record on disk and validates recovery behavior. |
| libs/storage/Tsavorite/cs/src/core/TsavoriteLog/TsavoriteLogScanIterator.cs | Tracks/logs the address of the invalid/torn record and stops scanning safely at the last valid commit. |
| libs/storage/Tsavorite/cs/src/core/TsavoriteLog/TsavoriteLogRecoveryInfo.cs | Parses into locals and only commits state after checksum validation to avoid poisoning recovery bounds. |
Comments suppressed due to low confidence (1)
libs/storage/Tsavorite/cs/src/core/TsavoriteLog/TsavoriteLogRecoveryInfo.cs:123
- Initialize(ReadOnlySpan) reuses the same TsavoriteLogRecoveryInfo instance while scanning multiple commit records (see ScanForwardForCommit). If a prior commit had a cookie and a later commit record encodes cookieLength = -1 (no cookie), the current code leaves Cookie unchanged, so recovery can return the wrong cookie for the recovered commit.
Cookie = cookie;
}
/// <summary>
/// Reset
/// </summary>
public void Reset()
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
badrishc
approved these changes
Jul 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An ungraceful shutdown can leave the final AOF commit record half-written (a "torn tail"). During recovery, TsavoriteLogRecoveryInfo.Initialize wrote its fields before validating the embedded checksum, so ScanForwardForCommit's catch left
infoholding the torn record's UntilAddress. That address flowed into the recovery upper bound and AOF replay then failed its checksum ("Invalid checksum found during commit recovery").