perf(store): move Update fsyncs out of the flock hold#126
Merged
Conversation
Sixteen-way clone bursts drain the VM-index flock as a staircase: every holder pays the file and parent-dir fsync inside the hold, so waiters queue behind stable-storage flushes (strace: 13/29 contended flock attempts, all fsync time in-hold). Update now renames under the lock and fsyncs after release — callers still return only once durable. Because the rename is unsynced, a crash can tear the fresh index on filesystems without rename-over data ordering; writeRaw therefore hardlinks the durable previous generation to <index>.prev (metadata only) before renaming, and load falls back to it loudly when the main file no longer decodes. Recovery keeps its promises narrow: a recovered main is never rotated into .prev (it is the torn one), read errors fail closed so a transient EIO cannot serve stale state, and the post-release fsync covers .prev too, closing the double-tear window when the previous writer died before its own fsync. Measured on the testbed: FC burst16 wall 131-167 -> 86-106ms, per-clone p50 -32%, burst64 p50 608 -> 467ms; serial unchanged; torn-index e2e recovers loudly.
Update composes UpdateNoDirSync so the .prev fsync covers both paths; the .prev rotation goes link+rename so a recovery generation exists at every instant (remove-then-link left a window with none); a double decode failure reports both errors; and the interface states the declared crash contract — the residual multi-writer window (all racing relaxed writers killed before their post-release fsyncs, then power loss) survives as old-or-new only under ext4-class rename-over data ordering, the deployment filesystem.
…aller's generation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Move the JSON store's
Updatefsyncs out of the flock hold, and make the store self-healing against the crash window that opens.Sixteen-way FC clone bursts drain the VM-index flock as a staircase: every holder pays the temp-file fsync and the parent-dir fsync inside the hold, so all waiters queue behind stable-storage flushes. In-burst strace of one clone: 13/29 contended flock attempts, ~all in-hold time spent in fsync; per-clone completions stagger ~8-10ms apart — the signature of a lock queue, not resource sharing.
Changes in
storage/json:Update/UpdateNoDirSyncnow rename under the lock and fsync (file, then parent dir forUpdate) after release — callers still return only once durable.writeRawtherefore hardlinks the durable previous generation to<index>.prev(metadata-only, no flush) before renaming, andloadfalls back to it loudly when the main file no longer decodes..prev(it is the torn one — rotating it would destroy the only good generation); read errors fail closed so a transient EIO cannot silently serve stale state to GC; the post-release fsync covers.prevtoo, closing the double-tear window when the previous writer died before its own fsync.Measured (bare-metal testbed, 16-core ext4 NVMe, rt:24.04 guest; interleaved rounds, same store)
Functional evidence
vms.jsonin a live store →vm listand a fullvm cloneboth pass through the loud.prevrecovery, and the next update rewrites a decodable main..prev, a second tear after a recovery update, read-error fail-closed, torn-without-prev erroring.go test -race ./...(linux): 29 packages ok.Declared crash contract (review-round closeout)
.prevrotation is link+rename, so a decodable recovery generation exists at every instant; a recovered (torn) main is never rotated in; read errors fail closed; the post-release fsync covers.prevtoo.auto_da_alloc, the deployment filesystem default). Closing this mechanically would put an fsync back inside the hold and re-serialize bursts; on ext4 the window is already closed by the kernel. Stated in theStore.Updatecontract.