Skip to content

perf(store): move Update fsyncs out of the flock hold#126

Merged
CMGS merged 3 commits into
masterfrom
perf/store-update-fsync
Jul 12, 2026
Merged

perf(store): move Update fsyncs out of the flock hold#126
CMGS merged 3 commits into
masterfrom
perf/store-update-fsync

Conversation

@CMGS

@CMGS CMGS commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

What

Move the JSON store's Update fsyncs out of the flock hold, and make the store self-healing against the crash window that opens.

Sixteen-way FC clone bursts drain the VM-index flock as a staircase: every holder pays the temp-file fsync and the parent-dir fsync inside the hold, so all waiters queue behind stable-storage flushes. In-burst strace of one clone: 13/29 contended flock attempts, ~all in-hold time spent in fsync; per-clone completions stagger ~8-10ms apart — the signature of a lock queue, not resource sharing.

Changes in storage/json:

  • Update/UpdateNoDirSync now rename under the lock and fsync (file, then parent dir for Update) after release — callers still return only once durable.
  • Because the rename is unsynced, a crash can tear the fresh index on filesystems without rename-over data ordering. writeRaw therefore hardlinks the durable previous generation to <index>.prev (metadata-only, no flush) before renaming, and load falls back to it loudly when the main file no longer decodes.
  • Review hardening (adversarial round): a recovered main is never rotated into .prev (it is the torn one — rotating it would destroy the only good generation); read errors fail closed so a transient EIO cannot silently serve stale state to GC; the post-release fsync covers .prev too, closing the double-tear window when the previous writer died before its own fsync.

Measured (bare-metal testbed, 16-core ext4 NVMe, rt:24.04 guest; interleaved rounds, same store)

metric master this PR
FC clone burst16 wall 131-167ms 86-113ms (−22~35%)
FC clone burst16 per-clone p50 66-99ms 52-70ms
FC clone burst64 per-clone p50 608-643ms 467-481ms
serial clone 24-27ms 24-26ms (unchanged)

Functional evidence

  • Torn-index e2e: corrupt vms.json in a live store → vm list and a full vm clone both pass through the loud .prev recovery, and the next update rewrites a decodable main.
  • Unit tests for all recovery scenarios: fallback, recovery-update preserving the good .prev, a second tear after a recovery update, read-error fail-closed, torn-without-prev erroring.
  • Both backends: clone/exec, stop/start cold boot, snapshot-of-clone → clone-of-clone — green.
  • Kill injection (5× SIGKILL at 5-30ms mid-clone): next clone heals, zero residue.
  • go test -race ./... (linux): 29 packages ok.

Declared crash contract (review-round closeout)

  • The .prev rotation is link+rename, so a decodable recovery generation exists at every instant; a recovered (torn) main is never rotated in; read errors fail closed; the post-release fsync covers .prev too.
  • Residual, explicitly accepted window: if several concurrent relaxed writers are ALL killed between unlock and their post-release fsyncs and power is then lost, the surviving main/.prev pair is old-or-new only under ext4-class rename-over data ordering (auto_da_alloc, the deployment filesystem default). Closing this mechanically would put an fsync back inside the hold and re-serialize bursts; on ext4 the window is already closed by the kernel. Stated in the Store.Update contract.
  • Pre-existing class, unchanged: a post-visibility fsync failure still returns an error after the mutation became readable to concurrent processes — identical in shape to master's rename-then-SyncParentDir failure path.

CMGS added 3 commits July 12, 2026 12:24
Sixteen-way clone bursts drain the VM-index flock as a staircase: every
holder pays the file and parent-dir fsync inside the hold, so waiters
queue behind stable-storage flushes (strace: 13/29 contended flock
attempts, all fsync time in-hold). Update now renames under the lock
and fsyncs after release — callers still return only once durable.
Because the rename is unsynced, a crash can tear the fresh index on
filesystems without rename-over data ordering; writeRaw therefore
hardlinks the durable previous generation to <index>.prev (metadata
only) before renaming, and load falls back to it loudly when the main
file no longer decodes. Recovery keeps its promises narrow: a recovered
main is never rotated into .prev (it is the torn one), read errors fail
closed so a transient EIO cannot serve stale state, and the post-release
fsync covers .prev too, closing the double-tear window when the previous
writer died before its own fsync.
Measured on the testbed: FC burst16 wall 131-167 -> 86-106ms, per-clone
p50 -32%, burst64 p50 608 -> 467ms; serial unchanged; torn-index e2e
recovers loudly.
Update composes UpdateNoDirSync so the .prev fsync covers both paths;
the .prev rotation goes link+rename so a recovery generation exists at
every instant (remove-then-link left a window with none); a double
decode failure reports both errors; and the interface states the
declared crash contract — the residual multi-writer window (all racing
relaxed writers killed before their post-release fsyncs, then power
loss) survives as old-or-new only under ext4-class rename-over data
ordering, the deployment filesystem.
@CMGS CMGS merged commit be8653e into master Jul 12, 2026
4 checks passed
@CMGS CMGS deleted the perf/store-update-fsync branch July 12, 2026 06:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant