fix(server): persist seal_on_add and sweep every store kind - #221
Merged
Conversation
Three follow-ups from lance-format#220, plus a regression guard for a bug I introduced while writing them. **`seal_on_add` is now persisted in the dataset.** It lived only in `GenericStoreOptions`, so a store created with it lost read-your-write the first time it was evicted from the LRU and reopened — the server passed a hardcoded `false`. Silent: no error, no data loss, just "sometimes I can't read what I just wrote", triggered by cache pressure rather than anything in the caller's code. It now sits in the schema metadata beside the spec, and a reopen restores it. Stores written before this fall back to the caller's option, so nothing breaks on upgrade. **The sweepers now visit every store kind.** Both hardcoded `state.rollout_stores`, so datagen's flushed generations accumulated forever (nothing ever merged them) and a generic store with the default deferred seal stayed invisible until someone called `/flush` by hand. Rather than copy the loop a third time — the exact duplication lance-format#214 removed one layer down — the traversal is generic over a new `Sweepable` trait in `sweeper.rs`. The trait is implemented on `Arc<RwLock<Store>>`, not on the store, so each kind keeps its own locking. That is load-bearing for rollout, whose merge splits into a shared-lock prepare (the expensive object-storage reads, during which appends keep flowing) and a brief exclusive-lock commit; a trait over `&mut Store` would have held the exclusive lock across the whole merge and quietly stalled the write path. Metric names keep their `rollout_` prefix and gain a `kind` label, so existing dashboards keep working. **Config naming: documented, not renamed.** `ROLLOUT_FLUSH_INTERVAL_SECS` and friends now govern all three store kinds, so the prefix is misleading — but renaming would silently break deployment manifests for no functional gain. The docs and the startup warning now say what they actually cover. **Regression guard for the CLI.** While rewriting those doc comments I deleted two `#[arg(..)]` attributes, which turned `--rollout-flush-interval-secs` and `--rollout-cleanup-interval-secs` into *required positional arguments*. Everything still compiled and all 203 Rust tests still passed; the server just refused to start. It surfaced only because the Python remote tests spawn the real binary. Two tests in `config.rs` now assert every field is an optional flag with a default, and that the minimal documented invocation parses. Verified each fix fails without its fix: reverting the persistence breaks both seal-mode tests, and dropping the `#[arg]` breaks both config tests. 9 new tests. Full suite: 203 core + 62 server + 18 api Rust, 199 Python; clippy, ruff, pyright all clean. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The three follow-ups from #220, plus a regression guard for a bug I introduced while writing them.
1.
seal_on_addis now persisted in the datasetIt lived only in
GenericStoreOptions, so a store created withseal_on_add: truelost read-your-write the first time it was evicted from the LRU and reopened — the server passed a hardcodedfalseatstate.rs:642.The failure mode is the nasty kind: no error, no data loss, just "sometimes I can't read what I just wrote" — and triggered by cache pressure, not by anything in the caller's code. Untestable locally, only shows up under load.
/merge-walwould accidentally paper over it (it flushes internally), making it look intermittent.The mode now sits in the schema metadata beside the spec, so a reopen restores what the store was created with. Stores written before this fall back to the caller's option, so nothing breaks on upgrade.
2. The sweepers now visit every store kind
Both
spawn_flush_sweeperandspawn_global_sweeperhardcodedstate.rollout_stores. Consequences:/flushby hand.Rather than copy the loop a third time — the exact duplication #214 removed one layer down — the traversal is generic over a new
Sweepabletrait insweeper.rs.Why the trait is on
Arc<RwLock<Store>>and not on the store: rollout's merge deliberately splits into a shared-lock prepare (the expensive object-storage reads, during which appends keep flowing) and a brief exclusive-lock commit. A trait over&mut Storewould have forced the exclusive lock across the whole merge and quietly stalled the write path — a performance regression that no test would have caught. Letting each kind own its locking preserves that.Metric names keep their
rollout_prefix and gain akindlabel, so existing dashboards and alerts keep working.3. Config naming: documented, not renamed
ROLLOUT_FLUSH_INTERVAL_SECSand friends now govern all three kinds, so the prefix is misleading. But renaming breaks deployment manifests for zero functional gain, so I documented the actual scope in the flag docs and the startup warning instead. Happy to add neutral aliases if you'd rather.4.⚠️ A regression I caused, and the guard for it
While rewriting those doc comments my edit deleted two
#[arg(..)]attributes, turning--rollout-flush-interval-secsand--rollout-cleanup-interval-secsinto required positional arguments:Everything still compiled. All 203 Rust tests still passed. The server just refused to start. It surfaced only because the Python remote tests spawn the real binary — and those tests swallow stderr, so it showed up as "server did not become healthy".
Two tests in
config.rsnow close that hole: every field must be an optional flag with a default, and the minimal documented invocation must parse.Verification
I checked each fix actually fails without its fix, rather than trusting a green run:
seal_mode_survives_*tests fail#[arg]→ both config tests fail9 new tests, including e2e ones that go through the real server path: seal mode surviving an actual LRU eviction (
generic_stores.lock().pop()), and a sweeper pass draining a real datagen generation.Full suite: 203 core + 62 server + 18 api Rust, 199 Python. clippy, ruff, pyright all clean.
🤖 Generated with Claude Code