Skip to content

fix(server): persist seal_on_add and sweep every store kind - #221

Merged
beinan merged 1 commit into
lance-format:mainfrom
beinan:sweeper-and-seal
Jul 28, 2026
Merged

fix(server): persist seal_on_add and sweep every store kind#221
beinan merged 1 commit into
lance-format:mainfrom
beinan:sweeper-and-seal

Conversation

@beinan

@beinan beinan commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

The three follow-ups from #220, plus a regression guard for a bug I introduced while writing them.

1. seal_on_add is now persisted in the dataset

It lived only in GenericStoreOptions, so a store created with seal_on_add: true lost read-your-write the first time it was evicted from the LRU and reopened — the server passed a hardcoded false at state.rs:642.

The failure mode is the nasty kind: no error, no data loss, just "sometimes I can't read what I just wrote" — and triggered by cache pressure, not by anything in the caller's code. Untestable locally, only shows up under load. /merge-wal would accidentally paper over it (it flushes internally), making it look intermittent.

The mode now sits in the schema metadata beside the spec, so a reopen restores what the store was created with. Stores written before this fall back to the caller's option, so nothing breaks on upgrade.

2. The sweepers now visit every store kind

Both spawn_flush_sweeper and spawn_global_sweeper hardcoded state.rollout_stores. Consequences:

  • datagen: seals on every append, so flush didn't matter — but nothing ever merged its generations. They accumulated forever and every read unioned all of them. This predates refactor(core): migrate DatagenStore onto StorageBase #217.
  • generic: defaults to deferred seal, so writes stayed invisible until someone called /flush by hand.

Rather than copy the loop a third time — the exact duplication #214 removed one layer down — the traversal is generic over a new Sweepable trait in sweeper.rs.

Why the trait is on Arc<RwLock<Store>> and not on the store: rollout's merge deliberately splits into a shared-lock prepare (the expensive object-storage reads, during which appends keep flowing) and a brief exclusive-lock commit. A trait over &mut Store would have forced the exclusive lock across the whole merge and quietly stalled the write path — a performance regression that no test would have caught. Letting each kind own its locking preserves that.

Metric names keep their rollout_ prefix and gain a kind label, so existing dashboards and alerts keep working.

3. Config naming: documented, not renamed

ROLLOUT_FLUSH_INTERVAL_SECS and friends now govern all three kinds, so the prefix is misleading. But renaming breaks deployment manifests for zero functional gain, so I documented the actual scope in the flag docs and the startup warning instead. Happy to add neutral aliases if you'd rather.

4. ⚠️ A regression I caused, and the guard for it

While rewriting those doc comments my edit deleted two #[arg(..)] attributes, turning --rollout-flush-interval-secs and --rollout-cleanup-interval-secs into required positional arguments:

error: the following required arguments were not provided:
  <ROLLOUT_CLEANUP_INTERVAL_SECS>
  <ROLLOUT_FLUSH_INTERVAL_SECS>

Everything still compiled. All 203 Rust tests still passed. The server just refused to start. It surfaced only because the Python remote tests spawn the real binary — and those tests swallow stderr, so it showed up as "server did not become healthy".

Two tests in config.rs now close that hole: every field must be an optional flag with a default, and the minimal documented invocation must parse.

Verification

I checked each fix actually fails without its fix, rather than trusting a green run:

  • reverting the persistence → both seal_mode_survives_* tests fail
  • dropping the #[arg] → both config tests fail

9 new tests, including e2e ones that go through the real server path: seal mode surviving an actual LRU eviction (generic_stores.lock().pop()), and a sweeper pass draining a real datagen generation.

Full suite: 203 core + 62 server + 18 api Rust, 199 Python. clippy, ruff, pyright all clean.

🤖 Generated with Claude Code

Three follow-ups from lance-format#220, plus a regression guard for a bug I introduced
while writing them.

**`seal_on_add` is now persisted in the dataset.** It lived only in
`GenericStoreOptions`, so a store created with it lost read-your-write the
first time it was evicted from the LRU and reopened — the server passed a
hardcoded `false`. Silent: no error, no data loss, just "sometimes I can't
read what I just wrote", triggered by cache pressure rather than anything in
the caller's code. It now sits in the schema metadata beside the spec, and a
reopen restores it. Stores written before this fall back to the caller's
option, so nothing breaks on upgrade.

**The sweepers now visit every store kind.** Both hardcoded
`state.rollout_stores`, so datagen's flushed generations accumulated forever
(nothing ever merged them) and a generic store with the default deferred seal
stayed invisible until someone called `/flush` by hand. Rather than copy the
loop a third time — the exact duplication lance-format#214 removed one layer down — the
traversal is generic over a new `Sweepable` trait in `sweeper.rs`.

The trait is implemented on `Arc<RwLock<Store>>`, not on the store, so each
kind keeps its own locking. That is load-bearing for rollout, whose merge
splits into a shared-lock prepare (the expensive object-storage reads, during
which appends keep flowing) and a brief exclusive-lock commit; a trait over
`&mut Store` would have held the exclusive lock across the whole merge and
quietly stalled the write path.

Metric names keep their `rollout_` prefix and gain a `kind` label, so
existing dashboards keep working.

**Config naming: documented, not renamed.** `ROLLOUT_FLUSH_INTERVAL_SECS` and
friends now govern all three store kinds, so the prefix is misleading — but
renaming would silently break deployment manifests for no functional gain.
The docs and the startup warning now say what they actually cover.

**Regression guard for the CLI.** While rewriting those doc comments I
deleted two `#[arg(..)]` attributes, which turned `--rollout-flush-interval-secs`
and `--rollout-cleanup-interval-secs` into *required positional arguments*.
Everything still compiled and all 203 Rust tests still passed; the server just
refused to start. It surfaced only because the Python remote tests spawn the
real binary. Two tests in `config.rs` now assert every field is an optional
flag with a default, and that the minimal documented invocation parses.

Verified each fix fails without its fix: reverting the persistence breaks both
seal-mode tests, and dropping the `#[arg]` breaks both config tests.

9 new tests. Full suite: 203 core + 62 server + 18 api Rust, 199 Python;
clippy, ruff, pyright all clean.

Co-Authored-By: Claude <noreply@anthropic.com>
@beinan
beinan merged commit 0ff8788 into lance-format:main Jul 28, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant