perf(master): skip re-observing experiments whose base version has not moved by beinan · Pull Request #209 · lance-format/lance-context

beinan · 2026-07-27T08:39:51Z

First of three steps toward surviving tens of thousands of experiments. #207 fixed how _stats is written; this fixes how much work a scan does.

The problem

A scan round fully observes every experiment, every 300s. observe is not cheap:

step	cost
`wal_shard_snapshots()`	list every MemWAL shard + read each manifest
`pending_wal_rows()`	open every flushed generation
`count_rows()`	scan the base table
`manifest.version`	free — already in memory

At 30k experiments with concurrency 8: ~19 min at an optimistic 0.3s each, ~90 min at object-storage latencies — against a 5 minute interval. The next round starts before the previous ends, forever. No choice of stats storage fixes this; the scan itself is the wall.

The fix

RolloutObservation already returned the base dataset version — StatRow just never persisted it. The cheapest possible change signal was being thrown away.

Opening the dataset reads its manifest, which is where the version comes from, so the comparison is free. Unchanged version ⇒ no write landed ⇒ every derived metric is necessarily unchanged ⇒ reuse the previous row and skip everything above.

Cold experiments now cost one open per round instead of a full observation.

Also removes a per-experiment stats.get() that took the stats lock once per experiment per round; the previous snapshot is now read once up front and passed down. observe_one no longer needs MasterState at all.

Two edges, both tested

1. Existing tables have no version column. Reads tolerate the absence and record UNKNOWN_VERSION, forcing one full observation which then persists the column. Erroring instead would break every read on upgrade — and reads are what the UI and both auto-sweeps depend on, so the master would come up blind until the table happened to be rewritten. There's a test that builds a legacy-schema table and asserts it reads, keeps its data, and migrates on first snapshot.

2. refresh_one must not skip. Its callers use it precisely when they believe something just changed (post-compaction, explicit refresh) — the version check would short-circuit exactly the observation they asked for. It passes None.

Observability

master_scan_experiments_skipped / _observed. A ratio near 1 is the healthy state at scale; trending to 0 means the incremental path stopped engaging (usually a stats table losing its rows).

Testing

4 new scanner tests + 1 migration test. Verified non-vacuous: disabling the skip makes 2 of the 4 fail.

Full workspace: 178 core + 24 master + 51 server + 5 concurrency + 1 merge-cleanup, all pass. clippy -D warnings and fmt clean.

C2 — retire experiments cold for 7 days: merge WAL → compact → verify pending_wal_generations == 0 → drop from the hot table. The ordering matters: compact_files does not touch WAL generations, so retiring before draining them would strand those generations forever (the WAL sweep only reads this table) and leak _mem_wal/ directories.
C3 — UI becomes "recently active N + search", with cold experiments observed on demand via the registry.

🤖 Generated with Claude Code

…t moved A scan round fully observed every experiment, every 300s. `observe` lists every MemWAL shard and reads its manifest, opens each flushed generation to count pending rows, and runs `count_rows` over the base table. At the tens of thousands of experiments this deployment is heading for -- the overwhelming majority of them cold and never written again -- a round cannot finish inside its interval: roughly 19 minutes at an optimistic 0.3s per experiment and concurrency 8, and closer to 90 on object storage. The next round starts before the previous one ends, forever. `RolloutObservation` already carried the base dataset version, but `StatRow` never persisted it, so the cheapest possible change signal was being discarded. Opening the dataset reads its manifest, which is where the version comes from, so comparing it against the last observed version costs nothing. An unchanged version means no write has landed, so every derived metric is necessarily unchanged and the previous row can be reused verbatim -- skipping all of the above. Cold experiments now cost one open per round rather than a full observation. Also removes a per-experiment `stats.get()` that took the stats lock once per experiment per round; the previous snapshot is read once up front and passed down. `observe_one` no longer needs `MasterState` at all. Two edges, both tested: `version` is absent from a `_stats` table written before this commit. Reads tolerate that and record `UNKNOWN_VERSION`, which forces one full observation and then persists the column. Erroring instead would break every read on upgrade -- and reads are what the UI and both auto-sweeps depend on, so the master would come up blind until the table happened to be rewritten. `refresh_one` passes `None` so it always observes fully. Its callers use it precisely when they believe something just changed (after a compaction, or an explicit refresh), and the version check would short-circuit exactly the observation they asked for. New gauges `master_scan_experiments_skipped` / `_observed` show how much of each round the version check avoided. A ratio near 1 is the healthy state at scale; a ratio trending to 0 means the incremental path has stopped engaging. Verified the tests are not vacuous: disabling the skip fails two of the four. Co-Authored-By: Claude <noreply@anthropic.com>

Stacked on lance-format#209. The stats table currently holds a row for every experiment that has ever existed, and each one costs an open on every scan round. At the scale this deployment is heading for — tens of thousands, the overwhelming majority written once and never again — that is the remaining reason a round cannot stay proportional to actual work. An experiment with no writes for `STATS_COLD_RETIRE_SECS` (default 7 days) is now dropped from the table. It stays in the registry and can still be observed on demand; it simply stops costing a row and an open every round. Retirement order is a correctness property, not an optimisation. Both auto-sweeps read the stats table and nothing else, so an experiment absent from it is invisible to them permanently. Dropping one that still has un-merged MemWAL generations would strand them: the rows stay readable because the read path unions the generations, but they are never folded into the base table, read amplification never recovers, the `_mem_wal/{shard}/` directories are never reclaimed, and no process is left that would notice. So retirement is: merge the WAL -> compact -> verify pending_wal_generations == 0 -> drop the row `compact_files` deliberately leaves MemWAL generations alone, so compaction on its own would not have been sufficient. Compaction is still done, because a retired table is one nothing will come back to tidy. Any step failing leaves the experiment in the table for the next round. Refusing to retire is always safe; retiring early is not. `STATS_COLD_RETIRE_SECS=0` disables retirement entirely. Every existing test config sets it to 0, so they exercise the unchanged path. Four tests, the important one asserting the shard is genuinely drained after retirement rather than that the function returned success. Verified it is not vacuous: removing the WAL-merge step makes it fail with exactly the stranding it exists to prevent. New: `master_stats_hot_experiments` (table size, now bounded rather than tracking the registry), `master_stats_experiments_retired_total`, `master_stats_retire_failures_total`. Co-Authored-By: Claude <noreply@anthropic.com>

…#210) ⚠️ **Stacked on #209** — that PR's commit is included here, so review #209 first and merge it before this one. Once #209 lands, this diff reduces to the retirement commit alone. ## Why #209 made cold experiments cheap (one open instead of a full observation). They still cost *something*, and the table still holds a row for every experiment that has ever existed. At tens of thousands — the overwhelming majority written once and never again — that's the remaining reason a scan round can't stay proportional to actual work. An experiment with no writes for `STATS_COLD_RETIRE_SECS` (default **7 days**) is dropped from the table. It stays in the registry and can still be observed on demand; it just stops costing a row and an open every round. ## Retirement order is a correctness property This is the part worth reviewing carefully. Both auto-sweeps read **the stats table and nothing else** (verified — the registry references in `scheduler.rs` are all test-only). So an experiment absent from it is **invisible to them permanently**. Dropping one that still has un-merged MemWAL generations would strand them: - rows stay readable (the read path unions generations) — so **nothing looks broken** - but they're never folded into the base table - read amplification never recovers - `_mem_wal/{shard}/` directories are never reclaimed - **and no process is left that would notice** So the order is: ``` merge WAL → compact → verify pending_wal_generations == 0 → drop the row ``` ⚠️ `compact_files` **deliberately leaves MemWAL generations alone**, so compaction alone would not have been sufficient. Compaction is still done, because a retired table is one nothing will come back to tidy. **Any step failing leaves the experiment in the table for the next round.** Refusing to retire is always safe; retiring early is not. ## Testing Four tests. The important one asserts the shard is **genuinely drained after retirement** — reopening the store and checking `pending_wal_generations == 0` — rather than that the function returned success. **Verified non-vacuous:** removing the WAL-merge step makes it fail with exactly the stranding it exists to prevent: ``` assertion failed: retirement must leave zero pending generations, or they are stranded forever ``` Full workspace: 178 core + 28 master + 51 server, all pass. clippy `-D warnings` and fmt clean. ## Config `STATS_COLD_RETIRE_SECS=0` disables retirement entirely. Every existing test config sets it to `0`, so they exercise the unchanged path. ## Metrics `master_stats_hot_experiments` (table size — now bounded, rather than tracking the registry), `master_stats_experiments_retired_total`, `master_stats_retire_failures_total`. ## Next **C3** — UI becomes "recently active N + search", with cold experiments resolved through the registry and observed on demand. That's what makes retirement invisible to users: today a retired experiment would simply vanish from `/experiments`, which is only acceptable once search covers it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com>

#211) ⚠️ **Stacked on #209 → #210.** Both of those commits are included here. Merge order: **#209 → #210 → this**. Once they land, this diff reduces to the UI commit alone. ## Why this exists #210 removes cold experiments from the stats table — that's what keeps a scan round proportional to *active* work. But on its own it also removes them from `GET /experiments`, which is only acceptable once they stay findable another way. **This PR is what makes retirement safe to turn on.** | | behaviour | |---|---| | **default list** | the stats table — experiments written recently enough not to have been retired | | **search** | stats table **plus the registry** for names it no longer holds; cold matches observed on demand | | **detail** | on a stats miss, resolve via registry and observe on demand rather than returning 404 | Retirement therefore removes an experiment from the *default view* and **never makes it unfindable**. The default is also the right one at scale on its own merits: a flat list of every experiment ever created is neither renderable nor interesting once there are tens of thousands. ## Two deliberate decisions **1. `observe_cold` does not write a stats row.** Reading about a retired experiment must not make it hot again — otherwise **browsing the UI would silently undo retirement**, and the table would creep back toward holding everything, which is exactly the state retirement exists to prevent. It also takes no `MasterState`, so it *structurally* cannot write one. A refactor that hands it one has to justify itself. There's a test asserting the no-rehydration property. **2. Search bounds its on-demand observations by page size.** A query matching thousands of cold experiments must not open thousands of datasets to render one page. ## Known gap — stating it rather than hiding it The two handlers changed here live in `routes.rs`, whose tests **all require etcd and are `#[ignore]`d**, so CI does not cover them. `observe_cold` is tested in `scanner.rs`, which CI does run. So the registry-fallback logic in the handlers is verified locally but not by CI. That's a pre-existing gap in the test structure, worth fixing as its own change rather than as a fourth concern here — and it's the kind of blind spot that only became visible after #202 made CI actually run integration tests. ## Testing Full workspace: 178 core + 30 master + 51 server + 5 concurrency + 1 merge-cleanup, all pass. clippy `-D warnings` and fmt clean. ## The complete picture This finishes the line of work from the original report (`/experiments` at 30–43s): | | problem | fix | |---|---|---| | **#207** | write amplification — 100 versions/round | one snapshot commit; **84× fewer versions** | | **#209** | scan cost — full observe of everything, every round | skip unchanged; cold = one open | | **#210** | table size — held every experiment ever created | retire after 7 days idle, drained first | | **this** | retirement made cold experiments invisible | active-N default + registry-backed search | | **#208** | all of the above failing silently | maintenance failure metrics | 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com>

This was referenced Jul 27, 2026

feat(master): retire experiments idle for a week from the stats table #210

Merged

feat(master): serve the experiment list as recently-active plus search #211

Merged

beinan force-pushed the feat/stats-hot-cold branch 2 times, most recently from 647790f to 1a53473 Compare July 27, 2026 17:27

beinan force-pushed the feat/stats-hot-cold branch from 1a53473 to 9060a37 Compare July 27, 2026 17:30

beinan merged commit 5ce5e54 into lance-format:main Jul 27, 2026
9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

perf(master): skip re-observing experiments whose base version has not moved - #209

perf(master): skip re-observing experiments whose base version has not moved#209
beinan merged 1 commit into
lance-format:mainfrom
beinan:feat/stats-hot-cold

beinan commented Jul 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

beinan commented Jul 27, 2026

The problem

The fix

Two edges, both tested

Observability

Testing

Next

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant