feat(master): retire experiments idle for a week from the stats table

[beinan]

⚠️ Stacked on #209 — that PR's commit is included here, so review #209 first and merge it before this one. Once #209 lands, this diff reduces to the retirement commit alone.

Why

#209 made cold experiments cheap (one open instead of a full observation). They still cost something, and the table still holds a row for every experiment that has ever existed. At tens of thousands — the overwhelming majority written once and never again — that's the remaining reason a scan round can't stay proportional to actual work.

An experiment with no writes for STATS_COLD_RETIRE_SECS (default 7 days) is dropped from the table. It stays in the registry and can still be observed on demand; it just stops costing a row and an open every round.

Retirement order is a correctness property

This is the part worth reviewing carefully.

Both auto-sweeps read the stats table and nothing else (verified — the registry references in scheduler.rs are all test-only). So an experiment absent from it is invisible to them permanently.

Dropping one that still has un-merged MemWAL generations would strand them:

• rows stay readable (the read path unions generations) — so nothing looks broken
• but they're never folded into the base table
• read amplification never recovers
• _mem_wal/{shard}/ directories are never reclaimed
• and no process is left that would notice

So the order is:

merge WAL → compact → verify pending_wal_generations == 0 → drop the row

⚠️ compact_files deliberately leaves MemWAL generations alone, so compaction alone would not have been sufficient. Compaction is still done, because a retired table is one nothing will come back to tidy.

Any step failing leaves the experiment in the table for the next round. Refusing to retire is always safe; retiring early is not.

Testing

Four tests. The important one asserts the shard is genuinely drained after retirement — reopening the store and checking pending_wal_generations == 0 — rather than that the function returned success.

Verified non-vacuous: removing the WAL-merge step makes it fail with exactly the stranding it exists to prevent:

assertion failed: retirement must leave zero pending generations,
                  or they are stranded forever

Full workspace: 178 core + 28 master + 51 server, all pass. clippy -D warnings and fmt clean.

Config

STATS_COLD_RETIRE_SECS=0 disables retirement entirely. Every existing test config sets it to 0, so they exercise the unchanged path.

Metrics

master_stats_hot_experiments (table size — now bounded, rather than tracking the registry), master_stats_experiments_retired_total, master_stats_retire_failures_total.

Next

C3 — UI becomes "recently active N + search", with cold experiments resolved through the registry and observed on demand. That's what makes retirement invisible to users: today a retired experiment would simply vanish from /experiments, which is only acceptable once search covers it.

🤖 Generated with Claude Code