refactor(core): migrate ContextStore onto StorageBase

[beinan]

Second step of #214, after #215. ContextStore now delegates its storage layer to StorageBase instead of carrying its own forked implementation.

Bugs this closes

WAL merge, which this store never had. Flushed generations accumulated under _mem_wal/ forever and every read unioned all of them, so read cost grew without bound in the number of writes. This was the single biggest gap between the three stores. Now exposed as maybe_merge_wal (count trigger), cleanup_wal (time trigger) and pending_wal_generations — the same machinery rollout uses.

Compaction dropped storage_options. Both compact and create_id_index reloaded via a bare Dataset::open(uri), silently discarding credentials — broken on any authenticated object store. Both now reload through StorageBase::load_with_options.

Compaction didn't defer index remap. The base table carries a fieldless MemWAL index and Lance's inline remap panics on it ("An index existed with no fields").

A writer per add. write_entries opened, put and closed a ShardWriter on every call, paying a cold DNS resolution + TCP/TLS handshake + epoch claim each time. Now one resident writer with fence-retry.

add is &self. The writer lives behind the base's mutex, so the server takes a read lock and concurrent appends stop serializing on the store's RwLock.

add visibility is preserved, not silently weakened

This was the one real design fork, and it's worth being explicit since the two stores genuinely differed:

• ContextStore.add did put + close, and close seals — so rows were visible on return, and validate_unique_ids, upsert_by_external_id and the tombstone path all depend on that read-your-write.
• RolloutStore.add deliberately gave visibility up: put only, with the flush sweeper (30s) or ?flush=true providing it.

Rather than pick one and break the other, StorageBase gains a seal_on_put switch. Rollout keeps false; ContextStore defaults to seal_on_add: true, preserving today's behavior and its correctness. Callers appending trusted batches that don't use the read-back paths can opt into rollout's throughput profile with seal_on_add: false.

Sharding now matches rollout

Writes go to the shard owned by this writer instance (ContextStoreOptions::shard_id) rather than one derived per (bot_id, session_id). One instance owns exactly one shard, so the epoch-fencing invariant holds by construction and merge has a single well-defined shard to drain. derive_region_id is removed.

This is a data-layout change: existing rows stay readable (the read path scans all shards from the manifests), but new writes land in a different shard.

ContextStore is no longer Clone

It owns a StorageBase, which owns the resident writer and a Drop that seals it — so it must have exactly one owner, or two writers race for one shard. Both clone sites now open their own handle:

• the background compactor, which only rewrites base-table fragments and would otherwise contend with the write path
• Python's Context.fork, which branches the in-memory context over the same dataset

The compactor's reopen needed one bit of care: open_inner can spawn the compaction task, which reopens via open_inner — a mutual recursion between two async fns that the compiler cannot prove Send, even though it's unreachable at runtime. Broken with a boxed dyn Future in a free function (open_for_compaction), with a comment explaining why it can't just be a flag.

Verification

Three new tests pin the behavior that was at risk:

• add_is_visible_on_return_by_default — the read-your-write contract the uniqueness/upsert/tombstone paths rely on
• deferred_seal_makes_add_durable_but_invisible_until_flush — mirrors rollout's equivalent test
• wal_generations_merge_into_the_base_table — generations are reclaimed and rows stay readable

Full workspace suite passes (180 core lib tests, up from 177), cargo clippy --workspace --all-targets clean.

⚠️ Not verified: the Python binding tests. maturin isn't available in this environment, so Context.fork's signature change is compile-checked only. Worth running python/tests/test_context.py and test_compaction.py before merge.

Follow-ups

DatagenStore is the last store to migrate (it has no compaction at all today), then the arbitrary-schema layer from Part 2 of #214.

🤖 Generated with Claude Code

[beinan]

Python bindings verified — installed maturin via uv and ran the suite:

$ uv run maturin develop
🛠 Installed lance-context-0.6.4

$ uv run pytest tests/ -q
182 passed, 2 skipped in 40.09s

That covers the two things that were compile-checked only: Context.fork (now opens its own handle instead of cloning) in test_context.py, and background compaction (now opens its own handle per tick) in test_compaction.py — 13 passed across those two files.

The ⚠️ caveat in the PR description is resolved; no further manual verification needed before merge.