ContextStore shards MemWAL by data, not by writer — multi-replica writes contend and can hang

[beinan]

Summary

ContextStore derives its MemWAL shard from the record's (bot_id, session_id) (derive_region_id, store.rs), not from writer identity. Two processes writing the same session therefore target the same shard and contend for its writer epoch. RolloutStore avoids this by sharding on writer identity (shard_id from the instance hostname), documented as "each instance owns exactly one shard and no two instances ever contend". ContextStore has no equivalent.

Not reachable in a single process today: AppState::get_or_open_context_store caches one Arc<RwLock<ContextStore>> per name and every mutating route holds the write lock. This is about multiple server replicas writing one context store.

Why retrying does not fix it

I tried the obvious thing (#201, now closed) — bounded retry around the claim — and it is actively worse:

ShardWriter::wait_for_flush_drain (lance 7.0.0 write.rs:1832) is an unbounded loop waiting on flush watchers. A writer fenced while inside close() waits on a watcher nobody will complete. Each retry opens a new writer, which claims the epoch, which fences the previous writer possibly still in close(). Lance also retries the claim 16x internally (manifest.rs:423), so a 5-attempt outer retry nests into up to 80 contending claims.

Measured: 2 of 5 runs hang indefinitely with two writers on one session; 0 of 3 with distinct sessions. In CI the hang ran 19 minutes until the job timeout.

The claim is not a lock you can politely re-take — reopening fences whoever is mid-close(). So retry is the wrong layer.

Two failure modes if a second writer ever appears

1. initialize_mem_wal race. It is a CreateIndex transaction; on a fresh dataset both writers see has_mem_wal == false, both try to create it, and the loser fails the whole append with "Retryable commit conflict ... preempted by concurrent transaction CreateIndex".
2. Shard claim race. add opens a fresh mem_wal_writer per group per call, and each open claims the epoch, so this recurs in steady state, not just at startup. Message: "Failed to claim shard ... another writer claimed epoch N" — note it contains neither "fenced" nor "Fenced", so the rollout store's is_fenced_error would not match it.

Suggested fix

Shard by writer identity, as RolloutStore does:

• add a writer_id / shard_id to ContextStoreOptions, defaulting to the instance hostname;
• derive the MemWAL shard from that rather than from (bot_id, session_id);
• keep (bot_id, session_id) as a grouping key for batching only.

Then two replicas never target one shard, no claim ever contends, and neither failure mode is reachable. initialize_mem_wal still wants a one-time guard, but with per-writer shards it stops being on the hot path.

Scope check before doing any of this

There is no server deployment manifest in this repo (deploy/kubernetes/ contains only master.yaml, and master does not write context stores). If a context store is never written by more than one replica, this is a documentation task — state the single-writer requirement on ContextStore — rather than a code change.

Worth deciding that first.