Roadmap priority: B6
Summary
Record every prompt/response pair in a run, and replay them offline for an identical re-run without API calls.
Why it matters
Three concrete uses:
- Reproducibility — an LLM model is not currently reproducible even with a fixed random seed. Replay makes a published run exactly repeatable.
- Demos without keys — present a model at a workshop or conference with no network and no API key.
- Classroom — students explore a recorded run at zero cost.
mesa-llm has an equivalent (@record_model decorator, SimulationRecorder, plus a CLI replay viewer), and it is one of the genuinely good ideas in their codebase.
What needs to be done
- Recording: commands like
llm:start-recording <file> / llm:stop-recording, capturing prompt, response, model, provider, timestamp, and agent identity.
- Replay:
llm:load-recording <file> puts the extension into a mode where calls are served from the recording instead of the network.
- Decide the matching rule on replay — strict call-order sequence, or content-addressed lookup (which converges with caching, B5).
- Behaviour on a miss: error, fall through to a live call, or return a placeholder. Must be explicit.
- File format — JSONL is the obvious fit and is diffable.
- Interaction with
clear-all and multiple agents; recordings must survive being shared alongside a .nlogox.
Open questions
- Should recording be automatic under BehaviorSpace, given B7 (experiment logging) wants much the same data?
- Does replay need to reproduce timing/latency, or return instantly? Instant is more useful; timing matters only for demo realism.
Roadmap priority: B6
Summary
Record every prompt/response pair in a run, and replay them offline for an identical re-run without API calls.
Why it matters
Three concrete uses:
mesa-llm has an equivalent (
@record_modeldecorator,SimulationRecorder, plus a CLI replay viewer), and it is one of the genuinely good ideas in their codebase.What needs to be done
llm:start-recording <file>/llm:stop-recording, capturing prompt, response, model, provider, timestamp, and agent identity.llm:load-recording <file>puts the extension into a mode where calls are served from the recording instead of the network.clear-alland multiple agents; recordings must survive being shared alongside a.nlogox.Open questions