Skip to content

feat: record and replay mode for reproducible runs #55

Description

@JNK234

Roadmap priority: B6

Summary

Record every prompt/response pair in a run, and replay them offline for an identical re-run without API calls.

Why it matters

Three concrete uses:

  • Reproducibility — an LLM model is not currently reproducible even with a fixed random seed. Replay makes a published run exactly repeatable.
  • Demos without keys — present a model at a workshop or conference with no network and no API key.
  • Classroom — students explore a recorded run at zero cost.

mesa-llm has an equivalent (@record_model decorator, SimulationRecorder, plus a CLI replay viewer), and it is one of the genuinely good ideas in their codebase.

What needs to be done

  1. Recording: commands like llm:start-recording <file> / llm:stop-recording, capturing prompt, response, model, provider, timestamp, and agent identity.
  2. Replay: llm:load-recording <file> puts the extension into a mode where calls are served from the recording instead of the network.
  3. Decide the matching rule on replay — strict call-order sequence, or content-addressed lookup (which converges with caching, B5).
  4. Behaviour on a miss: error, fall through to a live call, or return a placeholder. Must be explicit.
  5. File format — JSONL is the obvious fit and is diffable.
  6. Interaction with clear-all and multiple agents; recordings must survive being shared alongside a .nlogox.

Open questions

  • Should recording be automatic under BehaviorSpace, given B7 (experiment logging) wants much the same data?
  • Does replay need to reproduce timing/latency, or return instantly? Instant is more useful; timing matters only for demo realism.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions