Recording SDK traffic and replaying it with the API unreachable — a no-code-change alternative to VCR-style fixtures

[xizhuomengcontin]

Sharing a testing approach that has been useful to me and needs no changes to code that uses this SDK.

Disclosure: I work on the tool (OrcaReplay, Apache-2.0, no paid tier).

The idea

Instead of hand-writing response fixtures or monkeypatching the client, record the real HTTP exchange once and then serve those exact bytes back on later runs. Because the SDK reads OPENAI_BASE_URL from the environment, the capture needs no code change at all — no custom http_client, no base_url= in your source, no mock:

client = OpenAI()           # unchanged
client.chat.completions.create(model=..., messages=[...])

$ orca record generic-openai -- python your_script.py
info recorded run=run_… events=6 exit=0

$ orca replay last
info replaying exchanges=1 egress=blocked
info replay.done reused=1/1 exact=1 divergences=0 unmatched=0 exit=0

The second run reaches the API never. Verbatim bytes go back, including streaming frames, so the SDK's own parsing is exercised rather than skipped — which is the part a hand-written fixture usually gets subtly wrong.

Both clients are covered, and the async one on purpose

My integration matrix has separate checks for OpenAI and AsyncOpenAI, because the async client is what the OpenAI Agents SDK builds on and I did not want to infer one from the other. Each check records against a stub origin, kills the origin, replays, and asserts the exchange count — so "this still works with the current SDK" fails a build rather than quietly ageing.

I also drove it end to end against a local OpenAI-compatible server: record, kill the server, replay clean.

Where this is and is not the right tool

Is: re-running a script or a test suite deterministically for free; keeping the SDK's real deserialisation in the loop; debugging one bad response repeatedly without paying for it each time.

Is not: a mocking library. You cannot author a response by hand — you record a real one first. If you need to synthesise a case the API never produced, respx/unittest.mock remain the right answer.

One scope note: egress=blocked on replay means model-provider egress. Anything else the program does still happens for real. It is not a sandbox.

Happy to answer questions, and interested in whether people would rather have this as a pytest fixture than a CLI wrapper.