OpenAPI-generated Python API client plan

[AbhiPrasad]

Tracking issue for generating the public Braintrust REST API client from
braintrust-openapi and exposing it through
braintrust.api without changing existing authentication, routing, retries, or high-level SDK behavior.

This supersedes #673. Keep #673 as the detailed runtime/call-site audit and #682 as the reference for
the Experiments behavior and tests. This issue is the implementation source of truth.

Status

• Foundation: transport (ref(api): establish transport boundary for API client #641), policy-aware errors/retries (feat(api): add policy-aware HTTP transport #653), client/service facade (feat(api): add client bootstrap and service facade #655)
• Step 0: generator measurement spike
• Step 1: pinned spec, validation, generated models, drift CI
• Step 2: Projects vertical slice and endpoint-generator decision
• Step 3: Experiments vertical slice and Experiment.summarize() silently discards scores/metrics on any fetch failure instead of surfacing or retrying #639
• Step 4: all remaining supported endpoint bindings
• Step 5: public facade and REST type documentation
• Step 6: migrate SDK internals, one resource per PR
• Step 7: automate spec-update PRs

Next: Step 1.

Goal

Generate a synchronous, typed REST client from a pinned OpenAPI spec while reusing the SDK's existing
requests transport, auth, organization selection, endpoint routing, retry policies, and error types.
Generated source is committed; install, sdist, and wheel builds remain offline and do not run codegen.

Out of scope

• Replacing braintrust.generated_types or its backend generator.
• Async APIs, Pydantic, httpx, or another HTTP pool.
• Proxy/streaming, log ingestion, attachments/object storage, cache fallback, or other multi-step
  workflows already handled by specialized SDK paths.
• Observable changes to existing high-level braintrust APIs, except the explicit Experiment.summarize() silently discards scores/metrics on any fetch failure instead of surfacing or retrying #639 fix in Step 3.

Settled design

1. BraintrustClient remains the only public constructor and lifecycle owner.
2. Keep the hand-written shell: client.py, _transport.py, _routing.py, _service.py, auth,
   errors, retry policies, public resource wrappers, and specialized workflows.
3. Generate all REST models privately as dependency-free TypedDicts/type aliases. Export only types
   reachable from documented public methods through braintrust.api.types; never re-export them from
   braintrust.
4. Spec operationIds are private metadata. Public methods use ergonomic resource names, with explicit
   overrides for exceptions.
5. Generate sync only. A future async client requires a native async transport and separate design.
6. Add no runtime dependency beyond the standard library, typing_extensions, and existing
   braintrust.api modules.
7. Commit openapi/spec.json, a full spec commit SHA, and its SHA-256. Normal generation is hermetic;
   BRAINTRUST_OPENAPI_ROOT is the local-checkout override.
8. Exclude the Proxy tag for this project. Keep the exclusion explicit and coverage-tested; revisit
   only in a separate proposal after upstream fixes proxy{path+} and if generation benefits streaming.
9. Responses use generated mappings, not a parallel dataclass/raw convention. Unknown/additive keys
   must survive.
10. braintrust.api._generated is private. Only braintrust.api.__all__, documented wrappers, and
    braintrust.api.types.__all__ define compatibility commitments.

Required shape

openapi/
├── README.md
├── config.json                 # spec/tool pins, hashes, skip list, overrides
└── spec.json                   # committed snapshot
scripts/
├── fetch-openapi-spec.py
├── generate-api-client.py
└── check-api-client-codegen.py
py/src/braintrust/api/
├── <hand-written shell and resource wrappers>
├── types/__init__.py           # deliberate public REST exports
└── _generated/
    ├── __init__.py
    ├── models.py
    ├── operations.py
    └── resources/<tag>.py

Every generated package directory needs __init__.py; setuptools currently uses regular package
discovery. Wheel tests must import the generated package from an installed wheel.

Root scripts resolve paths relative to themselves. py/Makefile remains the SDK entry point:

cd py
make generate-api-client
make check-api-client-codegen

generate-api-client writes to a temporary directory, emits final ruff-formatted output, and
atomically replaces only generated files. check-api-client-codegen regenerates and diffs without
dirtying the worktree.

Generator contract

Inputs and determinism

Pin in openapi/config.json:

• full braintrust-openapi commit SHA and spec SHA-256;
• datamodel-code-generator==0.72.4;
• the ruff version, asserted equal to .pre-commit-config.yaml;
• generator Python version (recorded, not enforced); and
• endpoint-generator schema/version, skip tags, naming overrides, and runtime-policy overrides.

Generated headers contain the spec SHA, spec hash, generator versions, and a content hash. They must
not contain timestamps, hostnames, or absolute paths. Mark py/src/braintrust/api/_generated/** as
generated in .gitattributes and exclude it from pre-commit; the generator owns formatting and syntax.

Model flags proven by Step 0:

--input-file-type openapi
--output-model-type typing.TypedDict
--target-python-version 3.10
--use-union-operator
--enum-field-as-literal all
--use-generic-container-types
--use-field-description
--strict-nullable
--parent-scoped-naming
--no-use-closed-typed-dict
--disable-future-imports
--formatters ruff-format
--custom-file-header "..."

Do not copy backend cleanup that collapses “optional” into “nullable”; request types must preserve the
OpenAPI 3.0 distinction between missing fields and nullable: true.

Spec validation

Before generation, fail with an actionable error unless all of these hold:

• supported operations have unique, valid operationIds and usable tags;
• schema and parameter $refs resolve, including referenced parameters;
• inline names do not collide;
• every non-OPTIONS operation is generated or matches an explicit skip entry with a reason;
• the skip set matches exactly (no silent widening);
• supported success responses and request bodies use supported status/media types; error bodies may
  remain opaque text/plain;
• path templates map to declared scalar parameters; and
• naming/policy override keys reference real operations.

Normalize only by removing CORS OPTIONS and explicitly skipped operations. Report operation/schema
counts; do not hard-code them. Fix spec defects upstream instead of accumulating Python patches.

Generated endpoints and runtime adapter

Each operation records method, relative path, tag, typed path/query/header/body parameters,
style/explode rules, success status/content type, response type, RequestTarget, RetryMode, and any
timeout/budget override. Thin generated bindings delegate serialization and HTTP behavior to one
hand-written adapter built on ResourceAPI.

The adapter must preserve existing routing, auth, injected transports, custom deployments, typed
errors/request IDs, and opaque non-2xx bodies. Ignore the spec's absolute servers URL. Never send
Braintrust credentials to signed object-storage URLs.

Runtime policy guardrails

OpenAPI does not encode replay safety or product fallback behavior. Generate resolved metadata from
conservative defaults plus explicit overrides:

Operation	Default
GET / HEAD	SAFE_READ
writes	NONE
POST logical reads	explicit SAFE_READ override
verified idempotent writes	explicit IDEMPOTENT_WRITE override
ingestion	existing specialized path / LOG_INGESTION; never a second retry loop

Every generated operation must have an explicit resolved mode; coverage fails on unclassified
operations. Keep payload-dependent or multi-call behavior hand-written, including conditional
registration, base-experiment HTTP 400 -> None, attachment reconciliation, invocation, signed
uploads, caching/fallback, and lenient summary behavior. The API layer reports errors; higher layers
own visibly marked fallback. #673 remains the detailed replay-safety audit.

Import and type-surface guardrails

logger.py loads braintrust.api during import braintrust. Generated modules therefore load lazily
through module-level __getattr__; generated resources import models under TYPE_CHECKING with quoted
annotations. A bare import braintrust must leave every braintrust.api._generated.* module absent
from sys.modules.

Keep the two type surfaces separate:

• braintrust.generated_types: SDK logging/eval payload types.
• braintrust.api.types: generated REST request/response types.

Add a test that inventories overlapping public names and makes shape changes review-visible.

Rollout

Each step is independently landable and revertible. Generated output and the infrastructure that
creates it should be separate commits. Do not widen a step while its exit criteria are unmet.

Step 0 — measurement spike ✅

Measured against the then-current public spec with datamodel-code-generator==0.72.4:

• 6,815 lines / ~200 KB; 392 TypedDicts + 114 aliases; 506 unique names, no collisions.
• Byte-identical repeated generation and byte-identical output on Python 3.10 vs 3.13.
• Warm import: 18.9 ms on 3.13, 13.9 ms on 3.10; cold import: 37.9 ms.
• --disable-future-imports is required by repository policy.
• --formatters ruff-format is required for output clean under ruff 0.15.21.
• One models.py is sufficient; lazy loading remains required.

Step 1 — pinned spec, validator, models, drift CI

Deliver:

• committed config/spec snapshot, fetch/local-override flow, validator, and deterministic model generator;
• private _generated/models.py, imported by nothing at runtime;
• Make targets, ruff-pin sync check, .gitattributes, and pre-commit exclusion;
• dedicated Ubuntu drift job outside the nox shard matrix and included in checks-passed;
• generator tests for determinism, hash/pin validation, filtering/skip exactness, collisions, invalid IDs,
  media types, $ref parameters, nullable-vs-missing, composition types, and JSON-compatible scalars;
• Python 3.10–3.14 import/type coverage, lazy-import guard, and installed-wheel content test.

Commit order: .gitattributes first; infrastructure/tests second; generated output last.

Exit: a clean checkout passes make check-api-client-codegen; import braintrust loads no generated
module; build/test do not access the network; installed wheel contains _generated.

Step 2 — Projects vertical slice

Implement the five current Projects operations end-to-end: registry metadata, runtime adapter,
private bindings, ergonomic ProjectsAPI wrapper, request/response types, VCR tests, exact-wire local
HTTP tests, and type tests.

Then record one decision in this issue:

• Keep generation if the five methods remain deterministic metadata/templates with no
  operation-specific executable branches.
• Hand-write bindings if supporting them requires endpoint-specific generator logic; retain
  generated models and coverage metadata.

Exit: all five operations pass VCR, local-wire, custom-router/transport, additive-field, and type tests.

Step 3 — Experiments vertical slice and #639

Do this separately from all-tag expansion because it changes product behavior and exercises off-spec
wrappers. Use #682 as reference, but return generated TypedDicts.

Required behavior:

• Experiments get, get_base, and compare are logical reads using SAFE_READ.
• get_base() translates HTTP 400 to None.
• additive response fields survive.
• Experiment.summarize() / evals are strict by default and raise typed API errors after retries.
• explicit lenient mode returns the discriminated SummarySuccess / SummarySkipped /
  SummaryFailed result; deprecated top-level score/metric bridges remain read-only.
• real-backend VCR tests cover summarize and a fresh experiment with no base; local HTTP tests assert
  targets and additive fields; type tests cover the public result.

Exit: #639 is fixed without changing unrelated high-level behavior.

Step 4 — remaining supported bindings

Generate all remaining non-CORS, non-Proxy operations, split privately by tag. Add spec-to-registry
operation coverage and policy-completeness tests. Keep specialized/off-spec workflows hand-written.

Exit: every in-scope operation is generated; every excluded operation has an exact tested reason; no
high-level SDK call site has migrated merely because a binding now exists.

Step 5 — public facade and REST types

Expose reviewed ergonomic resource methods and only their reachable types. Document preview/stability
policy and the distinction between braintrust.api.types and braintrust.generated_types. Presence in
the spec is not automatic publication.

Exit: braintrust.api.__all__, resource wrappers, type exports, docs, and type tests agree on the
supported surface.

Step 6 — migrate internals

Migrate one resource per PR, starting with Projects. Use red -> green and VCR-backed coverage. Remove
hand-written wire shaping only after equivalent generated coverage exists. Do not migrate specialized
invocation, attachment/storage, proxy/streaming, or log-ingestion flows into the generic path.

Exit per PR: behavior and fallback semantics are unchanged, except the already-landed #639 fix.

Step 7 — automated spec updates

After 3–4 manual pin bumps, add a scheduled/manual workflow that updates the SHA and snapshot,
regenerates, runs codegen/runtime/type tests, and opens a PR with operation/schema summaries and the
upstream spec diff. Never auto-merge generated API changes.

Agent execution checklist

For every implementation PR:

1. Work from py/; inspect py/noxfile.py, py/pyproject.toml,
   py/src/braintrust/integrations/versioning.py when relevant, and .github/workflows/checks.yaml.
2. Add the smallest failing test first. Provider response-shape behavior is VCR-first; record real
   cassettes rather than using mocks as the primary regression test.
3. Modify generator/template/runtime source, never generated output directly.
4. Run the narrowest test first, then the exact nox/type/wheel sessions affected by the change.
5. Run make check-api-client-codegen and verify git status stays clean after the check.
6. In the PR body, report generated operation/schema deltas, policy overrides, exclusions, test
   commands, and any measured import change.

Definition of done

• One offline command regenerates byte-identical committed output from a hash-verified full spec SHA.
• Every non-CORS operation has a private binding or an explicit tested exclusion.
• Every binding has deterministic route/retry metadata and uses the existing transport/error stack.
• import braintrust loads no generated module and has no material import regression.
• Builds never fetch or generate; installed wheels contain the generated packages.
• Public exports are deliberate, typed, documented, and separated from braintrust.generated_types.
• Auth, org selection, custom routing, transport injection, retries, and specialized workflows retain
  existing behavior.
• Representative real responses are VCR-covered; runtime, type, lint, and wheel checks pass on Python
  3.10–3.14.