Skip to content

test: run examples end to end against the live API - #374

Open
jbeckwith-oai wants to merge 3 commits into
mainfrom
codex/examples-e2e-coverage
Open

test: run examples end to end against the live API#374
jbeckwith-oai wants to merge 3 commits into
mainfrom
codex/examples-e2e-coverage

Conversation

@jbeckwith-oai

Copy link
Copy Markdown
Contributor

Summary

  • add a workflow_dispatch-only suite that runs 22 Ruby examples against the live OpenAI API
  • collect and upload SimpleCov HTML/JSON reports, with a checked-in 89.96% Ruby 4.0 line-coverage floor
  • classify every example as covered or explicitly excluded, and validate expected output
  • update image examples and README usage to gpt-image-2

Fixes uncovered by the live suite

  • accept null finish_reason values in intermediate legacy completion stream chunks, including RBI/RBS updates and a regression test
  • use response_id when resuming response streams

Impact

The live suite is opt-in and only runs through manual workflow dispatch. A dispatched run fails if an example fails, the inventory becomes incomplete, or SDK line coverage falls below the current baseline.

Validation

  • 22/22 runnable examples passed against the live API on Ruby 4.0
  • SimpleCov: 89.96% (28,000/31,122 lines)
  • 629 tests, 2,356 assertions
  • 2,617 files passed RuboCop
  • Sorbet and RBS validation passed
  • Ruby 4.0 coverage-regression test verifies a below-baseline run exits non-zero while retaining its JSON report

Comment thread scripts/examples-e2e.rb Fixed
@jbeckwith-oai
jbeckwith-oai marked this pull request as ready for review August 12, 2026 20:03
@jbeckwith-oai
jbeckwith-oai requested a review from a team as a code owner August 12, 2026 20:03
@openai-sdks

openai-sdks Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

OkTest Summary

237/237 SDK tests passed in 12.08s for Ruby SDK PR #374.

Test results — 42 files
Test Result Time
tests/chat-completions-complex-body.test.ts ✅ Passed 399ms
tests/chat-completions-create.test.ts ✅ Passed 426ms
tests/chat-completions-stream.test.ts ✅ Passed 208ms
tests/files-content-binary.test.ts ✅ Passed 357ms
tests/files-create-multipart.test.ts ✅ Passed 471ms
tests/files-list-pagination.test.ts ✅ Passed 226ms
tests/initialize-config.test.ts ✅ Passed 428ms
tests/instance-isolation.test.ts ✅ Passed 321ms
tests/models-list.test.ts ✅ Passed 414ms
tests/responses-background-lifecycle.test.ts ✅ Passed 339ms
tests/responses-body-method-errors.test.ts ✅ Passed 603ms
tests/responses-cancel-timeout.test.ts ✅ Passed 279ms
tests/responses-cancel.test.ts ✅ Passed 363ms
tests/responses-compact-retries.test.ts ✅ Passed 441ms
tests/responses-compact.test.ts ✅ Passed 388ms
tests/responses-create-advanced-stream.test.ts ✅ Passed 208ms
tests/responses-create-advanced.test.ts ✅ Passed 295ms
tests/responses-create-disconnect.test.ts ✅ Passed 416ms
tests/responses-create-errors.test.ts ✅ Passed 431ms
tests/responses-create-malformed-api-responses.test.ts ✅ Passed 372ms
tests/responses-create-retries.test.ts ✅ Passed 550ms
tests/responses-create-stream-failures.test.ts ✅ Passed 225ms
tests/responses-create-stream-timeout.test.ts ✅ Passed 233ms
tests/responses-create-stream-wire.test.ts ✅ Passed 2.329s
tests/responses-create-stream.test.ts ✅ Passed 159ms
tests/responses-create-terminal-states.test.ts ✅ Passed 384ms
tests/responses-create-timeout.test.ts ✅ Passed 329ms
tests/responses-create.test.ts ✅ Passed 381ms
tests/responses-delete.test.ts ✅ Passed 318ms
tests/responses-input-items-errors.test.ts ✅ Passed 295ms
tests/responses-input-items-list.test.ts ✅ Passed 221ms
tests/responses-input-items-options.test.ts ✅ Passed 278ms
tests/responses-input-tokens-count-timeout.test.ts ✅ Passed 396ms
tests/responses-input-tokens-count.test.ts ✅ Passed 400ms
tests/responses-malformed-inputs.test.ts ✅ Passed 2.563s
tests/responses-not-found-errors.test.ts ✅ Passed 486ms
tests/responses-parse.test.ts ✅ Passed 239ms
tests/responses-retrieve-retries.test.ts ✅ Passed 394ms
tests/responses-retrieve.test.ts ✅ Passed 355ms
tests/responses-stored-method-errors.test.ts ✅ Passed 1.177s
tests/retry-behavior.test.ts ✅ Passed 2.12s
tests/sdk-error-shape.test.ts ✅ Passed 569ms

View OkTest run #31635745495

SDK merge (bc85b79a583a) · head (2234eda431f7) · base (919977919667) · OkTest (a845fa206fa4)

@jbeckwith-oai jbeckwith-oai added the generator Touches generated SDK files label Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

generator Touches generated SDK files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants