Skip to content

[test] Exercise the documented README and Getting Started server flows in CI #401

Description

@ting-hong-shieh

Problem

The README and Getting Started workflows describe their checks as executable coverage that catches malformed TOML, CLI drift, and endpoint regressions. The tests they run currently read Markdown and assert that a few strings are present and ordered; they do not parse the documented TOML, invoke switchyard-server --dry-run, start the server, or call an endpoint.

For example, replacing the Getting Started example's valid base_url with the syntactically invalid line below still satisfies every assertion in tests/getting_started/test_getting_started.py:

base_url = [

This is the exact malformed-config regression that .github/workflows/getting-started.yml says the workflow catches.

The same gap exists between .github/workflows/readme.yml and tests/readme/test_readme.py.

Proposed solution

Make the two flows hermetic and executable:

  1. Extract or maintain the documented TOML as a test fixture.
  2. Point its target at a local stub upstream, with no provider credentials or network access.
  3. Run switchyard-server --dry-run against the fixture.
  4. Start the server on an ephemeral loopback port.
  5. Verify /health, /v1/models, and one documented completion request.

crates/switchyard-server/tests/server.rs already contains a local Axum upstream pattern that could be reused. tests/getting_started/conftest.py also has a TODO for a credential-free local classifier/upstream fixture.

Alternatives considered

  • Keep the current substring checks: useful for catching renamed commands, but they cannot validate TOML syntax or runtime behavior. They can remain as small drift checks alongside the executable smoke test.
  • Weaken the workflow comments and job names: this would make the claims accurate but would leave the primary onboarding flows vulnerable to runtime drift.
  • Use a live provider: unnecessary and unsuitable for CI; a local deterministic upstream covers the documented contract without credentials or cost.

Scope notes

  • Test and workflow changes only; no new public API.
  • No live-provider calls or secrets.
  • Likely larger than a small documentation edit, so I am opening an issue before implementation per the contribution guide.

Additional context

Relevant files:

  • .github/workflows/readme.yml
  • .github/workflows/getting-started.yml
  • tests/readme/test_readme.py
  • tests/getting_started/test_getting_started.py
  • tests/getting_started/conftest.py

I searched open and closed issues and pull requests for README executable coverage, Getting Started credentials test, local classifier/upstream fixture, README test only checks strings, quickstart local fixture, and TOML schema coverage; I found no matching report or in-flight fix.

I'm happy to work on the local fixture and smoke coverage if this scope looks right.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions