Problem
The README and Getting Started workflows describe their checks as executable coverage that catches malformed TOML, CLI drift, and endpoint regressions. The tests they run currently read Markdown and assert that a few strings are present and ordered; they do not parse the documented TOML, invoke switchyard-server --dry-run, start the server, or call an endpoint.
For example, replacing the Getting Started example's valid base_url with the syntactically invalid line below still satisfies every assertion in tests/getting_started/test_getting_started.py:
This is the exact malformed-config regression that .github/workflows/getting-started.yml says the workflow catches.
The same gap exists between .github/workflows/readme.yml and tests/readme/test_readme.py.
Proposed solution
Make the two flows hermetic and executable:
- Extract or maintain the documented TOML as a test fixture.
- Point its target at a local stub upstream, with no provider credentials or network access.
- Run
switchyard-server --dry-run against the fixture.
- Start the server on an ephemeral loopback port.
- Verify
/health, /v1/models, and one documented completion request.
crates/switchyard-server/tests/server.rs already contains a local Axum upstream pattern that could be reused. tests/getting_started/conftest.py also has a TODO for a credential-free local classifier/upstream fixture.
Alternatives considered
- Keep the current substring checks: useful for catching renamed commands, but they cannot validate TOML syntax or runtime behavior. They can remain as small drift checks alongside the executable smoke test.
- Weaken the workflow comments and job names: this would make the claims accurate but would leave the primary onboarding flows vulnerable to runtime drift.
- Use a live provider: unnecessary and unsuitable for CI; a local deterministic upstream covers the documented contract without credentials or cost.
Scope notes
- Test and workflow changes only; no new public API.
- No live-provider calls or secrets.
- Likely larger than a small documentation edit, so I am opening an issue before implementation per the contribution guide.
Additional context
Relevant files:
.github/workflows/readme.yml
.github/workflows/getting-started.yml
tests/readme/test_readme.py
tests/getting_started/test_getting_started.py
tests/getting_started/conftest.py
I searched open and closed issues and pull requests for README executable coverage, Getting Started credentials test, local classifier/upstream fixture, README test only checks strings, quickstart local fixture, and TOML schema coverage; I found no matching report or in-flight fix.
I'm happy to work on the local fixture and smoke coverage if this scope looks right.
Problem
The README and Getting Started workflows describe their checks as executable coverage that catches malformed TOML, CLI drift, and endpoint regressions. The tests they run currently read Markdown and assert that a few strings are present and ordered; they do not parse the documented TOML, invoke
switchyard-server --dry-run, start the server, or call an endpoint.For example, replacing the Getting Started example's valid
base_urlwith the syntactically invalid line below still satisfies every assertion intests/getting_started/test_getting_started.py:base_url = [This is the exact malformed-config regression that
.github/workflows/getting-started.ymlsays the workflow catches.The same gap exists between
.github/workflows/readme.ymlandtests/readme/test_readme.py.Proposed solution
Make the two flows hermetic and executable:
switchyard-server --dry-runagainst the fixture./health,/v1/models, and one documented completion request.crates/switchyard-server/tests/server.rsalready contains a local Axum upstream pattern that could be reused.tests/getting_started/conftest.pyalso has a TODO for a credential-free local classifier/upstream fixture.Alternatives considered
Scope notes
Additional context
Relevant files:
.github/workflows/readme.yml.github/workflows/getting-started.ymltests/readme/test_readme.pytests/getting_started/test_getting_started.pytests/getting_started/conftest.pyI searched open and closed issues and pull requests for
README executable coverage,Getting Started credentials test,local classifier/upstream fixture,README test only checks strings,quickstart local fixture, andTOML schema coverage; I found no matching report or in-flight fix.I'm happy to work on the local fixture and smoke coverage if this scope looks right.