Skip to content

test: add live end-to-end validation harness - #75

Open
JNK234 wants to merge 1 commit into
feat/structured-outputs-22from
test/e2e-validation
Open

test: add live end-to-end validation harness#75
JNK234 wants to merge 1 commit into
feat/structured-outputs-22from
test/e2e-validation

Conversation

@JNK234

@JNK234 JNK234 commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #74. Targets feat/structured-outputs-22 because T6 exercises llm:chat-with-schema, llm:chat-json and llm:get. Merge #74 first and this retargets to main cleanly.

Why

The unit suite is deterministic by design — it asserts request payloads without a live call, so CI needs no keys. That proves the plumbing, not that a provider accepts the request or that its response parses.

Today's review cycle showed the gap is not theoretical. Two findings derived carefully from vendor documentation were each settled in the opposite direction by one real call:

Claim from doc review What the live call showed
Groq defaults to reasoning_format: "raw", so <think> tags leak into visible content — rated HIGH False for gpt-oss-20b. Returns clean content plus a separate reasoning field.
reasoning_effort mismatch — rated MEDIUM True and worse. xhigh passed extension validation and returned HTTP 400.

What

demos/e2e-tests/ — six test procedures making real API calls through the installed extension, with a chooser for the provider under test.

Test Covers
T1 plain chat round trip
T2 thinking-tag leakage in visible content
T3 thinking enabled — reasoning arrives separately, not duplicated
T4 llm:compile-error (#52)
T5 per-agent history isolation and clear-history
T6 structured output (#22)

T6 asserts types, not just shape. A provider can return well-formed JSON and still fail if the conversion drops types, so is-number? and is-boolean? are checked alongside the enum constraint, and a missing key must raise rather than default silently.

Verification

22/22 pass against live Groq (openai/gpt-oss-20b), headless, real HTTP, no stubs:

T6 structured output
  -> [[action eat] [confidence 0.9] [alive true]]
  PASS  action honours the schema enum
  PASS  confidence is a NUMBER, not text
  PASS  boolean field is a boolean
  PASS  missing key raises rather than defaulting
  chat-json -> {"colours":["red","blue"]}
TOTAL  passed 22   failed 0

config-ollama.txt needs no key, so the suite runs offline against a local model too.

Keys

config-groq.txt and config-anthropic.txt are gitignored and ship as .example templates. Only config-ollama.txt, which needs no key, is tracked.

Headless finding worth recording

NetLogo 7.0.3 only runs an experiment embedded in the .nlogox. An external --setup-file is accepted and then silently ignored, aborting with NoSuchElementException: head of empty list before setup runs — indistinguishable from the benign 0-step abort. The embedded block additionally requires sequentialRunOrder, timeLimit as an attribute rather than a child node, and metrics wrapped in <metrics>.

This cost real debugging time and is not documented anywhere in the repo.

Test procedures that make real API calls through the installed extension,
covering behaviour that unit tests cannot reach because it depends on what
a provider actually returns.

- T1 basic chat round trip
- T2 thinking-tag leakage: a plain llm:chat never sends reasoning_format,
  so a provider defaulting to "raw" would return reasoning inline in
  <think> tags with content copied verbatim. Only a live call settles this.
- T3 thinking enabled: reasoning must arrive separately, not duplicated
- T4 llm:compile-error
- T5 per-agent history isolation and clear-history
- T6 structured output: asserts fields arrive as NetLogo TYPES, not text —
  a provider can return well-formed JSON and still fail if the conversion
  drops types, so is-number?/is-boolean? are checked alongside the enum
  constraint, and a missing key must raise rather than default

Configs ship for ollama (no key) plus groq and anthropic as .example
templates; the key-bearing copies are gitignored.

Verified: 22/22 pass against live Groq (openai/gpt-oss-20b) via headless
BehaviorSpace, real HTTP, no stubs.

Headless note: NetLogo 7.0.3 only runs an experiment embedded in the
.nlogox. An external --setup-file is accepted and then silently ignored,
aborting with "head of empty list" before setup runs. The embedded block
requires sequentialRunOrder, timeLimit as an attribute, and metrics
wrapped in <metrics>.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant