Skip to content

feat: add structured output support (schema-constrained and JSON chat) - #74

Open
JNK234 wants to merge 2 commits into
mainfrom
feat/structured-outputs-22
Open

feat: add structured output support (schema-constrained and JSON chat)#74
JNK234 wants to merge 2 commits into
mainfrom
feat/structured-outputs-22

Conversation

@JNK234

@JNK234 JNK234 commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Closes #22. Roadmap B-series — the largest open capability gap, and the shape Jacob sketched at the 29 Jul meeting.

Why

Every primitive returned free text with no structural guarantee. A model that needs a number had to hope the phrasing stayed stable and pick it out of a sentence:

let reply llm:chat "How confident are you, 0 to 1?"
;=> "I'd say about 0.8, though it depends on the situation."
;   substring? position? what if it says "eighty percent"?

That works until the model words things differently, and then it fails quietly.

What

Three primitives:

Primitive Returns
llm:chat-with-schema prompt schema reply constrained to a JSON Schema, parsed into nested NetLogo lists
llm:chat-json prompt valid JSON with no schema, as a raw string
llm:get parsed key the value for a key in a [[key value] ...] list
let schema "{\"type\":\"object\",\"properties\":{\"action\":{\"type\":\"string\",\"enum\":[\"eat\",\"explore\",\"rest\"]},\"confidence\":{\"type\":\"number\"}},\"required\":[\"action\",\"confidence\"]}"

let reply llm:chat-with-schema "A turtle sees food. What now?" schema
;=> [[action eat] [confidence 0.9]]

llm:get reply "action"      ;=> "eat"   constrained to the enum
llm:get reply "confidence"  ;=> 0.9     a NUMBER

Native response formats are sent to OpenAI-compatible, Claude, Gemini, and Ollama providers. llm:choose now sends its option list as a schema constraint where the provider supports it, and keeps the existing fallback where it does not.

How JSON maps into NetLogo

NetLogo has no dictionary type, so objects become [key value] pair lists — the association-list shape modelers already use.

JSON NetLogo
{"a": 1} [[a 1]]
[1, 2] [1 2]
"text" string
10 number
true boolean
null "" — NetLogo has no null

Nested objects are more pair lists, so llm:get chains rather than taking a dotted path.

Demo

demos/structured-outputs/ — six turtles forage, each asking the model what to do with the reply schema-constrained.

Two consequences are visible on screen rather than only described:

  • action is enum-constrained, so act-on compares it directly — no fuzzy matching, no fallback branch for unexpected wording
  • confidence is a number, so recolor writes confidence >= 0.7. Turtles above the threshold are lime, the rest orange. That comparison is the whole argument for the feature.

A second button shows llm:chat-json for contrast: valid JSON, but still a string to handle.

Verification

182 tests pass (97 → 182), deterministic — request payloads are asserted without live API calls, so CI needs no keys.

Verified live against Groq (openai/gpt-oss-20b) through NetLogo, not only in unit tests:

llm:chat-json          -> {"colors":["red","blue"]}
llm:chat-with-schema   -> [[action eat] [confidence 0.9]]
llm:get r "confidence" -> 0.9      is-number? true
nested: llm:get stats "speed" -> 10, tags -> [fast turtle speedy]
missing key            -> raises, naming the available keys

Demo run headless: 12 schema-constrained calls over 2 ticks, 0 failures. Turtles standing on food chose eat at 0.9 confidence while others chose explore at 0.8, so replies track state rather than repeating a default.

The model compiles headless against NetLogo 7.0.3.

Sharp edges, documented

  • The schema is a JSON string, not a NetLogo list. The escaped quotes are unavoidable inside a NetLogo string literal. Passing a list raises an error naming the problem — but it is the first thing a modeler gets wrong.
  • A missing key raises rather than reporting a default, and lists the keys that were available. carefully is needed when a field is genuinely optional.
  • Key matching is exact and case-sensitive, because JSON keys are.
  • llm:get skips entries that are not two-element pairs rather than matching positionally, so asking a plain list for a key misses instead of returning a neighbour.

Scope

Schema constraint governs the shape of a reply, not its truth — a well-formed reply can still be a bad decision. Provider support is model-dependent; the fallback path is exercised by tests.

JNK234 added 2 commits August 13, 2026 22:06
- Add schema-constrained and JSON chat primitives with NetLogo conversion.
- Send native response formats to OpenAI-compatible, Claude, Gemini, and Ollama providers.
- Constrain llm:choose while preserving graceful fallback behavior.
- Add provider, schema, conversion, history, and NetLogo integration coverage.
- Document supported formats, validation boundaries, and provider differences.

Implements #22.

Risk: provider support remains model-dependent; deterministic tests verify request payloads without live API calls.
The structured output work shipped docs and tests but no runnable model, so
there was nothing to open and press go.

Six turtles forage, each asking the model what to do with the reply
constrained to a JSON Schema. Two consequences are visible on screen rather
than only described:

- action is enum-constrained, so act-on compares it directly with no fuzzy
  matching or fallback branch
- confidence arrives as a NUMBER, so recolor can write confidence >= 0.7 —
  lime above the threshold, orange below

A second button shows llm:chat-json for contrast: valid JSON, but still a
string the modeller has to handle.

config.txt.example ships with placeholders; config.txt is gitignored.

Verified live against Groq: 12 schema-constrained calls over 2 ticks, 0
failures. Turtles on food chose eat at 0.9 confidence while others chose
explore at 0.8, so replies track state rather than repeating a default.
Model compiles headless against NetLogo 7.0.3.

Relates to #22.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Add structured output / JSON mode support across all LLM providers

1 participant