feat: add structured output support (schema-constrained and JSON chat) - #74
Open
JNK234 wants to merge 2 commits into
Open
feat: add structured output support (schema-constrained and JSON chat)#74JNK234 wants to merge 2 commits into
JNK234 wants to merge 2 commits into
Conversation
- Add schema-constrained and JSON chat primitives with NetLogo conversion. - Send native response formats to OpenAI-compatible, Claude, Gemini, and Ollama providers. - Constrain llm:choose while preserving graceful fallback behavior. - Add provider, schema, conversion, history, and NetLogo integration coverage. - Document supported formats, validation boundaries, and provider differences. Implements #22. Risk: provider support remains model-dependent; deterministic tests verify request payloads without live API calls.
The structured output work shipped docs and tests but no runnable model, so there was nothing to open and press go. Six turtles forage, each asking the model what to do with the reply constrained to a JSON Schema. Two consequences are visible on screen rather than only described: - action is enum-constrained, so act-on compares it directly with no fuzzy matching or fallback branch - confidence arrives as a NUMBER, so recolor can write confidence >= 0.7 — lime above the threshold, orange below A second button shows llm:chat-json for contrast: valid JSON, but still a string the modeller has to handle. config.txt.example ships with placeholders; config.txt is gitignored. Verified live against Groq: 12 schema-constrained calls over 2 ticks, 0 failures. Turtles on food chose eat at 0.9 confidence while others chose explore at 0.8, so replies track state rather than repeating a default. Model compiles headless against NetLogo 7.0.3. Relates to #22.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #22. Roadmap B-series — the largest open capability gap, and the shape Jacob sketched at the 29 Jul meeting.
Why
Every primitive returned free text with no structural guarantee. A model that needs a number had to hope the phrasing stayed stable and pick it out of a sentence:
That works until the model words things differently, and then it fails quietly.
What
Three primitives:
llm:chat-with-schema prompt schemallm:chat-json promptllm:get parsed key[[key value] ...]listNative response formats are sent to OpenAI-compatible, Claude, Gemini, and Ollama providers.
llm:choosenow sends its option list as a schema constraint where the provider supports it, and keeps the existing fallback where it does not.How JSON maps into NetLogo
NetLogo has no dictionary type, so objects become
[key value]pair lists — the association-list shape modelers already use.{"a": 1}[[a 1]][1, 2][1 2]"text"10truenull""— NetLogo has no nullNested objects are more pair lists, so
llm:getchains rather than taking a dotted path.Demo
demos/structured-outputs/— six turtles forage, each asking the model what to do with the reply schema-constrained.Two consequences are visible on screen rather than only described:
actionis enum-constrained, soact-oncompares it directly — no fuzzy matching, no fallback branch for unexpected wordingconfidenceis a number, sorecolorwritesconfidence >= 0.7. Turtles above the threshold are lime, the rest orange. That comparison is the whole argument for the feature.A second button shows
llm:chat-jsonfor contrast: valid JSON, but still a string to handle.Verification
182 tests pass (97 → 182), deterministic — request payloads are asserted without live API calls, so CI needs no keys.
Verified live against Groq (
openai/gpt-oss-20b) through NetLogo, not only in unit tests:Demo run headless: 12 schema-constrained calls over 2 ticks, 0 failures. Turtles standing on food chose
eatat 0.9 confidence while others choseexploreat 0.8, so replies track state rather than repeating a default.The model compiles headless against NetLogo 7.0.3.
Sharp edges, documented
carefullyis needed when a field is genuinely optional.llm:getskips entries that are not two-element pairs rather than matching positionally, so asking a plain list for a key misses instead of returning a neighbour.Scope
Schema constraint governs the shape of a reply, not its truth — a well-formed reply can still be a bad decision. Provider support is model-dependent; the fallback path is exercised by tests.