Skip to content

feat(ai): structured outputs — response_format json_schema for JSON-returning calls where the provider supports it #670

Description

@devops-thiago

Split out of #668 so each deliverable can be scheduled and closed independently.

Problem Statement

Every JSON-returning call (review, verifier, summary — and the resolution check once #667 lands) spends significant prompt space specifying output shape: "Respond ONLY with valid JSON…", full schemas, field rules. And because the shape is only requested, malformed output needs the parse-salvage paths and TruncatedResponseSalvager machinery. With the multi-call decomposition (#669) multiplying calls, this boilerplate is paid on every one of them.

Proposed Solution

Where the configured OpenAI-compatible provider supports response_format: json_schema, request it for the JSON-returning calls via LangChain4j request options:

  • Per-model capability flag: thrillhousebot.ai.models.<model>.structured-outputs (default off), following the existing per-model settings pattern in ActiveModelSettings.
  • Prompt-side format instructions shrink to intent only; the schema moves into the request.
  • Graceful degradation: flag off or provider unsupported → current prompt-specified JSON + salvage paths, byte-identical behavior.
  • Truncation salvage stays regardless — finish_reason=length can still cut a schema-valid stream mid-array.

Interacts with #30 (native multi-provider support), which needs the same per-provider capability surface; whichever lands first should establish it.

Touchpoints

config/ThrillhouseConfig.java / config/ActiveModelSettings.java (capability flag), the @RegisterAiService request options for PrReviewer / FindingVerifier / PrSummarizer, review/DiffBudgetPlanner.java (overhead estimation shrinks when format boilerplate leaves the prompts).

Acceptance criteria

  • With the flag on against a supporting provider: zero parse-salvage activations in tests; measured prompt-token reduction on every JSON-returning call.
  • With the flag off or an unsupported provider: behavior unchanged.
  • Startup validation rejects the flag on models where the endpoint reports no support, rather than failing at review time.

Alternatives Considered

  • Keep prompt-specified JSON everywhere — simplest, but the boilerplate and salvage complexity now multiply across the decomposed pipeline.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestjavaPull requests that update java code

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions