Issues with structured output on gemini-3.6-flash & gemini-3.5-flash-lite with image input.

[iwanbolzern]

Hi all,
Just wanted to see if anyone else has issues with extracting information from images and returning it as pydantic models? I updated from gemini-3.1-pro-preview and suddenly get thinking steps content in the pydantic fields.

[Venkaiahbabuneelam]

Hey @iwanbolzern, Thank you for reporting this issue with us. To assist us in effectively investigating and reproducing it, please provide a minimal, runnable code example, including the complete error logs.
Thanks

[iwanbolzern]

It's difficult to provide you with a minimal workable example without leaking any company code, but I try my best... When we do something like the following:

import asyncio
import base64

from google import genai
from google.genai.interactions import GenerationConfig, ImageContent, TextContent, TextResponseFormat
from pydantic import BaseModel, ConfigDict, Field


class PersonInfo(BaseModel):
    model_config = ConfigDict(frozen=True)

    first_name: str = Field(..., description="First name")
    last_name: str = Field(..., description="Last name")


async def parse_image(data: bytes) -> PersonInfo:
    client = genai.Client(api_key="YOUR KEY").aio
    interaction = await client.interactions.create(
        model="gemini-3.6-flash",
        input=[
            TextContent(text="Extract first & last name from this image."),
            ImageContent(data=base64.b64encode(data).decode("utf-8"), mime_type="image/jpeg"),
        ],
        response_format=TextResponseFormat(
            mime_type="application/json",
            schema_=PersonInfo.model_json_schema(),
        ),
        generation_config=GenerationConfig(seed=42, thinking_level="low"),
        store=False,
    )

    return PersonInfo.model_validate_json(interaction.output_text)


if __name__ == "__main__":
    asyncio.run(parse_image(b"YOUR IMAGE DATA HERE"))

We receive with gemini-3.6-flash staff like person_info.first_name="Günter Test --> firstname lastname --> json field firstname --> Günter, which I would identify as model thinking steps... With gemini-3.1-pro-preview we never saw things like this.

[strangeleo02]

Hey @iwanbolzern ,

This happens because gemini-3.6-flash outputs reasoning tokens when thinking_level is on (3.1-pro-preview doesn't do this). Since interaction.output_text just concatenates all the text parts together, the model's internal reasoning ends up getting mixed in with the actual JSON payload.

A couple of ways to fix it:

Option 1: Filter out the "thought" parts before parsing

json_text = "".join(
    part.text
    for candidate in interaction.candidates
    for part in candidate.content.parts
    if not getattr(part, "thought", False) and getattr(part, "text", None)
)
return PersonInfo.model_validate_json(json_text)

Option 2: Just disable thinking for this call

Since you're only after structured JSON here anyway, you probably don't need the reasoning step — turning it off will also make things faster and cheaper:

generation_config = GenerationConfig(seed=42, thinking_level="off")

[iwanbolzern]

Hi @strangeleo02,

Thank you for your input. When I configure off for thinking level, I get:

E       google.genai._gaos.lib.compat_errors.BadRequestError: Error code: 400 - {'error': {'message': "The value 'off' is not supported for 'generation_config.thinking_level'. Supported values: 'minimal', 'low', 'medium', 'high'.", 'code': 'invalid_request'}}

[strangeleo02]

Hey @iwanbolzern,
Sorry about that! Try

generation_config = GenerationConfig(seed=42, thinking_level="minimal")

but you will need to filter out the thought part

[Venkaiahbabuneelam]

Hey @iwanbolzern, Thank you for the detailed explanation and the reproducible code, We didn't encounter any errors and successfully return structured output with the code below, could you please take a look at it..?

"# Define structured output schema via Pydantic
class ImageAnalysis(BaseModel):
description: str = Field(description="Summary description of the image content")
detected_objects: list[str] = Field(description="List of key objects present in the image")

Create a sample image in memory

img = Image.new('RGB', (100, 100), color = 'red')
img_byte_arr = io.BytesIO()
img.save(img_byte_arr, format='JPEG')
img_bytes = img_byte_arr.getvalue()

Call gemini-3.6-flash / gemini-3.5-flash-lite with structured output schema and image input

response = client.models.generate_content(
model='gemini-3.6-flash',
contents=[
types.Part.from_bytes(data=img_bytes, mime_type='image/jpeg'),
"Extract information from this image into the structured schema."
],
config=types.GenerateContentConfig(
response_mime_type="application/json",
response_schema=ImageAnalysis,
),
)

print("Response Text:")
print(response.text)"

Thanks

[github-actions]

This issue has been marked as stale because it has been open for 7 days with no activity. It will be closed in 2 days if no further activity occurs.