Skip to content

Conversation

@gr2m
Copy link
Collaborator

@gr2m gr2m commented Nov 21, 2025

Background

generateText() / streamText() doesn't pass thought signature for image parts in response messages. That breaks multi-turn chats with gemini-3-pro-image-preview

Summary

  • Gemini now supports non-PNG images in assistant messages
  • Add thought signature to image parts in response.messages
  • spec update: LanguageModelV3File#providerMetadata (optional field)

Manual Verification

import { google } from '@ai-sdk/google';
import { generateText } from 'ai';
import 'dotenv/config';
import { presentImages } from '../lib/present-image';

async function main() {
  const step1 = await generateText({
    model: google('gemini-3-pro-image-preview'),
    prompt: 'Create an image of Los Angeles where all car infrastructure has been replaced with bike infrastructure, trains, pedestrian zones, and parks. The image should be photorealistic and vibrant.',
  });

  await presentImages(step1.files);

  const step2 = await generateText({
    model: google('gemini-3-pro-image-preview'),
    messages: [
      ...step1.response.messages,
      {
        role: 'user',
        content: 'Now create a variation of the image, but in the style of a watercolor painting.',
      },
    ]
  });

  await presentImages(step2.files);
}

main().catch(console.error);

Checklist

  • Tests have been added / updated (for bug fixes / features)
  • Documentation has been added / updated (for bug fixes / features)
  • A patch changeset for relevant packages has been added (for bug fixes / features - run pnpm changeset in the project root)
  • I have reviewed this pull request (self-review)

Related Issues

Fixes #10441
Closes #7979

@gr2m
Copy link
Collaborator Author

gr2m commented Nov 21, 2025

I verified that body.contents[0].parts[0].thoughtSignature (or body.contents[0].parts[0].thought_signature) needs to be set in the request body.

Problem is that this will need a spec change because language-model-v3-file.ts does not defined providerMetadata?: SharedV3ProviderMetadata; yet (unlike all other LanguageModelV3Content types)

but we would need to pass it via providerMetadata here

6078060/packages/google/src/google-generative-ai-language-model.ts#L268-L274

same way we pass it for function calls above

Which means we cannot backport it.

Or I'm missing something 🤔

…and set it in `convertToGoogleGenerativeAIMessages()`
@gr2m gr2m added bug Something isn't working as documented provider/google and removed provider/black-forest-labs Issues related to the @ai-sdk/black-forst-labs provider. https://bfl.ai labels Nov 22, 2025
@gr2m
Copy link
Collaborator Author

gr2m commented Nov 22, 2025

we basically ran into #7979

gr2m added 2 commits November 23, 2025 14:24
…-gemini-3-pro-image-preview-in-middle-of-a-conversation-breaks-thought_signature
@gr2m gr2m marked this pull request as ready for review November 23, 2025 22:57
@gr2m gr2m changed the title fix(google): gemini 3 pro image parts and thought signature fix(google): add thought signature to gemini 3 pro image parts Nov 23, 2025
@gr2m gr2m added the backport label Nov 23, 2025
@gr2m
Copy link
Collaborator Author

gr2m commented Nov 23, 2025

I will start the backport pull request, but won't merge until it's absolutely necessary, since the spec change is a breaking change und might cause friction to users unaffected by the bug this fixes.

@gr2m gr2m enabled auto-merge (squash) November 23, 2025 23:09
@gr2m gr2m merged commit db913bd into main Nov 23, 2025
22 checks passed
@gr2m gr2m deleted the 10441-switching-from-gemini-3-pro-preview-to-gemini-3-pro-image-preview-in-middle-of-a-conversation-breaks-thought_signature branch November 23, 2025 23:11
vercel-ai-sdk bot pushed a commit that referenced this pull request Nov 23, 2025
@vercel-ai-sdk vercel-ai-sdk bot removed the backport label Nov 23, 2025
@vercel-ai-sdk
Copy link
Contributor

vercel-ai-sdk bot commented Nov 23, 2025

⚠️ Backport to release-v5.0 created but has conflicts: #10508

lgrammel pushed a commit that referenced this pull request Dec 1, 2025
…ults (#10734)

## Background

Provider metadata (such as Google Gemini's `thoughtSignature`) was being
lost in two scenarios:

1. **Tool result propagation**: Provider-executed tool results within
assistant message content were not receiving `providerOptions` from
`callProviderMetadata`. This was partially addressed in PR #10361 (which
fixed client-executed tool results) but missed the provider-executed
case.

2. **Invalid tool call handling**: When a tool call failed input
validation (e.g., Zod schema mismatch), the error handler was creating
an invalid tool call object without preserving the `providerMetadata`
and `providerExecuted` fields from the original tool call.

Both issues are critical for providers like Google Gemini 3 that use
provider metadata to maintain reasoning context across tool execution
cycles. Gemini 3 enforces strict validation and returns a 400 error when
signatures are missing: _"Unable to submit request because function call
... is missing a thought_signature"_.

## Summary

<!-- What did you change? -->

This PR implements two related fixes for provider metadata preservation:

### Fix 1: Provider-Executed Tool Result Metadata Propagation

**File**: `packages/ai/src/ui/convert-to-model-messages.ts`

Added `providerOptions` propagation from `callProviderMetadata` to
provider-executed tool-result parts in assistant message content (lines
223-225).

### Fix 2: Invalid Tool Call Metadata Preservation

**File**: `packages/ai/src/generate-text/parse-tool-call.ts`

Added `providerMetadata` and `providerExecuted` preservation in the
error handler (lines 95-96). When tool input validation fails, the
invalid tool call object now includes these fields from the original
`LanguageModelV3ToolCall`, matching the behavior of the success paths.

**After this PR:**
- All tool-result parts consistently receive `providerOptions` from
`callProviderMetadata`
- Invalid tool calls preserve provider metadata through validation
failures
- Gemini 3 `thoughtSignature` validation passes for both execution modes
and error paths

## Manual Verification>

I manually ran: 

[google-vertex-code-execution.](examples/ai-core/src/stream-text/google-vertex-code-execution.ts)

[google-vertex-fullstream](examples/ai-core/src/stream-text/google-vertex-fullstream.ts)

[google-vertex-grounding](examples/ai-core/src/stream-text/google-vertex-grounding.ts)

with both the `gemini-3-pro-preview` and `gemini-2.5-pro` models and
confirmed they function correctly

Finally, I manually ran the provided reproduction script in #10560, and
verified that it works correctly.

## Related Issues

## Related Issues

Fixes #10560
Fixes #10721

**Backport PR:**
- #10733 - Similar port of complete fix to `release-v5.0` branch

**Related PRs:**
- #10361 - Original partial fix (client-executed tool results only)

**Potentially Related PRs:**
- #10462 - `fix(google): add thought signature to gemini 3 pro image
parts`
- #10508 - Provider metadata handling improvements
lgrammel pushed a commit that referenced this pull request Dec 1, 2025
… results (#10733)

## Background

Provider metadata (such as Google Gemini's `thoughtSignature`) was being
lost in two scenarios:

1. **Tool result propagation**: When tool results were converted to
model messages, the `callProviderMetadata` from tool UI parts was not
being propagated to tool-result parts. This was fixed in main branch (PR
#10361, merged November 19, 2025) but not backported to v5.0.

2. **Invalid tool call handling**: When a tool call failed input
validation (e.g., Zod schema mismatch), the error handler was creating
an invalid tool call object without preserving the `providerMetadata`
field from the original tool call.

This is particularly problematic for providers like Google Gemini that
use provider metadata to track `thoughtSignature` through reasoning and
tool execution cycles.

## Summary

<!-- What did you change? -->

This PR includes two related fixes for provider metadata preservation on
the `release-v5.0` branch:

### Fix 1: Tool Result Metadata Propagation (Backport from main)

Backported changes ensure that `callProviderMetadata` from tool UI parts
is correctly propagated to tool-result parts as `providerOptions` in two
locations:

1. **Provider-executed tool results** (within assistant message content)
-
[convert-to-model-messages.ts:223-225](packages/ai/src/ui/convert-to-model-messages.ts#L223-L225)
2. **Client-executed tool results** (in separate tool messages) -
[convert-to-model-messages.ts:281-283](packages/ai/src/ui/convert-to-model-messages.ts#L281-L283)

### Fix 2: Invalid Tool Call Metadata Preservation (New)

Added `providerMetadata` preservation in the error handler of
`parse-tool-call.ts` (line 88). When tool input validation fails, the
invalid tool call object now includes the `providerMetadata` from the
original `LanguageModelV2ToolCall`, matching the behavior of the success
path.

**File**:
[packages/ai/src/generate-text/parse-tool-call.ts:88](packages/ai/src/generate-text/parse-tool-call.ts#L88)

Both fixes ensure consistency in how provider metadata flows through the
entire tool execution lifecycle (tool-call → tool-result) and error
paths.

## Manual Verification

Added comprehensive test coverage:
- Client-executed tool result with provider metadata
- Provider-executed tool result with provider metadata  
- Error state tool result with provider metadata
- Dynamic tool result with provider metadata
- Updated existing test snapshot to reflect the fix

In addition, I manually ran: 


[google-vertex-code-execution.](examples/ai-core/src/stream-text/google-vertex-code-execution.ts)

[google-vertex-fullstream](examples/ai-core/src/stream-text/google-vertex-fullstream.ts)

[google-vertex-grounding](examples/ai-core/src/stream-text/google-vertex-grounding.ts)

with both the `gemini-3-pro-preview` and `gemini-2.5-pro` models and
confirmed they function correctly.

Finally, I manually ran the provided reproduction script in #10560, and
verified that it works correctly.

## Related Issues

Fixes #10560
Fixes #10721

**Related PR:**
- #10361 - Original fix in main branch

**Potentially Related PRs:**
- #10462 - `fix(google): add thought signature to gemini 3 pro image
parts`
- #10508 - Potentially related to provider metadata handling
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai/provider bug Something isn't working as documented provider/google

Projects

None yet

2 participants