-
Notifications
You must be signed in to change notification settings - Fork 3.4k
fix(google): add thought signature to gemini 3 pro image parts #10462
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(google): add thought signature to gemini 3 pro image parts #10462
Conversation
|
I verified that Problem is that this will need a spec change because language-model-v3-file.ts does not defined providerMetadata?: SharedV3ProviderMetadata; yet (unlike all other LanguageModelV3Content types) but we would need to pass it via providerMetadata here 6078060/packages/google/src/google-generative-ai-language-model.ts#L268-L274 same way we pass it for function calls above Which means we cannot backport it. Or I'm missing something 🤔 |
…and set it in `convertToGoogleGenerativeAIMessages()`
|
we basically ran into #7979 |
…-gemini-3-pro-image-preview-in-middle-of-a-conversation-breaks-thought_signature
|
I will start the backport pull request, but won't merge until it's absolutely necessary, since the spec change is a breaking change und might cause friction to users unaffected by the bug this fixes. |
|
|
…ults (#10734) ## Background Provider metadata (such as Google Gemini's `thoughtSignature`) was being lost in two scenarios: 1. **Tool result propagation**: Provider-executed tool results within assistant message content were not receiving `providerOptions` from `callProviderMetadata`. This was partially addressed in PR #10361 (which fixed client-executed tool results) but missed the provider-executed case. 2. **Invalid tool call handling**: When a tool call failed input validation (e.g., Zod schema mismatch), the error handler was creating an invalid tool call object without preserving the `providerMetadata` and `providerExecuted` fields from the original tool call. Both issues are critical for providers like Google Gemini 3 that use provider metadata to maintain reasoning context across tool execution cycles. Gemini 3 enforces strict validation and returns a 400 error when signatures are missing: _"Unable to submit request because function call ... is missing a thought_signature"_. ## Summary <!-- What did you change? --> This PR implements two related fixes for provider metadata preservation: ### Fix 1: Provider-Executed Tool Result Metadata Propagation **File**: `packages/ai/src/ui/convert-to-model-messages.ts` Added `providerOptions` propagation from `callProviderMetadata` to provider-executed tool-result parts in assistant message content (lines 223-225). ### Fix 2: Invalid Tool Call Metadata Preservation **File**: `packages/ai/src/generate-text/parse-tool-call.ts` Added `providerMetadata` and `providerExecuted` preservation in the error handler (lines 95-96). When tool input validation fails, the invalid tool call object now includes these fields from the original `LanguageModelV3ToolCall`, matching the behavior of the success paths. **After this PR:** - All tool-result parts consistently receive `providerOptions` from `callProviderMetadata` - Invalid tool calls preserve provider metadata through validation failures - Gemini 3 `thoughtSignature` validation passes for both execution modes and error paths ## Manual Verification> I manually ran: [google-vertex-code-execution.](examples/ai-core/src/stream-text/google-vertex-code-execution.ts) [google-vertex-fullstream](examples/ai-core/src/stream-text/google-vertex-fullstream.ts) [google-vertex-grounding](examples/ai-core/src/stream-text/google-vertex-grounding.ts) with both the `gemini-3-pro-preview` and `gemini-2.5-pro` models and confirmed they function correctly Finally, I manually ran the provided reproduction script in #10560, and verified that it works correctly. ## Related Issues ## Related Issues Fixes #10560 Fixes #10721 **Backport PR:** - #10733 - Similar port of complete fix to `release-v5.0` branch **Related PRs:** - #10361 - Original partial fix (client-executed tool results only) **Potentially Related PRs:** - #10462 - `fix(google): add thought signature to gemini 3 pro image parts` - #10508 - Provider metadata handling improvements
… results (#10733) ## Background Provider metadata (such as Google Gemini's `thoughtSignature`) was being lost in two scenarios: 1. **Tool result propagation**: When tool results were converted to model messages, the `callProviderMetadata` from tool UI parts was not being propagated to tool-result parts. This was fixed in main branch (PR #10361, merged November 19, 2025) but not backported to v5.0. 2. **Invalid tool call handling**: When a tool call failed input validation (e.g., Zod schema mismatch), the error handler was creating an invalid tool call object without preserving the `providerMetadata` field from the original tool call. This is particularly problematic for providers like Google Gemini that use provider metadata to track `thoughtSignature` through reasoning and tool execution cycles. ## Summary <!-- What did you change? --> This PR includes two related fixes for provider metadata preservation on the `release-v5.0` branch: ### Fix 1: Tool Result Metadata Propagation (Backport from main) Backported changes ensure that `callProviderMetadata` from tool UI parts is correctly propagated to tool-result parts as `providerOptions` in two locations: 1. **Provider-executed tool results** (within assistant message content) - [convert-to-model-messages.ts:223-225](packages/ai/src/ui/convert-to-model-messages.ts#L223-L225) 2. **Client-executed tool results** (in separate tool messages) - [convert-to-model-messages.ts:281-283](packages/ai/src/ui/convert-to-model-messages.ts#L281-L283) ### Fix 2: Invalid Tool Call Metadata Preservation (New) Added `providerMetadata` preservation in the error handler of `parse-tool-call.ts` (line 88). When tool input validation fails, the invalid tool call object now includes the `providerMetadata` from the original `LanguageModelV2ToolCall`, matching the behavior of the success path. **File**: [packages/ai/src/generate-text/parse-tool-call.ts:88](packages/ai/src/generate-text/parse-tool-call.ts#L88) Both fixes ensure consistency in how provider metadata flows through the entire tool execution lifecycle (tool-call → tool-result) and error paths. ## Manual Verification Added comprehensive test coverage: - Client-executed tool result with provider metadata - Provider-executed tool result with provider metadata - Error state tool result with provider metadata - Dynamic tool result with provider metadata - Updated existing test snapshot to reflect the fix In addition, I manually ran: [google-vertex-code-execution.](examples/ai-core/src/stream-text/google-vertex-code-execution.ts) [google-vertex-fullstream](examples/ai-core/src/stream-text/google-vertex-fullstream.ts) [google-vertex-grounding](examples/ai-core/src/stream-text/google-vertex-grounding.ts) with both the `gemini-3-pro-preview` and `gemini-2.5-pro` models and confirmed they function correctly. Finally, I manually ran the provided reproduction script in #10560, and verified that it works correctly. ## Related Issues Fixes #10560 Fixes #10721 **Related PR:** - #10361 - Original fix in main branch **Potentially Related PRs:** - #10462 - `fix(google): add thought signature to gemini 3 pro image parts` - #10508 - Potentially related to provider metadata handling
Background
generateText()/streamText()doesn't pass thought signature for image parts in response messages. That breaks multi-turn chats withgemini-3-pro-image-previewSummary
response.messagesLanguageModelV3File#providerMetadata(optional field)Manual Verification
Checklist
pnpm changesetin the project root)Related Issues
Fixes #10441
Closes #7979