Backport: fix(google): add thought signature to gemini 3 pro image parts #10508
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4 tasks
gr2m
requested changes
Nov 23, 2025
Collaborator
gr2m
left a comment
gr2m
left a comment
There was a problem hiding this comment.
blocking merge until deemed necessary
| */ | ||
| data: string | Uint8Array; | ||
|
|
||
| providerMetadata?: SharedV2ProviderMetadata; |
10 tasks
This was referenced Dec 1, 2025
lgrammel
pushed a commit
that referenced
this pull request
Dec 1, 2025
…ults (#10734) ## Background Provider metadata (such as Google Gemini's `thoughtSignature`) was being lost in two scenarios: 1. **Tool result propagation**: Provider-executed tool results within assistant message content were not receiving `providerOptions` from `callProviderMetadata`. This was partially addressed in PR #10361 (which fixed client-executed tool results) but missed the provider-executed case. 2. **Invalid tool call handling**: When a tool call failed input validation (e.g., Zod schema mismatch), the error handler was creating an invalid tool call object without preserving the `providerMetadata` and `providerExecuted` fields from the original tool call. Both issues are critical for providers like Google Gemini 3 that use provider metadata to maintain reasoning context across tool execution cycles. Gemini 3 enforces strict validation and returns a 400 error when signatures are missing: _"Unable to submit request because function call ... is missing a thought_signature"_. ## Summary <!-- What did you change? --> This PR implements two related fixes for provider metadata preservation: ### Fix 1: Provider-Executed Tool Result Metadata Propagation **File**: `packages/ai/src/ui/convert-to-model-messages.ts` Added `providerOptions` propagation from `callProviderMetadata` to provider-executed tool-result parts in assistant message content (lines 223-225). ### Fix 2: Invalid Tool Call Metadata Preservation **File**: `packages/ai/src/generate-text/parse-tool-call.ts` Added `providerMetadata` and `providerExecuted` preservation in the error handler (lines 95-96). When tool input validation fails, the invalid tool call object now includes these fields from the original `LanguageModelV3ToolCall`, matching the behavior of the success paths. **After this PR:** - All tool-result parts consistently receive `providerOptions` from `callProviderMetadata` - Invalid tool calls preserve provider metadata through validation failures - Gemini 3 `thoughtSignature` validation passes for both execution modes and error paths ## Manual Verification> I manually ran: [google-vertex-code-execution.](examples/ai-core/src/stream-text/google-vertex-code-execution.ts) [google-vertex-fullstream](examples/ai-core/src/stream-text/google-vertex-fullstream.ts) [google-vertex-grounding](examples/ai-core/src/stream-text/google-vertex-grounding.ts) with both the `gemini-3-pro-preview` and `gemini-2.5-pro` models and confirmed they function correctly Finally, I manually ran the provided reproduction script in #10560, and verified that it works correctly. ## Related Issues ## Related Issues Fixes #10560 Fixes #10721 **Backport PR:** - #10733 - Similar port of complete fix to `release-v5.0` branch **Related PRs:** - #10361 - Original partial fix (client-executed tool results only) **Potentially Related PRs:** - #10462 - `fix(google): add thought signature to gemini 3 pro image parts` - #10508 - Provider metadata handling improvements
lgrammel
pushed a commit
that referenced
this pull request
Dec 1, 2025
… results (#10733) ## Background Provider metadata (such as Google Gemini's `thoughtSignature`) was being lost in two scenarios: 1. **Tool result propagation**: When tool results were converted to model messages, the `callProviderMetadata` from tool UI parts was not being propagated to tool-result parts. This was fixed in main branch (PR #10361, merged November 19, 2025) but not backported to v5.0. 2. **Invalid tool call handling**: When a tool call failed input validation (e.g., Zod schema mismatch), the error handler was creating an invalid tool call object without preserving the `providerMetadata` field from the original tool call. This is particularly problematic for providers like Google Gemini that use provider metadata to track `thoughtSignature` through reasoning and tool execution cycles. ## Summary <!-- What did you change? --> This PR includes two related fixes for provider metadata preservation on the `release-v5.0` branch: ### Fix 1: Tool Result Metadata Propagation (Backport from main) Backported changes ensure that `callProviderMetadata` from tool UI parts is correctly propagated to tool-result parts as `providerOptions` in two locations: 1. **Provider-executed tool results** (within assistant message content) - [convert-to-model-messages.ts:223-225](packages/ai/src/ui/convert-to-model-messages.ts#L223-L225) 2. **Client-executed tool results** (in separate tool messages) - [convert-to-model-messages.ts:281-283](packages/ai/src/ui/convert-to-model-messages.ts#L281-L283) ### Fix 2: Invalid Tool Call Metadata Preservation (New) Added `providerMetadata` preservation in the error handler of `parse-tool-call.ts` (line 88). When tool input validation fails, the invalid tool call object now includes the `providerMetadata` from the original `LanguageModelV2ToolCall`, matching the behavior of the success path. **File**: [packages/ai/src/generate-text/parse-tool-call.ts:88](packages/ai/src/generate-text/parse-tool-call.ts#L88) Both fixes ensure consistency in how provider metadata flows through the entire tool execution lifecycle (tool-call → tool-result) and error paths. ## Manual Verification Added comprehensive test coverage: - Client-executed tool result with provider metadata - Provider-executed tool result with provider metadata - Error state tool result with provider metadata - Dynamic tool result with provider metadata - Updated existing test snapshot to reflect the fix In addition, I manually ran: [google-vertex-code-execution.](examples/ai-core/src/stream-text/google-vertex-code-execution.ts) [google-vertex-fullstream](examples/ai-core/src/stream-text/google-vertex-fullstream.ts) [google-vertex-grounding](examples/ai-core/src/stream-text/google-vertex-grounding.ts) with both the `gemini-3-pro-preview` and `gemini-2.5-pro` models and confirmed they function correctly. Finally, I manually ran the provided reproduction script in #10560, and verified that it works correctly. ## Related Issues Fixes #10560 Fixes #10721 **Related PR:** - #10361 - Original fix in main branch **Potentially Related PRs:** - #10462 - `fix(google): add thought signature to gemini 3 pro image parts` - #10508 - Potentially related to provider metadata handling
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is an automated backport of #10462 to the release-v5.0 branch.
🚨 this pull request includes a spec change which is a breaking change. We might need to backport it anyway since it is needed for a fix that enables multi-turn chats with gemini-3-pro-image-preview.