How to handle non-standard fields (e.g., images) lost in ChatResponse when using non-OpenAI providers?

[nblog]

When using OpenAIChatClient with non-OpenAI providers (such as OpenRouter with google/gemini-3-pro-image-preview), the response may contain additional fields like images that are not part of the standard OpenAI response schema. These fields are currently being lost in the _create_chat_response method.

Problem Description

Consider the following example using OpenRouter to generate images:

from openai import OpenAI

client = OpenAI(
  base_url="https://openrouter.ai/api/v1",
  api_key="<OPENROUTER_API_KEY>",
)

# Generate an image
response = client.chat.completions.create(
  model="google/gemini-3-pro-image-preview",
  messages=[
          {
            "role": "user",
            "content": "Generate a beautiful sunset over mountains"
          }
        ],
  extra_body={"modalities": ["image", "text"]}
)

# The generated image will be in the assistant message
message = response.choices[0].message
if message.images:
  for image in message.images:
    image_url = image['image_url']['url']  # Base64 data URL
    print(f"Generated image: {image_url[:50]}...")

When this response passes through agent_framework/openai/_chat_client.py, the _create_chat_response method only extracts:

• Text content via _parse_text_from_choice()
• Tool calls via _get_tool_calls_from_chat_choice()

The images field (and potentially other provider-specific fields) is not captured and is therefore lost in the returned ChatResponse.

Relevant Code

In _chat_client.py line 200-224:

def _create_chat_response(self, response: ChatCompletion, chat_options: ChatOptions) -> "ChatResponse":
    """Create a chat message content object from a choice."""
    response_metadata = self._get_metadata_from_chat_response(response)
    messages: list[ChatMessage] = []
    finish_reason: FinishReason | None = None
    for choice in response.choices:
        response_metadata.update(self._get_metadata_from_chat_choice(choice))
        if choice.finish_reason:
            finish_reason = FinishReason(value=choice.finish_reason)
        contents: list[Contents] = []
        if text_content := self._parse_text_from_choice(choice):
            contents. append(text_content)
        if parsed_tool_calls := [tool for tool in self._get_tool_calls_from_chat_choice(choice)]:
            contents. extend(parsed_tool_calls)
        messages.append(ChatMessage(role="assistant", contents=contents))
    # ... images field is not handled here

Questions

Is there a recommended way to preserve these non-standard fields when using OpenAIChatClient with alternative providers?

[moonbox3]

Adding @dmytrostruk