[bug] image and document blocks in tool_result are serialized into a text string when translating to OpenAI Chat

[Atharva-Kanherkar]

Symptom

An Anthropic tool_result whose content is a list of blocks (for example, a text block plus an image block) is translated to an OpenAI Chat tool message as a single string, with every non-text block dumped as raw JSON. The downstream model therefore cannot consume the image or document as multimodal input. No error is raised and the request can still return 200.

Anthropic explicitly allows text, image, document, and search_result blocks inside tool_result.content. However, OpenAI Chat tool messages are text-only: the official generated ChatCompletionToolMessageParam schema accepts a string or an array of text parts, not image_url or file parts:

https://github.com/openai/openai-python/blob/main/src/openai/types/chat/chat_completion_tool_message_param.py

Therefore, emitting an image_url directly inside role: "tool" would not be a valid OpenAI Chat representation. The bug is that Switchyard silently destroys the structured modality instead of performing a valid lowering or reporting the lossy/unsupported conversion.

Reproduction

Inbound format: Anthropic Messages. Route: passthrough to an openai_chat target. A small mock backend is used only to record the exact bytes Switchyard forwards.

routes.toml:

schema_version = 1
[llm_clients.local]
format = "openai_chat"
base_url = "http://127.0.0.1:9999/v1"
[llm_clients.local.extra_headers]
authorization = "Bearer x"
[targets.cap]
id = "captured-model"
llm_client = "local"
[routes.primary]
id = "captured-model"
type = "passthrough"
target = "cap"

switchyard-server --config routes.toml --port 9000

curl -s http://localhost:9000/v1/messages \
  -H 'content-type: application/json' \
  -H 'anthropic-version: 2023-06-01' \
  -d '{
    "model": "captured-model",
    "max_tokens": 100,
    "tools": [{"name":"screenshot","description":"take a screenshot","input_schema":{"type":"object","properties":{"t":{"type":"string"}},"required":["t"]}}],
    "messages": [
      {"role":"user","content":"screenshot the page"},
      {"role":"assistant","content":[{"type":"tool_use","id":"toolu_1","name":"screenshot","input":{"t":"page"}}]},
      {"role":"user","content":[{"type":"tool_result","tool_use_id":"toolu_1","content":[
        {"type":"text","text":"here it is:"},
        {"type":"image","source":{"type":"base64","media_type":"image/png","data":"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAAC0lEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="}}
      ]}]}
    ]
  }'

Expected vs. actual

• Expected: Switchyard should preserve the modality using a representation valid for OpenAI Chat—for example, emit the textual tool result as the required tool message and lower the image/document into a subsequent multimodal user message, while preserving ordering for parallel tool calls. If a faithful lowering is not supported, the conversion should be rejected or produce a clear lossy-conversion/unsupported-modality diagnostic.
• Actual: the forwarded tool message contains one plain string with the image block serialized as JSON. There is no multimodal content part and no diagnostic.

role: "tool"
content: "here it is: {\"source\":{\"data\":\"iVBORw0KGgoAAA...==\",\"media_type\":\"image/png\",\"type\":\"base64\"},\"type\":\"image\"}"

A document block inside tool_result behaves the same way: it is serialized into the text string rather than preserved or explicitly rejected.

Impact

This affects real agent workflows in which tools return images or documents, including browser automation, screenshot tools, visual inspection, OCR, and document readers. The tool appears to succeed, but the downstream vision-capable model is blind to the result.

For realistic screenshots or documents, serializing base64 into literal text can also consume a large number of tokens or cause a context-limit failure, in addition to losing the modality.

Environment

• Switchyard version (or commit SHA): 2bef154 (2bef154), release build from source
• Python version: 3.9.6
• OS / arch: macOS arm64
• Install path: source build (cargo build --release -p switchyard-server)
• Inbound format: Anthropic Messages
• Backend: OpenAI-compatible (a local mock was used to capture the forwarded request)

Additional context

Anthropic documents multimodal tool results here:

https://platform.claude.com/docs/en/agents-and-tools/tool-use/handle-tool-calls

Streaming tool-call translation looked correct in the same setup; this report concerns non-text blocks nested inside buffered tool_result content.

Possibly related to #152 (filtering unsupported input modalities per target), though here the block is neither filtered nor rejected—it is silently stringified.