Python: Support detail field in OpenAI Chat API image_url payload

[moonbox3]

Motivation and Context

The OpenAI Chat API supports an optional detail field (high, low, auto) in image_url content parts to control image processing fidelity, but the SDK silently dropped this value even when users set it via additional_properties. This made it impossible to control vision token costs or image resolution behavior.

Fixes #4616

Description

The _prepare_content_for_openai method in _chat_client.py hard-coded image_url payloads as {"url": content.uri} without reading the detail field from additional_properties, even though the pattern for forwarding extra properties already existed elsewhere in the function (e.g., for filename). The fix reads detail from content.additional_properties when present and includes it in the image_url dictionary. Tests cover all three valid detail values (high, low, auto) as well as the default case where no detail is specified.

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the Contribution Guidelines
• All unit tests pass, and I have added new tests where possible
• Is this a breaking change? If yes, add "[BREAKING]" prefix to the title of the PR.

---

Note: PR autogenerated by moonbox3's agent

[moonbox3]

Automated Code Review

Reviewers: 4 | Confidence: 95%

✓ Correctness

The diff deletes a temporary reproduction report markdown file (REPRODUCTION_REPORT.md) that was accidentally committed to the repository root. This is a cleanup-only change removing a non-code artifact — there are no correctness, logic, or runtime concerns. The file contained investigation notes and local filesystem paths that should not be in the repository.

✓ Security Reliability

The diff deletes a REPRODUCTION_REPORT.md file that contained internal filesystem paths (/repos/agent-framework/.worktrees/agent/fix-4616-2) and operational metadata (timestamps, gate results). Removing this file is a cleanup action with no security or reliability concerns — it eliminates minor information-leakage risk from committed internal paths. No code changes are involved.

✓ Test Coverage

The diff only deletes a REPRODUCTION_REPORT.md file, which is a markdown investigation artifact and not source code or test code. There are no behavioral changes, no new features, and no test modifications to assess for coverage. Removing this file from the repository is appropriate cleanup.

✓ Design Approach

The diff only deletes a temporary REPRODUCTION_REPORT.md file that was a planning/investigation artifact committed to the repository. There are no code changes to review for design approach.

Suggestions

• Consider also removing the failing test file referenced in the report (python/packages/core/tests/openai/test_image_url_detail_bug.py) if it was only created as a temporary reproduction artifact and is not part of the final fix.
• Reproduction reports and investigation artifacts should not be committed to the repository in the first place; use session-local scratch files instead.

---

Automated review by moonbox3's agents

[Copilot]

Pull request overview

Adds support in the Python OpenAI Chat client for forwarding the optional detail field inside image_url content parts, enabling callers to control image processing fidelity/cost via Content.additional_properties.

Changes:

• Update _prepare_content_for_openai to include image_url.detail when provided in Content.additional_properties.
• Add unit tests covering detail values (high, low, auto) and the default behavior when omitted.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
python/packages/core/agent_framework/openai/_chat_client.py	Propagates detail from Content.additional_properties into the OpenAI image_url payload.
python/packages/core/tests/openai/test_openai_chat_client.py	Adds coverage verifying detail passthrough for valid values and absence by default.

---

You can also share your feedback on Copilot code review. Take the survey.

[markwallace-microsoft]

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
packages/core/agent_framework/openai
_chat_client.py	333	27	91%	212, 293–294, 298, 423, 430, 506–513, 515–518, 528, 606, 608, 625, 646, 674, 687, 711, 731
TOTAL	24008	2638	89%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
5254	20 💤	0 ❌	0 🔥	1m 24s ⏱️

[moonbox3]

Automated Code Review

Reviewers: 4 | Confidence: 67%

✓ Correctness

✓ Security Reliability

The change passes the detail field from additional_properties directly into the OpenAI API request without validating that it is one of the allowed string values (low, high, auto). While additional_properties is typically developer-controlled, the lack of validation means an arbitrary value (including non-string types like dicts or lists) can be injected into the API payload. The responses client at line 1116 has the same pattern but defaults to "auto". The test coverage is thorough for the happy path but does not test invalid or non-string values. This is a low-severity input validation gap at a trust boundary (outbound API call construction).

✓ Test Coverage

✓ Design Approach

The change adds detail support for OpenAI image content by reading it from Content.additional_properties. The approach of threading an OpenAI Vision API-specific parameter through a generic additional_properties bag is a leaky abstraction — it encodes provider-specific semantics into a transport-level escape hatch with no type safety or documented contract. However, given that additional_properties exists precisely for this extensibility purpose and no typed alternative (e.g., ImageContent subtype) is established in the codebase, this is an acceptable interim approach. The more concrete and actionable issue is the hasattr(content, 'additional_properties') guard: additional_properties is always initialized to a dict (never absent, never None) on every Content instance, so the hasattr check is both redundant and misleading — it suggests the field might not exist on all Content objects, which is false and will confuse future readers.

Suggestions

• Simplify the detail extraction by removing the redundant hasattr guard (since additional_properties is always initialized to {} on every Content instance) and validate that detail is one of the accepted values ('low', 'high', 'auto') before including it in the API payload. This prevents sending unexpected types or values to the OpenAI API and accurately reflects the actual contract.
• Consider adding a test case for when detail is set to an invalid value (e.g., a non-string type or an unrecognized string) to document expected behavior.
• Using additional_properties as a grab-bag for the OpenAI-specific detail field is workable but leaky — the key 'detail' is undiscoverable from the type system and silently ignored by all non-OpenAI backends. A longer-term improvement would be a typed ImageDetail field on a dedicated image content subtype or on OpenAIChatOptions, so the contract is explicit and discoverable.

---

Automated review by moonbox3's agents

[eavanvalkenburg]

Automated Code Review

Reviewers: 4 | Confidence: 71%

✓ Correctness

✓ Security Reliability

The change adds support for passing the detail field in OpenAI image_url payloads, sourced from content.additional_properties. The implementation properly validates the value against an allow-list of ('low', 'high', 'auto') before including it in the API payload, which prevents injection of arbitrary values. The additional_properties attribute is always initialized as a dict in Content.__init__, so .get() is safe without a hasattr guard. Tests cover all valid values, missing detail, invalid strings, and non-string types. No security or reliability issues found.

✓ Test Coverage

The test coverage for the new detail field in image_url is thorough and well-structured. The test covers all three valid detail values ('high', 'low', 'auto'), the absence of detail, an invalid string value, and a non-string value. Assertions are meaningful — they verify both inclusion and exclusion of the detail key. One minor gap: all tests use Content.from_uri with external URIs (type='uri'), but the code path also matches type='data' (data URIs for images). Adding a test with a data URI image would strengthen coverage of that branch. This is a suggestion, not a blocking issue.

✗ Design Approach

The change correctly threads the OpenAI detail field through additional_properties for image content in the Chat Completions client. However, two design-level problems exist: (1) the implementation is inconsistent with the parallel code in _responses_client.py, which always emits detail defaulting to "auto" without validation, while this code silently drops any value not in a hardcoded allowlist; and (2) the hardcoded allowlist ("low", "high", "auto") is fragile—if OpenAI adds a new valid value, valid caller-supplied strings will be silently discarded rather than passed through to the API. The better approach is either to pass any string value through unconditionally (let the API enforce validity, consistent with how this codebase handles similar pass-through fields) or at minimum align with _responses_client.py's behavior so both clients treat the same Content object identically.

Flagged Issues

• Inconsistency between _chat_client.py and _responses_client.py: the responses client always emits detail (defaulting to "auto"), while the chat client silently drops detail when absent or unrecognised. The same Content object produces different API payloads depending on which client is used.
• The hardcoded allowlist ("low", "high", "auto") will silently discard any new valid value OpenAI introduces in the future. The value should be passed through as-is (any non-None string), leaving validation to the API.

Suggestions

• Consider adding a test case using a data URI image (e.g., data:image/png;base64,...) with the detail property to cover the "data" branch of the "data" | "uri" match, ensuring the detail field works for both content types.

---

Automated review by eavanvalkenburg's agents