Proposal: Cross-Platform Testing APIs or Guidelines

[hydrosquall]

Pre-submission Checklist

• I have verified this would not be more appropriate as a feature request in a specific repository
• I have searched existing discussions to avoid duplicates

Your Idea

Motivation

When we write MCP apps, we want to them to work everywhere

• Remote hosts like (ChatGPT / Claude) across (web, desktop, mobile) across operating systems (Mac, Windows, Linux)
• Locally hosts including VSCode, Goose, Cursor, Pi

Currently, it is possible to write apps where updating an app tool for 1 host will break appearance or behavior for others.

In the past 3 months of expanding support from an MCP app for Datadog from 1 host to others we encountered a range of vendor-specific differences :

1. ChatGPT or Claude both cache tool definitions at app submission time. When the underlying MCP server updates, the agent still calls the tool with "old schemas" until the app is resubmitted and re-approved. This delay led to incorrect triggers until we adapted to design around this. (This behavior doesn't exist when connecting directly to an MCP server in Goose, Codex, VSCode).
2. Whether your app's tool triggers at all depends on how the host implements tool search / tool selection (ChatGPT example)
3. Hosts vary widely in what features they support and whether they report this to the consumer. Hosts have a way to advertise which features they support, they don't always fill this in. (For example Goose supports callServerTool but didn't advertise this until recently, so when we gated on hostCapabilities it inadvertently removed a feature that should have worked).
4. Several hosts (Cursor, OpenAI do not provide the _meta field in tool results which apps depend on for having persistence between multiple tool calls, especially see given the recent removal of mcp-session-id from the July 2026 release candidate Properly clarify content model visibility - content vs structuredContent vs _meta #380 (comment) )
5. Host are not consistent in how they handle tools that were marked as app-only tools, we found both Claude Code CLI and Cursor IDE were calling tools that were meant to be only visible to the model.
6. There have been times where adding a field for 1 vendor broke the app for another. For example, a user may provide a ui.domain as requested by ChatGPT , but the format was incompatible with Claude. We have a fix now, but I want to show that this gets increasingly complicated if Gemini, Microsoft Copilot etc each have their own requirements and your app has to adjust capabilities based on the vendor

OpenAI requested a URL	Unfortunately this broke behavior in Anthropic

(I'm aware the reference host advertises a way to calculate an Anthropic-friendly URI)

7. Differences in how responsive width, insetWidth, etc caused smaller display bugs, it was ambiguous as to whether width should represent the space available vs the space currently requested by the iframe. These types of rendering issues are handled by adding user agent detection to the app client to change the logic based on the host

Each time we fixed an issue, we added a manual check to our pre-flight checklist.

We have managed to automate some of these checks, but are interested to see if there is ecosystem/documentation/API level change we can share or drive from our learnings that would help to smooth adoption of MCP apps for others in the community.

Questions / Suggestions for a path forward

For remote hosts (OpenAI, Anthropic)

• For builders of electron apps (ChatGPT, Claude): Could we recommend allow the remote debugger port to be open? Cursor and VSCode support this, it has made making testing much easier
• Could hosts offer webMCP handles to make it easier to automate tests involving MCP apps
• Is there some alternative recommended way to do automated testing of MCP apps aside from scripting on our dev accounts? We don't want our agents to run afoul of anti-scraping/bot protection mechanisms. The goal here is just to make sure our apps work, we're not trying to turn the webapp into an alternative API.

For local hosts (VSCode, Goose, OpenWebUI, Pi etc)

• Would there be interest in having a Conformance test MCP App, which you could use to quickly see check portion of the MCP app surface has been implemented, and flag common issues? (I empathize with having a big surface to support, as I had to implement a custom host without mcp-ui, and forgot to advertise hostCapabilities )

For Clients / MCP App creators

• How are you managing testing if your app works across mulitple platforms, where fixes for one don't break others? Is this pain point relatable?

Scope

• Protocol Specification
• SDK Features
• Documentation
• Developer Experience
• Other

[chelojimenez]

+1 on @hydrosquall 's remarks. apart from the differences mentioned above, we've noticed that you can be a MCP compatible host but implement subsets of the bridge methods (eg Microsoft Copilot is MCP App compliant but doesn't support app.ontoolinputpartial, it also supports Apps SDK but some methods like requestModal aren't supported). Hosts also diverge on how strictly they enforce CSP. Some grant extra script-src keywords beyond the spec default, some deny iframe nesting regardless of declared frameDomains, some emit almost no CSP directives at all...

our solution to this problem was creating an abstraction of a host that can be used to do live/programatic testing. The definition spans protocol level capabilities, CSP directives, and others (eg supports progressive tool discovery). the product is early, but it being OSS allows developers to update these templates as they encounter divergences in the wild.

clients.mp4

client-compare.mp4

[hydrosquall]

creating an abstraction of a host that can be used to do live/programatic testing

Ah this is neat! Have you found this abstraction has been able to capture all of the meaningful differences, or are there some nuances that haven't lent themselves to being emulated?

[chelojimenez]

I think that agents/harnesses have enough complexity that make them impossible to emulate fully, but this is very robust at emulating their MCP behavior given the abstraction hoists behaviors like iframe permissions or progressive discovery that have proven a very reliable smoke test

[hydrosquall]

Noting ideas for next steps from today's meeting that there are at least two separate tracks we could pursue. I'd be interested to speak / collaborate with anyone who is working on a host (in particular at OpenAI, Anthropic, Microsoft (VSCode), Goose, or Cursor) around their answers to these questions.

Absent that, we can split this discussion into 2 threads / issues based on those discussions

1. Tooling for Hosts to check conformance / users to see which features each host supports

Related work on conformance testing specific to MCP apps (currently distinct from the existing server conformance testing) if hosts would be interested:

• For web platform ( https://wpt.fyi/ ) for browser platforms today
• For web rendering: https://www.acidtests.org/

As an intermediary step , we could move towards something like a https://caniuse.com/

Related (but inactive) work at the MCP layer: modelcontextprotocol/modelcontextprotocol#1814

2. Tooling or guidelines for testing MCP Apps within each of these hosts

On this front, curious to hear more from others who are building/supporting MCP apps before trying to generalize a solution.