Define the API for in-page Agents to use a site's declared tools

[khushalsagar]

The modelContext API currently only provides APIs for a site to declare tools. We also need an API surface for an Agent embedded on the site to use these tools. Technically this could all be managed within the site but having a dedicated web API is better for multiple reasons. #43 (comment) has a detailed explanation, adding a quick summary below:

• We can a common registry of tools from all sources for all Agents.
• The browser can help mediate which Agent is accessing the site's functionality at a time.

We should first resolve on the fact that this API is necessary.

In terms of API details, a few things on my mind:

• It would be nice to have one interface for declaring functionality (right now tools, likely context going forward) since that's shared across all Agents. And a separate one for an Agent to use this functionality.
• Should we assume that there's only one "in-page Agent" on a site? That's related to the agentIdentifier question on Managing action specific permissions #44. The browser can either allocate a "in-site agent" identifier if the site can only have one agent. Or we need an API for the site to request these.
• We also added an interface called agent for the dependencies the site needs. I now think these are more like user-agent/browser dependencies. The site asks the browser for something which may need to ask the Agent.

[43081j]

isn't this just the web API we're proposing overall?

i.e. navigator.modelContext will have a means to call one of the registered tools and all agents, in page or not, will call that

maybe the name of this issue doesn't need to include "for in-page Agents" since its just the tool calling API overall, for all agents.

[khushalsagar]

Meeting notes from TPAC discussion here.

isn't this just the web API we're proposing overall?

No, the API resolved so far is sufficient for Agents accessing these tools through browser internal APIs (browser's built-in Agent, extensions etc). That's a detail specific to each browser implementation, it doesn't need to be defined by the spec.

[43081j]

No, the API resolved so far is sufficient for Agents accessing these tools through browser internal APIs (browser's built-in Agent, extensions etc). That's a detail specific to each browser implementation, it doesn't need to be defined by the spec

this seems an odd one. an in-page agent would call the API you linked (navigator.modelContext) just the same way an agent using CDP to access the current page would.

can you clarify why you think an in-page agent can't call this?

we pretty much do this today with an in-page agent the way im describing, using the API definition you linked

[43081j]

aha i think i understand the gap now

Chrome happens to have chosen to implement their tool execution method internally (i.e. it isn't part of the navigator API)

Meanwhile, the proposal suggests using DOM events to execute tools here.

Separately, MCP-B, our implementation, and others simply have an executeTool method.

I would argue that Chrome is the outlier here. The navigator API should have a means to execute calls (a method or DOM events) and that is what all consumers use, whether they are "in-page" or not.

Chrome has implemented this in a way that means ONLY chrome can execute registered tools (internally). that obviously doesn't fly. Chrome should be calling the same code path that a public executeTool method calls. These tools can't be constrained to the browser's internals, its just implemented in a way that doesn't agree with the proposal.

[KenjiBaheux]

I'm not sure we can assume that what is exposed via WebMCP would be a fit for all agents.

The site might want to expose some tools to the browser agent, and a different set to other agents.
There might also be a need for restricting what is exposed to agents in cross origin Iframes, if anything.

Finally, agents that are implemented with on-device means (e.g. built-in AI) won't have the same capabilities as the browser's agent. For this particular case, it seems better to expose the tools via the built-in AI APIs so that the underlying model can properly communicate what it supports (e.g. context window size, and other aspects if needed), etc.

[43081j]

The site might want to expose some tools to the browser agent, and a different set to other agents.

This sounds like part of the tool definition rather than anything to do with the browser API.

An agent isn't always an LLM agent either. It could be me (a human using DevTools), a Chrome extension, a browser automation script, etc.

We're defining a new web API here (navigator.modelContext). Making that change shape based on who is running the browser feels like a can of worms and something no other web API has ever done.

I would've expected that navigator.modelContext has a way to define, list and execute tools (and it does, just not in Chrome's implementation).

Chrome's own agent calls some ExecuteTool function internally, that is fine. But Chrome also needs to implement executeTool in the JS realm, which calls their internal ExecuteTool under the hood.

Importantly there is one API whether you're in-page or not

[khushalsagar]

I agree that usage by in-page Agents (i.e. run by scripts running within the page’s JS context) should be in scope. That’s why I flagged this issue for discussion at TPAC and clarified the 2 motivations for this:

Better ergonomics for the site by having a standardized unified API surface across all Agents.

The browser can mediate access across multiple Agents, avoiding issues with concurrent access.

The group wasn’t convinced a) was necessary for the MVP since it can be polyfilled. There was agreement that b) would need a browser API but the surface for that can be smaller and doesn’t need to consider tool calling. But the group wanted to hear more concrete use-cases before deciding what to do for b).

That said, I’m happy to sketch the API proposal for this in parallel. I had something like this in mind:

dictionary ModelContextTool {
  required DOMString name;
  required DOMString description;
  object inputSchema;
  AnnotationsDict annotations;
};

dictionary ModelContextClientOptions {
  // Defined [here](https://github.com/webmachinelearning/webmcp/blob/main/docs/proposal.md#agent). Provides dependencies from the Agent.
  required Agent agent;
};

interface ModelContextClient : EventTarget {
  // The set of currently registered tools.
  readonly attribute sequence<ModelContextTool> tools;

  // Notification for when the set of tools changes.
  attribute EventHandler ontoolschanged;

  // Executes a tool with the given name.
  Promise<any> execute(DOMString name, DOMString arguments);
};

interface ModelContext : EventTarget {
  ModelContextClient createClient(ModelContextClientOptions options);
};

Here’s some sample code:

const siteAgent = {
  requestUserInteraction: async (callback) => {
    // Add any logic to unblock user interaction with the site.
    return await callback();
  }

// Create a new client to use registered tools. The options list can be expanded to configure the set of tools exposed.
let client = navigator.modelContext.createClient({agent: siteAgent});

console.log(client.tools); // Prints the current list of known tools

function runATool() {
  If (client.tools.length > 0) {
    myTool = client.tools[0];
    client.execute(mytool.name, "{arg1: arg1_value, arg2: arg2_value}");
  }
}

client.addEventListener("toolschanged", () => {
  // New tools added or removed.
  runATool();
}

Should we decide to go with the event proposal here, the browser can do it automatically by changing how execute is implemented.

---

I’m not sure about agent directly coming from the client. We’ve talked about adding state like agentIdentifier and it should ideally come from the browser. In general, we’ll have more flexibility if the browser supplied the agent interface and proxied calls as necessary to the client. For example:

interface ModelContextClient : EventTarget {
  …

  // Executes a tool with the given name.
  Promise<any> execute(DOMString name, DOMString arguments);

  // Invoked when the execution of a tool is blocked on the user interacting with
  // the site. Returns a promise which resolves when user interaction can start.
  Promise<undefined> requestUserInteraction();

  // Notifies when the user interaction has finished.
  undefined endUserInteraction();
};

[43081j]

The group wasn’t convinced a) was necessary for the MVP since it can be polyfilled. There was agreement that b) would need a browser API but the surface for that can be smaller and doesn’t need to consider tool calling. But the group wanted to hear more concrete use-cases before deciding what to do for b).

this is a real shame. what chrome has implemented is basically a nerfed webmcp API

let client = navigator.modelContext.createClient({agent: siteAgent});

You've lost me here, how come there's a concept of a "client" now? This is just a very simple registry of tools, non-Chrome implementations already have navigator.modelContext.executeTool which is what I would've expected.

here's some sample code corrected to what I would've expected:

console.log(await navigator.modelContext.listTools());

navigator.modelContext.executeTool("some-tool", {arg1: arg1_value, arg2: arg2_value});

you and other Googlers are trying to design around something others (including me) are missing. is it concurrency?

basically why you have a client, why you're hesitant to expose executeTool, why any of this isn't just the API.

is it all because you're trying to solve concurrency problems?

i'd love to better understand this as it feels like an extremely limited, jailed implementation that only Chrome can use internally. concurrency seems like a userland problem and a distraction from the proposal. we've had concurrency to deal with in MCP since the beginning because filesystems are stateful, and userland tools and agents solved that, not the MCP spec.

on a side note: this stuff matters to me a lot but i seem to be on a totally different page to the people making the decisions here somehow. neither discord nor github is ever going to be efficient enough to get to the bottom of that. if any of you can one day hop on a call with me, i'd really appreciate that. or some existing call i can join

[khushalsagar]

The modelContextClient interface makes it feasible to know when a Agent is active to deal with concurrency.

It also allows tying tool execution to an Agent. As @KenjiBaheux mentioned, the author wouldn't want to expose the same set of tools to all Agents. For example, it's possible the in-page Agent can be backed by multiple models and the tool set is configurable based on that.

And lastly it makes it easier to support functionality like elicitation which needs both the browser and the Agent. For example, when the author calls requestUserInteraction on agent, the browser needs to be notified to foreground the tab and the Agent needs to be notified as well (showing something in its UI for example).

The modelContextClient proposal adds the functionality you've mentioned: listing current tools and executing them. I understand that this looks more complicated than you were envisioning with simple function calling but the complexity is warranted to keep the API extensible.

WebML CG has bi-weekly meetings, please feel free to join. We can take up this issue on a call. @anssiko FYI.

[43081j]

i think that is where we're not currently aligned. you're trying to solve concurrency in the webmcp proposal rather than userland/agent code.

the only reason there's a client, no executeTool and the rest of this misalignment/confusion is all to try tackle concurrency it seems.

the author wouldn't want to expose the same set of tools to all Agents. For example, it's possible the in-page Agent can be backed by multiple models and the tool set is configurable based on that.

why is this a concern of the page author?

if i run the salesforce UI, i want to expose a listTickets function or some such thing to provide better UX for people trying to interact with my site using agents. i shouldn't have to care what agent YOU, the user, are using, and what models that has. that's not my problem, there's countless agents out there and many more models.

now i have to write my code in a special way to account for different types of agent and models. you've just moved the burden to the web author instead of the agent, which seems like a poor choice of direction to me.

i want to write this:

navigator.agentContext.registerTool('list-tickets', {
  schema,
  id,
  execute() {
    // do some basic, typical async calls to get tickets in my current session and return them as a CallToolResult
  }
});

but i can't anymore because now i need to take into account what agents and models might call my code...