Skip to content

node: retry reaching the MCP provider, send tools/call once - #856

Merged
aojea merged 2 commits into
google:mainfrom
mrunalpendem123:fix/mcp-no-retry-on-answered-error
Nov 21, 2026
Merged

aojea merged 2 commits into
google:mainfrom
mrunalpendem123:fix/mcp-no-retry-on-answered-error

Conversation

@mrunalpendem123

@mrunalpendem123 mrunalpendem123 commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

What

Two commits, as agreed in the review thread:

  1. node: retry reaching the MCP provider, send tools/call once. The retry loop moves from CallMCPTool to connectMCPSessionWithRetry, which retries only ConnectMCPSession: dial, relay, auth frame, provider check and MCP initialize. tools/call is sent once, after the loop, and any error goes back to the caller. The existing stops for ErrAuthRejected, ErrProviderNotGranted and a non-transient authorizer limit are unchanged. The loop no longer sleeps after its last attempt.
  2. node, api: answer a task denial with JSON-RPC 1001 and say what it means. authorizeMCPRequest answers a task denial with api.JSONRPCCodeUnauthorized (1001) instead of CodeInvalidRequest (-32600), and the message now says that the task does not allow the call, that nothing was sent to the service, and that the same call under the same task is denied again. The message keeps its current start, so the regexes in development/examples/*/audit.py and dashboard.py, and the record.sh expectation, still match.

Tests

  • TestCallMCPTool_ToolCallIsSentOnce (new): the backend runs the tool and the connection drops before the reply. On main it fails with backend received tools/call 3 times, want 1 (7.1s); with the change it passes in 0.06s.
  • TestCallMCPTool_TaskDeniedToolIsNotRetried (new): a tool outside the caller's task comes back once, with code 1001 and the new message, and the same task still reaches the tool it allows. On main it fails with failed after 3 retries: ... (7.1s).
  • A 60-call burst against the MCP handshake limiter still succeeds 60/60, with the 30 reset streams recovered by the connect retry (local probe, not added).
  • go test -race ./internal/node/ and go test ./api/... ./internal/... pass, except TestListenLocalSocket, which fails the same way on main on macOS, and the envoy unix-socket test, which hits the macOS socket path limit. Both need a short TMPDIR locally.
  • golangci-lint v2.14.0 reports 0 issues, and the deadcode check in hack/lint.sh is clean.

Docs

The sample output in private-diy-stack.md and multi-tier-guardrails.md is updated: no failed after 3 retries: prefix, and the new message ending.

Trade-off

A stream that drops after tools/call is sent is no longer retried inside the node; the caller gets the error and decides.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces explicit handling for JSON-RPC errors in CallMCPTool to prevent retries, accompanied by a new integration test to verify this behavior. The review feedback points out a violation of the style guide's logging rules, specifically that the warning log for the JSON-RPC error should include the targetPeer identifier to ensure proper auditability.

Comment thread internal/node/mcp.go Resolved
Comment thread internal/node/mcp.go Outdated
Comment on lines +234 to +238
var rpcErr *jsonrpc.Error
if errors.As(err, &rpcErr) {
logger.Warnf("[MCP] Tool call to %s answered with an error, not retrying: %v", targetPeer, err)
return nil, err
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont' know if we should go so deep down in the stack , which jsonrpc errors are retriable and which are not

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point, the loop shouldn't be sorting JSON-RPC errors. The case I hit is a tools/call that the provider refuses because the tool is outside the caller's task. That answer is the same on every try, so retrying only delays it by about 7s.

One option that doesn't look at the error: retry only failures before tools/call is sent, and never re-send it. The trade-off is that a stream dropped mid-call would no longer be retried, which may matter on relayed connections. If that isn't worth it, I'm happy to close this.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my question is , who sends this error and what type of error is? I'm unclear from the description if this is something agent mesh block or something the mcp server blocks

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AgentMesh sends it. The provider node's MCP service checks every tools/call against the caller's task token before forwarding it to the MCP server (authorizeMCPRequest, internal/node/mcp_service.go:387-394). If the tool isn't in the task's allowed_tools, the node answers itself with a JSON-RPC error (CodeInvalidRequest, task authorization denied tool "..."). The MCP server never sees the call.

So it's the same kind of refusal as ErrProviderNotGranted in #722, but it can't use a sentinel error: it happens after the handshake, once the tool name is known, so it reaches the caller as a JSON-RPC error.

You're right that the PR goes too far. It stops retries on every JSON-RPC error, including ones from the MCP server like unknown tool. I can narrow it to the node's own denial: give that denial an AgentMesh-specific error code and have CallMCPTool stop on that code only, so the loop doesn't sort any other errors.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah , we can only retry errors that are retriable

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume we added the retry logic for any reason, so we should understand if that was incorrect and we should never retry and just delegate the decision to the caller

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coming back to the option of retrying only before tools/call: I checked it on main (8474138) with tests and prototyped it locally (not pushed).

Why the retry exists. It came in with 7c47a8d (#10), when each attempt opened a new stream and sent the call. The later changes (e45dd76, b66d6ba) were about relays and reaching the peer.

main retry only the connect step
backend runs the tool, then the connection drops before the reply tool runs 3 times, EOF after 7s runs once
tool outside the task 7s, 3 sessions immediate
60 concurrent calls, 30 reset by the MCP handshake limiter 60/60 60/60
expired task, or service outside the task refused at the auth frame same

Same in 3 runs each, and the node package tests give the same result with and without the change. #375 shows the first row in practice: the backend answered tools/call "once per retry". A relayed circuit reaching its lifetime (1h by default) mid-call would do the same.

Why split by phase rather than by error. Every exit in the loop today (#722, #842, and this PR) is a refusal that happens before tools/call, in ConnectMCPSession. Once tools/call is sent, the provider may have run the tool, so whatever the error, only the caller knows if running it again is safe. Retrying only the connect step keeps those exits as they are, and the loop never sorts JSON-RPC errors.

The code still helps. With a 1001 constant in api/ for the task denial, callers (call_remote_tool, the FFI) can tell a denial from a failure, and the connect loop can stop on it if initialize is ever refused. 1001 is outside the reserved -32768 to -32000 range, and no go-sdk code uses it.

The loop also sleeps after its last attempt (1s + 2s + 4s), so every exhausted call waits 4s for nothing; the prototype skips that.

Trade-off: a stream that drops mid-call is no longer retried inside the node, and FFI callers see the error instead.

I'll push whichever way you agree on; the duplicate-run and denial cases can go in as tests.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kaisoz I like the two commits proposal: 1. split phase so retries belong to ConnectMCPSession 2. enhance the JSON-RPC error with good information so llm and user understand the problem , in this case unauthenticated .. .WDYT?

@kaisoz kaisoz Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sgtm 👍🏻

As a note, the JSON-RPC error code never gets to the llm because the MCP go-sdk builds the MCP error message from the JSON-RPC error decription. However, the error code is still valuable for both the js and python SDKs

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed the two commits as agreed:

  1. f7bf83c moves the retry to connectMCPSessionWithRetry, so only reaching the provider is retried and tools/call is sent once.
  2. b855fad adds api.JSONRPCCodeUnauthorized (1001) in api/mcp.go and uses it for the task denial. Since the model only sees the message, as @kaisoz noted, the message now also says the task does not allow the call, nothing was sent to the service, and the same call under the same task is denied again.

I used "unauthorized" rather than "unauthenticated": the caller's identity was verified at the auth frame, and it is the task that does not allow the call.

The old commits are replaced, so the "as before" comment from the earlier review is gone. The two new tests fail on main and pass here. Details are in the PR description.

@mrunalpendem123
mrunalpendem123 requested a review from aojea November 5, 2026 21:21
mrunalpendem123 added a commit to mrunalpendem123/agentmesh that referenced this pull request Nov 6, 2026
Review on google#856: the retry loop should not decide which JSON-RPC errors
are retriable. The previous commit stopped on every JSON-RPC error,
including ones the MCP backend returns, such as an unknown tool.

The denial that is never retriable is AgentMesh's own: the provider
node's task check (authorizeMCPRequest) refuses the request before the
backend sees it. It now answers with codeTaskDenied (-31403), an
AgentMesh code outside the range JSON-RPC reserves, instead of
CodeInvalidRequest. The message text is unchanged. CallMCPTool stops on
that code only; every other error, including any backend error, is
retried as before.
Comment thread internal/node/mcp.go Outdated
}
// The provider's task check refusing the call is a denial too. It
// answers with its own JSON-RPC code; any other JSON-RPC error, such
// as one from the MCP backend, is retried as before.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Nice, clear comment! "as before" compares the code to an older version. A reader only sees the current code, so I would drop it.

Suggested change
// as one from the MCP backend, is retried as before.
// as one from the MCP backend, is retried.

Comment thread internal/node/mcp.go Outdated
Comment on lines +234 to +238
var rpcErr *jsonrpc.Error
if errors.As(err, &rpcErr) {
logger.Warnf("[MCP] Tool call to %s answered with an error, not retrying: %v", targetPeer, err)
return nil, err
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mrunalpendem123 I think I prefer your initial idea: retry only ConnectMCPSession, and send tools/call once, after the loop. CallMCPTool gets simpler, because callMCPToolOnce can be dropped. It also comes with some benefits:

  • No new error code, so the loop does not sort JSON-RPC errors.
  • Retries happen only before tools/call, where they are safe.
  • The task denial, and errors such as unknown tool, go back to the model at once.
  • A tool never runs twice. Today, if the stream drops after the provider node forwarded tools/call, the retry sends it again.
  • Dial, relay and handshake failures are still retried.

The trade-off is the one you named: a stream that drops during a call is not retried inside the node. But the model sees the error and decides, because only the model knows if a second run is safe.

@aojea WDYT?

CallMCPTool retried the whole call: open a stream, hand over the task
biscuit, run the MCP handshake, send tools/call. A failure after
tools/call was sent was retried too, though the provider may already
have run the tool. If the stream dropped after the backend ran it, the
tool ran three times for one call (google#375 reported a backend answering
"once per retry"). A tool outside the caller's task, which the
provider's task check answers inside tools/call, came back after three
attempts and about 7 seconds.

Only reaching the provider is retried now: dial, relay, auth frame,
provider check and MCP initialize, in connectMCPSessionWithRetry.
Nothing has reached the MCP backend at that point, so a retry cannot
repeat work. The stops for a denial, an ungranted provider and a
non-transient authorizer limit stay where they were. tools/call is sent
once, after the loop, and any error goes back to the caller, which is
the one that knows whether running the tool again is safe.

The loop no longer sleeps after its last attempt, which made every
exhausted call wait 4 more seconds before returning.
authorizeMCPRequest answered a request outside the caller's task with
CodeInvalidRequest (-32600), the code for a malformed request, so a
client could not tell a denial from a protocol error. The message named
the rule that denied the call but not what that means for the caller.

api.JSONRPCCodeUnauthorized (1001) is the code for it now, in a new
section of api/mcp.go for the JSON-RPC codes AgentMesh itself returns.
It is outside the -32768 to -32000 range JSON-RPC 2.0 reserves, and the
MCP go-sdk does not use it. Clients reading the code, such as the js and
python SDKs, can tell a denial apart from a failure.

A calling model sees only the message, because the MCP go-sdk builds its
error from it. The message keeps its current start, which the examples
match on, and adds that the task does not allow the call, that nothing
was sent to the service, and that the same call under the same task is
denied again.
@mrunalpendem123
mrunalpendem123 force-pushed the fix/mcp-no-retry-on-answered-error branch from 64b82fb to b855fad Compare November 21, 2026 03:10
@mrunalpendem123 mrunalpendem123 changed the title node: do not retry a tool call the provider answered with an error node: retry reaching the MCP provider, send tools/call once Nov 21, 2026
@aojea

aojea commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator

Thanks

@aojea
aojea merged commit 3f6e3ed into google:main Nov 21, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants