Repository navigation
node: retry reaching the MCP provider, send tools/call once - #856
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces explicit handling for JSON-RPC errors in CallMCPTool to prevent retries, accompanied by a new integration test to verify this behavior. The review feedback points out a violation of the style guide's logging rules, specifically that the warning log for the JSON-RPC error should include the targetPeer identifier to ensure proper auditability.
| var rpcErr *jsonrpc.Error | ||
| if errors.As(err, &rpcErr) { | ||
| logger.Warnf("[MCP] Tool call to %s answered with an error, not retrying: %v", targetPeer, err) | ||
| return nil, err | ||
| } |
There was a problem hiding this comment.
I dont' know if we should go so deep down in the stack , which jsonrpc errors are retriable and which are not
There was a problem hiding this comment.
Fair point, the loop shouldn't be sorting JSON-RPC errors. The case I hit is a tools/call that the provider refuses because the tool is outside the caller's task. That answer is the same on every try, so retrying only delays it by about 7s.
One option that doesn't look at the error: retry only failures before tools/call is sent, and never re-send it. The trade-off is that a stream dropped mid-call would no longer be retried, which may matter on relayed connections. If that isn't worth it, I'm happy to close this.
There was a problem hiding this comment.
my question is , who sends this error and what type of error is? I'm unclear from the description if this is something agent mesh block or something the mcp server blocks
There was a problem hiding this comment.
AgentMesh sends it. The provider node's MCP service checks every tools/call against the caller's task token before forwarding it to the MCP server (authorizeMCPRequest, internal/node/mcp_service.go:387-394). If the tool isn't in the task's allowed_tools, the node answers itself with a JSON-RPC error (CodeInvalidRequest, task authorization denied tool "..."). The MCP server never sees the call.
So it's the same kind of refusal as ErrProviderNotGranted in #722, but it can't use a sentinel error: it happens after the handshake, once the tool name is known, so it reaches the caller as a JSON-RPC error.
You're right that the PR goes too far. It stops retries on every JSON-RPC error, including ones from the MCP server like unknown tool. I can narrow it to the node's own denial: give that denial an AgentMesh-specific error code and have CallMCPTool stop on that code only, so the loop doesn't sort any other errors.
There was a problem hiding this comment.
yeah , we can only retry errors that are retriable
There was a problem hiding this comment.
I assume we added the retry logic for any reason, so we should understand if that was incorrect and we should never retry and just delegate the decision to the caller
There was a problem hiding this comment.
Coming back to the option of retrying only before tools/call: I checked it on main (8474138) with tests and prototyped it locally (not pushed).
Why the retry exists. It came in with 7c47a8d (#10), when each attempt opened a new stream and sent the call. The later changes (e45dd76, b66d6ba) were about relays and reaching the peer.
| main | retry only the connect step | |
|---|---|---|
| backend runs the tool, then the connection drops before the reply | tool runs 3 times, EOF after 7s | runs once |
| tool outside the task | 7s, 3 sessions | immediate |
| 60 concurrent calls, 30 reset by the MCP handshake limiter | 60/60 | 60/60 |
| expired task, or service outside the task | refused at the auth frame | same |
Same in 3 runs each, and the node package tests give the same result with and without the change. #375 shows the first row in practice: the backend answered tools/call "once per retry". A relayed circuit reaching its lifetime (1h by default) mid-call would do the same.
Why split by phase rather than by error. Every exit in the loop today (#722, #842, and this PR) is a refusal that happens before tools/call, in ConnectMCPSession. Once tools/call is sent, the provider may have run the tool, so whatever the error, only the caller knows if running it again is safe. Retrying only the connect step keeps those exits as they are, and the loop never sorts JSON-RPC errors.
The code still helps. With a 1001 constant in api/ for the task denial, callers (call_remote_tool, the FFI) can tell a denial from a failure, and the connect loop can stop on it if initialize is ever refused. 1001 is outside the reserved -32768 to -32000 range, and no go-sdk code uses it.
The loop also sleeps after its last attempt (1s + 2s + 4s), so every exhausted call waits 4s for nothing; the prototype skips that.
Trade-off: a stream that drops mid-call is no longer retried inside the node, and FFI callers see the error instead.
I'll push whichever way you agree on; the duplicate-run and denial cases can go in as tests.
There was a problem hiding this comment.
@kaisoz I like the two commits proposal: 1. split phase so retries belong to ConnectMCPSession 2. enhance the JSON-RPC error with good information so llm and user understand the problem , in this case unauthenticated .. .WDYT?
There was a problem hiding this comment.
sgtm 👍🏻
As a note, the JSON-RPC error code never gets to the llm because the MCP go-sdk builds the MCP error message from the JSON-RPC error decription. However, the error code is still valuable for both the js and python SDKs
There was a problem hiding this comment.
Pushed the two commits as agreed:
- f7bf83c moves the retry to
connectMCPSessionWithRetry, so only reaching the provider is retried andtools/callis sent once. - b855fad adds
api.JSONRPCCodeUnauthorized(1001) inapi/mcp.goand uses it for the task denial. Since the model only sees the message, as @kaisoz noted, the message now also says the task does not allow the call, nothing was sent to the service, and the same call under the same task is denied again.
I used "unauthorized" rather than "unauthenticated": the caller's identity was verified at the auth frame, and it is the task that does not allow the call.
The old commits are replaced, so the "as before" comment from the earlier review is gone. The two new tests fail on main and pass here. Details are in the PR description.
Review on google#856: the retry loop should not decide which JSON-RPC errors are retriable. The previous commit stopped on every JSON-RPC error, including ones the MCP backend returns, such as an unknown tool. The denial that is never retriable is AgentMesh's own: the provider node's task check (authorizeMCPRequest) refuses the request before the backend sees it. It now answers with codeTaskDenied (-31403), an AgentMesh code outside the range JSON-RPC reserves, instead of CodeInvalidRequest. The message text is unchanged. CallMCPTool stops on that code only; every other error, including any backend error, is retried as before.
| } | ||
| // The provider's task check refusing the call is a denial too. It | ||
| // answers with its own JSON-RPC code; any other JSON-RPC error, such | ||
| // as one from the MCP backend, is retried as before. |
There was a problem hiding this comment.
nit: Nice, clear comment! "as before" compares the code to an older version. A reader only sees the current code, so I would drop it.
| // as one from the MCP backend, is retried as before. | |
| // as one from the MCP backend, is retried. |
| var rpcErr *jsonrpc.Error | ||
| if errors.As(err, &rpcErr) { | ||
| logger.Warnf("[MCP] Tool call to %s answered with an error, not retrying: %v", targetPeer, err) | ||
| return nil, err | ||
| } |
There was a problem hiding this comment.
@mrunalpendem123 I think I prefer your initial idea: retry only ConnectMCPSession, and send tools/call once, after the loop. CallMCPTool gets simpler, because callMCPToolOnce can be dropped. It also comes with some benefits:
- No new error code, so the loop does not sort JSON-RPC errors.
- Retries happen only before
tools/call, where they are safe. - The task denial, and errors such as unknown tool, go back to the model at once.
- A tool never runs twice. Today, if the stream drops after the provider node forwarded tools/call, the retry sends it again.
- Dial, relay and handshake failures are still retried.
The trade-off is the one you named: a stream that drops during a call is not retried inside the node. But the model sees the error and decides, because only the model knows if a second run is safe.
@aojea WDYT?
CallMCPTool retried the whole call: open a stream, hand over the task biscuit, run the MCP handshake, send tools/call. A failure after tools/call was sent was retried too, though the provider may already have run the tool. If the stream dropped after the backend ran it, the tool ran three times for one call (google#375 reported a backend answering "once per retry"). A tool outside the caller's task, which the provider's task check answers inside tools/call, came back after three attempts and about 7 seconds. Only reaching the provider is retried now: dial, relay, auth frame, provider check and MCP initialize, in connectMCPSessionWithRetry. Nothing has reached the MCP backend at that point, so a retry cannot repeat work. The stops for a denial, an ungranted provider and a non-transient authorizer limit stay where they were. tools/call is sent once, after the loop, and any error goes back to the caller, which is the one that knows whether running the tool again is safe. The loop no longer sleeps after its last attempt, which made every exhausted call wait 4 more seconds before returning.
authorizeMCPRequest answered a request outside the caller's task with CodeInvalidRequest (-32600), the code for a malformed request, so a client could not tell a denial from a protocol error. The message named the rule that denied the call but not what that means for the caller. api.JSONRPCCodeUnauthorized (1001) is the code for it now, in a new section of api/mcp.go for the JSON-RPC codes AgentMesh itself returns. It is outside the -32768 to -32000 range JSON-RPC 2.0 reserves, and the MCP go-sdk does not use it. Clients reading the code, such as the js and python SDKs, can tell a denial apart from a failure. A calling model sees only the message, because the MCP go-sdk builds its error from it. The message keeps its current start, which the examples match on, and adds that the task does not allow the call, that nothing was sent to the service, and that the same call under the same task is denied again.
64b82fb to
b855fad
Compare
|
Thanks |
What
Two commits, as agreed in the review thread:
node: retry reaching the MCP provider, send tools/call once. The retry loop moves fromCallMCPTooltoconnectMCPSessionWithRetry, which retries onlyConnectMCPSession: dial, relay, auth frame, provider check and MCPinitialize.tools/callis sent once, after the loop, and any error goes back to the caller. The existing stops forErrAuthRejected,ErrProviderNotGrantedand a non-transient authorizer limit are unchanged. The loop no longer sleeps after its last attempt.node, api: answer a task denial with JSON-RPC 1001 and say what it means.authorizeMCPRequestanswers a task denial withapi.JSONRPCCodeUnauthorized(1001) instead ofCodeInvalidRequest(-32600), and the message now says that the task does not allow the call, that nothing was sent to the service, and that the same call under the same task is denied again. The message keeps its current start, so the regexes indevelopment/examples/*/audit.pyanddashboard.py, and therecord.shexpectation, still match.Tests
TestCallMCPTool_ToolCallIsSentOnce(new): the backend runs the tool and the connection drops before the reply. Onmainit fails withbackend received tools/call 3 times, want 1(7.1s); with the change it passes in 0.06s.TestCallMCPTool_TaskDeniedToolIsNotRetried(new): a tool outside the caller's task comes back once, with code 1001 and the new message, and the same task still reaches the tool it allows. Onmainit fails withfailed after 3 retries: ...(7.1s).go test -race ./internal/node/andgo test ./api/... ./internal/...pass, exceptTestListenLocalSocket, which fails the same way onmainon macOS, and the envoy unix-socket test, which hits the macOS socket path limit. Both need a shortTMPDIRlocally.golangci-lintv2.14.0 reports 0 issues, and thedeadcodecheck inhack/lint.shis clean.Docs
The sample output in
private-diy-stack.mdandmulti-tier-guardrails.mdis updated: nofailed after 3 retries:prefix, and the new message ending.Trade-off
A stream that drops after
tools/callis sent is no longer retried inside the node; the caller gets the error and decides.