[dotnet][1.0.1] Custom client tool (AIFunction) never invoked - runtime returns opaque "Tool execution failed" without dispatching to the registered handler

[mit2nil]

Summary

A registered custom client tool (a Microsoft.Extensions.AI AIFunction, exposed via SessionConfig.Tools + AvailableTools = new ToolSet().AddCustom("*")) is never invoked. The model emits a valid tool call with valid arguments and the PreToolUse hook fires, but the runtime then reports PostToolUseFailure with the opaque message "Tool execution failed" and a failed ToolExecutionComplete — without ever calling the tool's AIFunction.InvokeAsync / DelegatingAIFunction.InvokeCoreAsync.

The same code/config works in a single‑user local run but fails consistently in a deployed multi‑user Linux server.

Environment

• GitHub.Copilot.SDK (.NET): 1.0.1
• Mode: CopilotClientMode.Empty (in‑process; no CLI subprocess)
• Provider: Azure OpenAI BYOK (provider.type = "azure"), wire API responses, api‑version 2025-04-01-preview
• Model: an Azure OpenAI chat model with tool calling (GPT‑5‑class)
• Runtime: .NET 8
• Failing: Linux x64 containers, long‑running multi‑user server (concurrent sessions)
• Working: Windows, single‑user local session — identical provider/model/tool config
• Logging: CopilotClientOptions.LogLevel = "all", Logger bridged to host ILogger

Observed event sequence

For a single failing tool call the SDK emits (in order):

1. AssistantMessage / AssistantReasoning — model decides to call the custom tool.
2. ToolExecutionStart → HookStart/HookEnd → Hook.PreToolUse — pre-tool hook runs and succeeds (tool name + input args present and correct).
3. PermissionRequested → PermissionCompleted — permission check succeeds.
4. ExternalToolRequested — runtime requests the client tool (args still valid).
5. Hook.PostToolUseFailure — res = tool_invocation_failed, Error = "Tool execution failed" (21 chars; no exception type/message/stack).
6. ExternalToolCompleted → HookStart/HookEnd.
7. ToolExecutionComplete — res = tool execution failed.
8. AssistantTurnEnd.

The registered AIFunction handler is never entered between steps 4 and 5. The model then retries and fails identically (a second PreToolUse → ExternalToolRequested → PostToolUseFailure cycle).

Trace (SDK event names only — no identifiers/payloads)

Both red rows are the failure: Hook.PostToolUseFailure (tool_invocation_failed) and SdkEvent.ToolExecutionComplete (tool execution failed). Note Hook.PreToolUse and PermissionCompleted immediately above succeed, and ExternalToolRequested/ExternalToolCompleted bracket the failure — yet the client handler is never called.

How we know the handler is never invoked

• Every custom tool is wrapped in a DelegatingAIFunction that logs on entry (Debug) and on throw (Error) inside InvokeCoreAsync. With the host and SDK at trace/all verbosity, neither the entry nor the throw log is ever emitted for the failing call.
• The handler's own first‑line "invoking" log (Information) is also absent.
• ⇒ The failure happens upstream of the handler, inside the runtime's client‑tool dispatch.

Arguments are valid (host‑side binding ruled out)

The model produced valid args on multiple retries, e.g.:

{"RequiredStringProp":"<short text>"}
{"RequiredStringProp":"<short text>","OptionalProp":null}

The single required parameter is always present; optional params carry = null defaults. So this is not a host‑side argument‑binding / schema‑validation failure (those would surface inside InvokeCoreAsync, which is never reached).

Observability gap

The only failure signal the host receives is PostToolUseFailureHookInput.Error.ToString() == "Tool execution failed" — no exception, inner error, stack, or dispatch reason. Even with LogLevel = all and Logger bridged to ILogger, the runtime emits no diagnostic explaining why the client tool was not dispatched. (Related observability gap: #1220.)

Questions for maintainers

1. Under what conditions does the runtime fail a client tool with "Tool execution failed" without invoking the registered handler? (dispatch/lookup/marshaling failure, callId correlation, multi‑session tool‑registry issues, etc.)
2. Where does the runtime log the underlying reason, and how can we surface it (config / log category / event) from the .NET SDK in a server deployment? LogLevel="all" + bridged Logger did not surface it.
3. Is there a known difference between single‑session and concurrent multi‑session (multi‑user server) client‑tool dispatch in 1.0.1?
4. Possibly related: Anthropic BYOK provider handles tool execution internally — tool.call JSON-RPC never sent to python sdk #709 (BYOK provider handling tool execution internally / not delegating tool.call to the SDK — reported for Anthropic/Python, where it was noted OpenAI BYOK worked). Is there an analogous path for Azure OpenAI BYOK with the responses wire API on the .NET runtime?

Minimal repro sketch (generic)

var client = factory.Create(new CopilotClientOptions
{
    Mode = CopilotClientMode.Empty,
    LogLevel = new CopilotLogLevel("all"),
    Logger = logger,
});
await client.StartAsync();

AIFunction tool = AIFunctionFactory.Create(
    ([Description("Short required text")] string requiredProp,
     [Description("Optional")] string? optionalProp = null,
     CancellationToken ct = default) =>
    {
        logger.LogInformation("HANDLER ENTERED"); // never logged in the failing env
        return ValueTask.FromResult<object?>("ok");
    },
    name: "my_custom_tool",
    description: "Example custom client tool");

var config = new SessionConfig
{
    Model = "<azure-openai-deployment>",   // Azure OpenAI BYOK, wireApi = responses
    Tools = new List<AIFunctionDeclaration> { tool },
    AvailableTools = new ToolSet()
        .AddBuiltIn(BuiltInTools.Isolated)
        .AddCustom("*"),
    Hooks = /* PreToolUse + PostToolUse + PostToolUseFailure handlers */,
};

// Drive a prompt that makes the model call my_custom_tool.
// Observed: PreToolUse fires, then PostToolUseFailure Error="Tool execution failed";
// "HANDLER ENTERED" is never logged.

Note: we cannot share a fully self‑contained repro yet — it reproduces only in our concurrent multi‑user Linux server and not in a single‑user local session with identical provider/model/tool config. Happy to provide more detail, traces (privately), or test a patch.

[mit2nil]

Update — additional diagnostics: load-independent, intermittent, biased to the first client tool

Refining the original report with more data from aggregated server logs (7–14 day window):

• Not version-specific. Same failure on both 1.0.0 and 1.0.1 — not a regression introduced by 1.0.1.
• Not load/concurrency-related, and reproduces across multiple independent deployments. It reproduces in three separate pre-production environments, each at near-zero load (low single-digit tool calls/day) — there is no concurrency pressure when it fails. Reproducing across independent clusters/pods/nodes also rules out a single bad instance. (This supersedes the earlier "multi-user server" framing — load is not the trigger.)
• Intermittent, not a clean 100%. On the same day and same deployment we see both successful dispatches and failures: some calls where our registered AIFunction handler was invoked (it logs on entry), alongside calls that failed with the opaque "Tool execution failed" and no handler entry. So the runtime still dispatches client tools correctly sometimes, under identical conditions.
• Appears biased to the first client-tool invocation of a session. Manual repros that always begin by calling the first custom tool see that first call fail almost every time, while later tools succeed in aggregate. This looks like a registration/correlation timing race on the first tool.call after a session/connection is established — not a problem with any specific tool, its arguments (validated well-formed), or the model output.

Net

Evidence points to a nondeterministic race in the runtime's client-tool dispatch: the registered AIFunction.InvokeAsync / DelegatingAIFunction.InvokeCoreAsync is simply never invoked, and only the opaque 21-char "Tool execution failed" reaches the PostToolUseFailure hook. It is independent of SDK version (1.0.0 & 1.0.1), OS/container base, model, arguments, and load.

Questions

1. Is there a known race in client-tool dispatch around the first tool.call of a session in mode: "empty" (tool-registry or callId→handler correlation not fully established when the first call arrives)?
2. Where does the runtime log the dispatch decision/failure reason? Even at max verbosity with the SDK Logger bridged to our ILogger, we only ever receive the 21-char "Tool execution failed" — no dispatch/correlation diagnostics from the runtime itself.
3. Is there a recommended ordering/await between createSession (with custom tools registered via AvailableTools/AddCustom) and the first send, to guarantee client-tool handlers are fully registered before the model can issue a tool.call?

Happy to provide structured logs or test a patch/diagnostic build.