[Self-hosting]: Tool call check fails for OpenAI compatible endpoint

[DarthAffe]

Preflight

• I searched existing issues and docs before opening this.
• I removed secrets, tokens, private keys, customer data, and private repository details from logs and screenshots.
• I included the exact command, service, or upgrade step that failed.

Summary

On the setup-wizard at "Configure inference provider" i'm trying to connect to a OpenAI-compatible endpoint.
The endpoint is served by Unsloth desktop (I've tried textgenwebui as well) using the Qwen3.8-27B model (which supports tool-calling).
The setup procedure sends the prompt "Reply with pong." which is (correctly?) answerd with "pong". The setup then complaints "Found 1 model that meets Roomote's 7B minimum, but none support the required tool calling. The model streamed a response but did not call the required tool. Choose a model with OpenAI-compatible tool calling."

Why should that prompt cause a tool call, what would be the expected response and is there a known way/workaround/configuration to get around this issue?

Problem area

Fresh install

Installation method

Docker Compose

Roomote version

v0.45.1

Previous version

not applicable

Environment

Clean debian 13.6, roomote installed using the install-script.

Command or upgrade path

Setup - Configure inference provider
configuring a OpenAI-compatible endpoint

Expected behavior

Accepting the model Qwen3.8-27B since it supports tool-calling.

Actual behavior

The model get's rejected with the message "Found 1 model that meets Roomote's 7B minimum, but none support the required tool calling. The model streamed a response but did not call the required tool. Choose a model with OpenAI-compatible tool calling."

Diagnostics output

No response

Logs

Workarounds tried

Changing from Unsloth desktop to textgenwebui as the API-provider and retrying multiple times - the issue is 100% persistent.
Edit: I also tried ollama now which leads to the same behavior.

[roomote-community]

I've analyzed this issue. The prompt is only one part of the request: Roomote also sends a ping function definition and forces that function with OpenAI's tool_choice field. The expected streamed response therefore contains an assistant tool_calls delta naming ping, for example:

data: {"choices":[{"delta":{"tool_calls":[{"function":{"name":"ping","arguments":"{}"}}]}}]}

A streamed text response containing pong shows that the endpoint handles chat streaming, but not the structured OpenAI tool-call contract Roomote needs. This can happen when a server's tool-call parser/chat template is not enabled or when its OpenAI-compatible layer ignores tools or forced tool_choice. Bypassing this check would allow setup to finish but would leave Roomote tasks unable to use tools reliably.

Here's my proposed implementation plan:

1. Update apps/web/src/trpc/commands/task-models/local-provider-discovery.ts so the qualification message explicitly asks the model to call ping, while retaining the forced tool_choice, and parse the SSE deltas structurally instead of accepting raw substring matches.
2. Return distinct actionable diagnostics for a plain-text answer, malformed tool-call stream, wrong function name, and provider HTTP rejection. The text-only case should point operators to the server's tool-call parser, tool-aware chat template, and forced-tool-choice configuration.
3. Update apps/web/src/app/(onboarding)/setup/StepInferenceProvider.tsx as needed to preserve and present that specific qualification error instead of leaving the operator with only the generic model-support conclusion.
4. Expand apps/web/src/trpc/commands/task-models/local-provider-discovery.test.ts to assert the complete request contract and cover chunked valid tool calls, text-only pong, malformed deltas, and the wrong function. Keep the setup-wizard error coverage in StepInferenceProvider.client.test.tsx synchronized with the revised guidance.
5. Update apps/docs/providers/inference/openai-compatible.mdx to document the qualification request, the expected tool_calls response, and troubleshooting for servers such as Unsloth Desktop and text-generation-webui: model-level tool support alone is insufficient unless the serving layer exposes structured OpenAI-compatible tool calls.

This should make the setup check unambiguous and provide a direct configuration path without weakening a capability Roomote requires at runtime.

Please tag @roomote if you'd like me to implement this, or reply with feedback on the plan.

[DarthAffe]

@roomote based on this analysis I investigated a bit more and ran a llama.cpp server directly.
It outputs "Wrong type supplied for parameter 'tool_choice', using default value: [json.exception.type_error.302] type must be string, but is object" when roomote sends it's test request.
Any known way to get around this? What kind of openai compatible endpoint is supposed to work?

[roomote-community]

I saw the mention, but I need your GitHub account linked to Roomote before issue and PR comments can start work here. Link it from Settings -> Linked Accounts and then mention me again. If you'd like an invite to join the community Roomote, please send MrUbens a message on Discord: http://discord.gg/roomote

[mrubens]

@roomote please answer

[roomote-community]

I'm on it. I started a task for this issue, and I'll keep updates here.

See task

[roomote-community]

That llama.cpp log identifies the immediate incompatibility.

Roomote v0.45.1 sends the named-function form of OpenAI's tool_choice:

"tool_choice": {
  "type": "function",
  "function": { "name": "ping" }
}

That is a valid OpenAI Chat Completions request, but llama.cpp's OpenAI-compatible endpoint currently parses tool_choice as a string and supports only "auto", "none", or "required". Its warning means it discarded Roomote's named choice and fell back to "auto"; with the current prompt, the model then reasonably generated plain-text pong instead of a structured call.

For this qualification request, "tool_choice": "required" is equivalent because Roomote provides exactly one tool (ping). There is no setting in Roomote v0.45.1 to change the probe, so the practical options today are:

• use an endpoint that supports named function choice; vLLM explicitly supports {"type":"function","function":{"name":"..."}}
• put a request-transforming proxy in front of llama.cpp that rewrites this one-tool named choice to "required"
• if building Roomote yourself, change the qualification probe's tool_choice to "required"

For llama.cpp, also run a current build with Jinja/tool-call parsing enabled and a tool-aware template for the model. Its function-calling guide recommends --jinja and checking /props for chat_template or chat_template_tool_use. That configuration is still needed after the request-shape workaround so the response is emitted as choices[].delta.tool_calls[], not ordinary text.

The endpoint Roomote expects must therefore support all of: /v1/chat/completions, SSE streaming, tools, forced tool choice, and streamed OpenAI-style tool_calls deltas. Merely loading a model that can reason about tools is not sufficient if the serving layer implements only a subset of that protocol.

References:

• Roomote probe: https://github.com/RooCodeInc/Roomote/blob/develop/apps/web/src/trpc/commands/task-models/local-provider-discovery.ts#L482-L560
• llama.cpp parser: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/server-common.cpp#L1132-L1150
• llama.cpp function-calling setup: https://github.com/ggml-org/llama.cpp/blob/master/docs/function-calling.md
• vLLM named function calling: https://docs.vllm.ai/en/stable/features/tool_calling/#named-function-calling

This is worth fixing on the Roomote side: using "required" in the one-tool qualification probe should retain the capability check while also supporting servers such as llama.cpp.

[sizzlecar]

I maintain Ferrum. I ran Roomote v0.45.1's exact qualification payload unchanged against Ferrum 0.8.4 on an M1 Max with the supported Qwen3.5 4B Q4_K_M model.

The response was HTTP 200 with content-type: text/event-stream, followed by:

{"delta":{"tool_calls":[{"function":{"name":"ping","arguments":"{\"status\":\"pong\"}"}}]}}

It then emitted finish_reason: "tool_calls" and one [DONE]. So Ferrum is one local endpoint whose current API surface accepts the named-function form Roomote sends.

Scope note: I tested the exact wire request, not the full Roomote UI. Ferrum's published Metal model matrix also does not currently include your Qwen3.8-27B GGUF, so this does not solve the llama.cpp parser behavior or imply support for that artifact. It does confirm that the qualification contract itself works with a local endpoint and a supported model.