Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ export const metadata = {

## Configuring Azure AI Foundry

> Foundry chat models and memory embeddings require the Go agent runtime (`runtime: go`). Chat models must be available through Foundry's OpenAI-compatible chat completions API. Claude and other models are not yet supported.
> **Note:** Foundry chat models and memory embeddings require the Go agent runtime (`runtime: go`). Chat models must be available through Foundry's OpenAI-compatible chat completions API. Claude and other models are not yet supported.

The following steps use API key authentication. To authenticate without an API key, see [Workload Identity](#workload-identity).

Expand Down Expand Up @@ -61,7 +61,7 @@ spec:
systemMessage: "You are a helpful assistant."
```

4. To create the agent declaratively, save the preceding manifests as `foundry-model.yaml` and `foundry-agent.yaml`, then apply them to your cluster.
4. Save the manifests from steps 2 and 3 as `foundry-model.yaml` and `foundry-agent.yaml`, then apply them to your cluster.

```bash
kubectl apply -f foundry-model.yaml
Expand Down Expand Up @@ -96,7 +96,7 @@ The runtime chooses a credential based on the fields in the `ModelConfig`.

### Workload Identity

Omit `apiKeySecret` and `apiKeyPassthrough` to use Azure Workload Identity. For local development, the runtime will attempt to authenticate with your Azure CLI login.
Omit `apiKeySecret` and `apiKeyPassthrough` to use Azure Workload Identity. For local development, the runtime tries to authenticate using your Azure CLI login.

> **Note:** The runtime validates Azure credentials at startup. If credentials cannot be resolved, the agent does not become ready.

Expand Down Expand Up @@ -259,4 +259,4 @@ For the complete memory and embedding configuration and model requirements, see

- **The `ModelConfig` reports `Accepted=False`:** Check whether the required `endpointFrom` ConfigMap and key exist with `kubectl describe modelconfig MODEL_CONFIG_NAME --namespace NAMESPACE`.
- **The agent fails to become ready with a Workload Identity credential error:** Confirm the pod label and ServiceAccount annotation, and verify that the federated credential subject matches the ServiceAccount used by the pod.
- **Foundry returns `401 Unauthorized` or `403 Forbidden`:** Confirm that the managed identity has the `Cognitive Services User` role on the Foundry resource, or that the configured API key has access to the resource.
- **Foundry returns `401 Unauthorized` or `403 Forbidden`:** Confirm that the managed identity has the `Cognitive Services User` role on the Foundry resource, or that the configured API key has access to the resource.
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ After you apply the `ModelConfig`, you can select it from the **Model** dropdown

> **Note:** Azure Workload Identity requires the Go agent runtime (`runtime: go`).

Omit `apiKeySecret` and `apiKeyPassthrough` to use Azure Workload Identity. For local development, the runtime will attempt to authenticate with your Azure CLI login.
Omit `apiKeySecret` and `apiKeyPassthrough` to use Azure Workload Identity. For local development, the runtime tries to authenticate using your Azure CLI login.

> **Note:** The runtime validates Azure credentials at startup. If credentials cannot be resolved, the agent does not become ready.

Expand Down
8 changes: 4 additions & 4 deletions src/app/docs/kagent/supported-providers/byo-openai/page.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -60,19 +60,19 @@ Good job! You added a model to kagent. Next, you can [create or update an agent]

## Self-hosted vLLM behind an OpenAI-compatible gateway

A common self-hosted pattern places an OpenAI-compatible gateway such as [Bifrost](https://github.com/maximhq/bifrost) or [LiteLLM](https://docs.litellm.ai/) in front of a [vLLM](https://docs.vllm.ai/) server (kagent → gateway → vLLM). Configure it as an OpenAI-compatible provider exactly as above, with two extra things to get right.
A common self-hosted pattern places an OpenAI-compatible gateway such as [Bifrost](https://github.com/maximhq/bifrost) or [LiteLLM](https://docs.litellm.ai/) in front of a [vLLM](https://docs.vllm.ai/) server (kagent → gateway → vLLM). Configure the gateway as an OpenAI-compatible provider, exactly as shown above, with two extra things to get right.

### Enable tool calling in vLLM

kagent sends a `tools` array with `tool_choice: "auto"` on every request. Its runtime registers a built-in `ask_user` tool on every agent, so this happens even when you configure no tools yourself. The vLLM backend **must** be launched with automatic tool choice enabled, or every agent turn fails.
kagent sends a `tools` array with `tool_choice: "auto"` on every request. kagent's runtime registers a built-in `ask_user` tool on every agent, so a `tools` array is always sent, even when you configure no tools yourself. The vLLM backend **must** be launched with automatic tool choice enabled, or every agent turn fails.

```bash
vllm serve Qwen/Qwen2.5-7B-Instruct \
--enable-auto-tool-choice \
--tool-call-parser hermes
```

The correct `--tool-call-parser` depends on your model family. See the [vLLM tool calling docs](https://docs.vllm.ai/en/latest/features/tool_calling.html) for the current list. At the time of writing, Qwen2.5 uses `hermes` and Llama 3.1 uses `llama3_json`, but parser names change across vLLM releases, so check the docs for your model.
The correct `--tool-call-parser` depends on your model family. For example, Qwen2.5 uses `hermes` and Llama 3.1 uses `llama3_json`. Parser names change across vLLM releases, so check the [vLLM tool calling docs](https://docs.vllm.ai/en/latest/features/tool_calling.html) for your model's current parser name.

### Use the gateway's model identifier

Expand All @@ -95,7 +95,7 @@ spec:

### Troubleshooting: `provider API error (status 400)`

If every agent message fails with a generic `status 400`, vLLM was almost certainly started without `--enable-auto-tool-choice` and a matching `--tool-call-parser`. Because kagent always sends `tool_choice: "auto"`, vLLM rejects the request shape until automatic tool choice is enabled. Restart vLLM with the flags above and retry.
If every agent message fails with a generic `status 400`, the most common cause is that vLLM started without `--enable-auto-tool-choice` and a matching `--tool-call-parser`. Because kagent always sends `tool_choice: "auto"`, vLLM rejects the request until automatic tool choice is enabled. Restart vLLM with the flags above and retry.

## TLS Configuration

Expand Down
Loading