-
Notifications
You must be signed in to change notification settings - Fork 5.1k
CAMEL-23500: Document camel-openai usage with OpenAI-compatible providers #23614
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
793c04e
3062455
2264fbd
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -256,17 +256,6 @@ from("direct:conversation") | |
| .log("Second response: ${body}"); // Will remember "Alice" | ||
| ---- | ||
|
|
||
| === Using Third-Party or Local OpenAI-Compatible Endpoint | ||
|
|
||
| .Usage example: | ||
| [source,java] | ||
| ---- | ||
| from("direct:local") | ||
| .setBody(constant("Hello from local LLM")) | ||
| .to("openai:chat-completion?baseUrl=http://localhost:1234/v1&model=local-model") | ||
| .log("${body}"); | ||
| ---- | ||
|
|
||
| == Input Handling | ||
|
|
||
| The component accepts the following types of input in the message body: | ||
|
|
@@ -492,17 +481,136 @@ String-valued fields are set directly. Non-string fields (numbers, booleans, obj | |
|
|
||
| NOTE: This maps fields from the response message's additional properties (fields not part of the standard schema). Standard response fields like `content`, `role`, and `tool_calls` are not accessible through this option. | ||
|
|
||
| == Compatibility | ||
| == OpenAI-Compatible Providers | ||
|
|
||
| Because the component speaks the OpenAI API, you do not need a separate component to use third-party | ||
| gateways or local model servers that expose an OpenAI-compatible API. Point `baseUrl` at the provider, | ||
| set `apiKey` if it requires one, and use the same operations and options as with OpenAI. | ||
|
|
||
| NOTE: Ensure the provider supports the chat completions and/or embeddings endpoint format you use. | ||
| Authentication requirements and minor API variations differ between providers. | ||
|
|
||
| [cols="1,2,2",options="header"] | ||
| |=== | ||
| | Provider | `baseUrl` | Notes | ||
|
|
||
| | OpenAI | `https://api.openai.com/v1` | Default; `baseUrl` can be omitted | ||
| | OpenRouter | `https://openrouter.ai/api/v1` | Multi-model gateway with provider routing and fallbacks | ||
| | Ollama | `http://localhost:11434/v1` | Local LLM server | ||
| | LM Studio | `http://localhost:1234/v1` | Local model runner | ||
| | vLLM | `http://localhost:8000/v1` | High-throughput, self-hosted serving engine | ||
| |=== | ||
|
|
||
| === Ollama (local) | ||
|
|
||
| https://ollama.com[Ollama] runs models locally and does not require an API key. Install Ollama and | ||
| pull the model used in the example below: | ||
|
|
||
| [source,bash] | ||
| ---- | ||
| ollama run llama3.2 | ||
| ---- | ||
|
|
||
| [tabs] | ||
| ==== | ||
| Java:: | ||
| + | ||
| [source,java] | ||
| ---- | ||
| from("direct:chat") | ||
| .setBody(constant("What is Apache Camel?")) | ||
| .to("openai:chat-completion?baseUrl=http://localhost:11434/v1&model=llama3.2"); | ||
| ---- | ||
|
|
||
| YAML:: | ||
| + | ||
| [source,yaml] | ||
| ---- | ||
| - to: | ||
| uri: openai:chat-completion | ||
| parameters: | ||
| baseUrl: http://localhost:11434/v1 | ||
| model: llama3.2 | ||
| userMessage: What is Apache Camel? | ||
| ---- | ||
| ==== | ||
|
|
||
| For local embeddings, use an embedding model such as `nomic-embed-text` (see the *Embedding Models by | ||
| Provider* table below). | ||
|
|
||
| === LM Studio (local) | ||
|
|
||
| https://lmstudio.ai[LM Studio] serves the model currently loaded in the app. Set `model` to the | ||
| identifier shown in its UI. | ||
|
|
||
| [source,java] | ||
| ---- | ||
| from("direct:chat") | ||
| .to("openai:chat-completion?baseUrl=http://localhost:1234/v1&model=local-model"); | ||
| ---- | ||
|
|
||
| === vLLM (self-hosted) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you add the vLLM command to install and run For example, locally I am using vllm-mlx a lot on mac os, can you mention this as well? For example, this is how I run it on my mac |
||
|
|
||
| https://docs.vllm.ai[vLLM] is a high-throughput LLM serving engine. Install it with `pip install vllm` | ||
| and start the model used in the example below: | ||
|
|
||
| [source,bash] | ||
| ---- | ||
| vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000 | ||
| ---- | ||
|
|
||
| [source,java] | ||
| ---- | ||
| from("direct:chat") | ||
| .to("openai:chat-completion?baseUrl=http://localhost:8000/v1" | ||
| + "&model=meta-llama/Llama-3.1-8B-Instruct"); | ||
| ---- | ||
|
|
||
| This component works with any OpenAI API-compatible endpoint by setting the `baseUrl` parameter. This includes: | ||
| If vLLM was started with `--api-key`, pass the same value via the `apiKey` option. | ||
|
|
||
| - OpenAI official API (`https://api.openai.com/v1`) | ||
| - Local LLM servers (e.g., Ollama, LM Studio, LocalAI) | ||
| - Third-party OpenAI-compatible providers | ||
| On Apple Silicon, the `vllm-mlx` variant uses Apple's MLX framework and runs the same way: | ||
|
|
||
| [source,bash] | ||
| ---- | ||
| vllm-mlx serve mlx-community/Qwen2.5-7B-Instruct-4bit --port 8000 | ||
| ---- | ||
|
|
||
| === OpenRouter | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Did you manage to actually test it with openrouter? To be honest, when I opened the jira issue, I never used openrouter, I just supposed it would work :D if you have openrouter, and can do a quick test would be nice, otherwise, no worries, maybe @orpiske can test and review the docs (if needed) in the future |
||
|
|
||
| https://openrouter.ai[OpenRouter] is an OpenAI-compatible gateway that routes requests across many | ||
| model providers. Set `baseUrl` to its endpoint and select a model with a cross-provider identifier: | ||
|
|
||
| [source,java] | ||
| ---- | ||
| from("direct:chat") | ||
| .to("openai:chat-completion?baseUrl=https://openrouter.ai/api/v1" | ||
| + "&apiKey={{openrouter.api.key}}" | ||
| + "&model=anthropic/claude-sonnet-4-20250514"); | ||
| ---- | ||
|
|
||
| ==== Provider Routing | ||
|
|
||
| OpenRouter accepts a `provider` object in the request body to control routing order and fallbacks. | ||
| Pass it through the `additionalBodyProperty` option as a JSON value — the component parses JSON-valued | ||
| properties and adds them to the request body: | ||
|
|
||
| [source,yaml] | ||
| ---- | ||
| - to: | ||
| uri: openai:chat-completion | ||
| parameters: | ||
| baseUrl: https://openrouter.ai/api/v1 | ||
| apiKey: "{{openrouter.api.key}}" | ||
| model: anthropic/claude-sonnet-4-20250514 | ||
| additionalBodyProperty.provider: '{"order":["anthropic","google"],"allow_fallbacks":false}' | ||
| ---- | ||
|
|
||
| [NOTE] | ||
| ==== | ||
| When using local or third-party providers, ensure they support the chat completions and/or embeddings API endpoint format. Some providers may have different authentication requirements or API variations. | ||
| OpenRouter's optional attribution headers (`HTTP-Referer` and `X-Title`, which identify your app on | ||
| the OpenRouter rankings) are sent as HTTP request headers, not body properties. The component does not | ||
| currently expose a way to set custom HTTP request headers, so they cannot be configured today. They | ||
| are optional and do not affect chat completions. | ||
| ==== | ||
|
|
||
| === Embedding Models by Provider | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This new section overlaps with the existing
== Compatibilitysection further down (currently at line 495 onmain), which lists the same providers. It also overlaps with=== Using Third-Party or Local OpenAI-Compatible Endpoint(line 259 onmain) which has a near-identical LM Studio example.Now that this comprehensive section exists, consider:
=== Using Third-Party or Local OpenAI-Compatible Endpointsection (line 259) since it's now a subset of this content== Compatibilitysection to avoid repeating the provider list, or merging it into this sectionThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for catching the overlap. Pushed 2264fbd which:
=== Using Third-Party or Local OpenAI-Compatible Endpoint(was line 259); its example is now covered by=== LM Studio (local)in the new section.== Compatibilityheading along with its intro paragraph, bullet list, and NOTE block (redundant with the new provider table).== CompatibilityNOTE as aNOTE:in the intro of== OpenAI-Compatible Providers.=== Embedding Models by Providerto sit under== OpenAI-Compatible Providers, right before== Embeddings Operation, so all provider info stays grouped.Net: -24/+3 lines, single source of truth for OpenAI-compatible provider docs.
Reported by Claude Code on behalf of Karol Krawczyk