Skip to content

Commit dddc5bd

Browse files
committed
Add OpenRouter samples: prompt batch and budget gate
Call OpenRouter from Activities with Temporal-owned retries, error classification, Retry-After handling, heartbeats, and OpenRouter response caching so a retried request is billed at zero. The budget_gate sample pauses the batch on a soft budget or on OpenRouter's 402 and resumes on a raise_budget Update. Also adds OpenRouter as a model provider for the OpenAI Agents SDK plugin sample.
1 parent 98b6d63 commit dddc5bd

26 files changed

Lines changed: 1779 additions & 99 deletions

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,7 @@ Some examples require extra dependencies. See each sample's directory for specif
9898
without wrapping them in a workflow.
9999
* [open_telemetry](open_telemetry) - Trace workflows with OpenTelemetry.
100100
* [openai_agents](openai_agents) - Run OpenAI Agents SDK agents as durable Temporal workflows.
101+
* [openrouter](openrouter) - Call OpenRouter from Activities: fan out a prompt batch, and pause instead of failing when the budget or credits run out.
101102
* [patching](patching) - Alter workflows safely with `patch` and `deprecate_patch`.
102103
* [polling](polling) - Recommended implementation of an activity that needs to periodically poll an external resource waiting its successful completion.
103104
* [prometheus](prometheus) - Configure Prometheus metrics on clients/workers.

‎openai_agents/model_providers/README.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,23 @@ The example uses Anthropic Claude by default but can be modified to use other Li
3030

3131
Find more LiteLLM providers at: https://docs.litellm.ai/docs/providers
3232

33+
#### OpenRouter
34+
Uses [OpenRouter](https://openrouter.ai/) as the model provider, so the agent can run on any of the hundreds of models OpenRouter serves through one API key. OpenRouter speaks the OpenAI Chat Completions API, so the stock `OpenAIProvider` works once it is pointed at OpenRouter's base URL.
35+
36+
Start the OpenRouter worker:
37+
```bash
38+
export OPENROUTER_API_KEY="your_openrouter_api_key"
39+
40+
uv run openai_agents/model_providers/run_openrouter_worker.py
41+
```
42+
43+
Then run the example in a separate terminal:
44+
```bash
45+
uv run openai_agents/model_providers/run_openrouter_workflow.py
46+
```
47+
48+
The workflow uses `openai/gpt-4o-mini`; change `OPENROUTER_MODEL` in [workflows/openrouter_workflow.py](workflows/openrouter_workflow.py) to any OpenRouter model slug, or to `openrouter/auto` to let OpenRouter pick. See the [openrouter](../../openrouter) sample for calling OpenRouter directly from Activities with cost tracking and budgets.
49+
3350
### Extra
3451

3552
#### GPT-OSS with Ollama
Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
import asyncio
2+
import logging
3+
import os
4+
from datetime import timedelta
5+
6+
from agents import OpenAIProvider, set_tracing_disabled
7+
from openai import AsyncOpenAI
8+
from temporalio.client import Client
9+
from temporalio.contrib.openai_agents import ModelActivityParameters, OpenAIAgentsPlugin
10+
from temporalio.worker import Worker
11+
12+
from openai_agents.model_providers.workflows.openrouter_workflow import (
13+
OpenRouterAgentWorkflow,
14+
)
15+
16+
17+
# @@@SNIPSTART python-openai-agents-openrouter-provider
18+
def openrouter_provider() -> OpenAIProvider:
19+
"""OpenAI Agents SDK model provider backed by OpenRouter.
20+
21+
OpenRouter speaks the OpenAI Chat Completions API, so the stock provider
22+
works once it is pointed at OpenRouter's base URL. Client retries are off:
23+
the plugin runs each model call as a Temporal Activity, and Temporal owns
24+
the retries.
25+
"""
26+
default_headers: dict[str, str] = {}
27+
# Optional app attribution for OpenRouter's rankings.
28+
if referer := os.getenv("OPENROUTER_HTTP_REFERER"):
29+
default_headers["HTTP-Referer"] = referer
30+
if title := os.getenv("OPENROUTER_APP_TITLE"):
31+
default_headers["X-OpenRouter-Title"] = title
32+
33+
client = AsyncOpenAI(
34+
base_url="https://openrouter.ai/api/v1",
35+
api_key=os.environ["OPENROUTER_API_KEY"],
36+
max_retries=0,
37+
default_headers=default_headers or None,
38+
)
39+
# Chat Completions is OpenRouter's primary endpoint; the Agents SDK
40+
# defaults to the Responses API, which OpenRouter offers only in beta.
41+
return OpenAIProvider(openai_client=client, use_responses=False)
42+
43+
44+
# @@@SNIPEND
45+
46+
47+
async def main():
48+
# Disable Agents SDK tracing: the default exporter sends traces to OpenAI's
49+
# backend, which needs an OpenAI API key that this sample does not have.
50+
set_tracing_disabled(disabled=True)
51+
52+
logging.basicConfig(level=logging.WARNING)
53+
logging.getLogger("temporalio.workflow").setLevel(logging.DEBUG)
54+
55+
client = await Client.connect(
56+
"localhost:7233",
57+
plugins=[
58+
OpenAIAgentsPlugin(
59+
model_params=ModelActivityParameters(
60+
start_to_close_timeout=timedelta(seconds=60)
61+
),
62+
model_provider=openrouter_provider(),
63+
),
64+
],
65+
)
66+
67+
worker = Worker(
68+
client,
69+
task_queue="openai-agents-model-providers-task-queue",
70+
workflows=[
71+
OpenRouterAgentWorkflow,
72+
],
73+
)
74+
await worker.run()
75+
76+
77+
if __name__ == "__main__":
78+
asyncio.run(main())
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
import asyncio
2+
3+
from temporalio.client import Client
4+
from temporalio.contrib.openai_agents import OpenAIAgentsPlugin
5+
6+
from openai_agents.model_providers.workflows.openrouter_workflow import (
7+
OpenRouterAgentWorkflow,
8+
)
9+
10+
11+
async def main():
12+
client = await Client.connect(
13+
"localhost:7233",
14+
plugins=[
15+
OpenAIAgentsPlugin(),
16+
],
17+
)
18+
19+
result = await client.execute_workflow(
20+
OpenRouterAgentWorkflow.run,
21+
"What's the weather in Tokyo?",
22+
id="openai-agents-openrouter-workflow-id",
23+
task_queue="openai-agents-model-providers-task-queue",
24+
)
25+
print(f"Result: {result}")
26+
27+
28+
if __name__ == "__main__":
29+
asyncio.run(main())
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
from __future__ import annotations
2+
3+
from agents import Agent, Runner, function_tool
4+
from temporalio import workflow
5+
6+
# Any OpenRouter model slug works here. A fixed, tool-capable model keeps the
7+
# sample reproducible; swap in "openrouter/auto" to let OpenRouter choose.
8+
OPENROUTER_MODEL = "openai/gpt-4o-mini"
9+
10+
11+
@workflow.defn
12+
class OpenRouterAgentWorkflow:
13+
@workflow.run
14+
async def run(self, prompt: str) -> str:
15+
@function_tool
16+
def get_weather(city: str):
17+
workflow.logger.debug(f"Getting weather for {city}")
18+
return f"The weather in {city} is sunny."
19+
20+
agent = Agent(
21+
name="Assistant",
22+
instructions="You only respond in haikus. When asked about the weather always use the tool to get the current weather.",
23+
model=OPENROUTER_MODEL,
24+
tools=[get_weather],
25+
)
26+
27+
result = await Runner.run(agent, prompt)
28+
return result.final_output

‎openrouter/README.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# OpenRouter
2+
3+
These samples call [OpenRouter](https://openrouter.ai/) from Temporal Activities. OpenRouter serves hundreds of models from many providers behind one OpenAI-compatible API and one API key, and picks providers and models per request. Temporal handles everything around those calls: retries with backoff, fan-out with bounded concurrency, crash recovery, pausing for a human, and a durable per-attempt record of what was called and what it cost.
4+
5+
| Sample | Description |
6+
|--------|-------------|
7+
| [prompt_batch](prompt_batch) | Fan one OpenRouter call out per prompt with OpenRouter's Auto Router, and collect answer, model, and cost per prompt. Shows Temporal-owned retries, `Retry-After` handling, and retries served for free from OpenRouter's response cache. Start here. |
8+
| [budget_gate](budget_gate) | The same batch, but it pauses instead of failing when money runs out, whether a soft budget in the Workflow or OpenRouter's own "insufficient credits" error, and resumes on a `raise_budget` Update. |
9+
10+
For OpenRouter as the model provider behind the [OpenAI Agents SDK plugin](../openai_agents), see [openai_agents/model_providers](../openai_agents/model_providers#openrouter).
11+
12+
## Prerequisites
13+
14+
1. Follow the [repository prerequisites](../README.md), then install this sample's dependencies:
15+
16+
```bash
17+
uv sync --group openrouter
18+
```
19+
20+
2. Start a local dev server with the [Temporal CLI](https://docs.temporal.io/cli):
21+
22+
```bash
23+
temporal server start-dev
24+
```
25+
26+
3. Set an [OpenRouter API key](https://openrouter.ai/settings/keys) in the Worker's environment. A few cents of credit is enough for these samples.
27+
28+
```bash
29+
export OPENROUTER_API_KEY="sk-or-v1-..."
30+
```
31+
32+
Optional: set `OPENROUTER_HTTP_REFERER` and `OPENROUTER_APP_TITLE` for [app attribution](https://openrouter.ai/docs/app-attribution) in OpenRouter's rankings.
33+
34+
The API key stays in the Worker process. Prompts, answers, models, and costs go through the Workflow and are recorded in Event History; the key never does.
35+
36+
## Running a sample
37+
38+
Each sample has a Worker and a starter. Run them in separate terminals:
39+
40+
```bash
41+
# Terminal 1
42+
uv run --group openrouter openrouter/prompt_batch/run_worker.py
43+
44+
# Terminal 2
45+
uv run --group openrouter openrouter/prompt_batch/run_workflow.py "Explain retries in one sentence." "Write a haiku about databases."
46+
```
47+
48+
## How the Activity calls OpenRouter
49+
50+
[activities.py](activities.py) uses the `openai` SDK pointed at `https://openrouter.ai/api/v1`, which is the setup OpenRouter documents for OpenAI-compatible clients. OpenRouter-specific fields go in `extra_body`. Four things matter for durable execution:
51+
52+
- **Temporal owns retries.** The client is created with `max_retries=0`, so every attempt is one HTTP call and shows up in Event History. If you use OpenRouter's official `openrouter` package instead, pass `retry_config=RetryConfig("none", ...)`: by default it retries 5xx and connection errors for up to an hour, invisibly.
53+
- **Errors are classified.** 408, 429, and 5xx raise a retryable `ApplicationError`; 400, 401, 402 (out of credits), 403 (moderation), and other 4xx raise a non-retryable one. A `Retry-After` header becomes the next retry delay. OpenRouter can also return HTTP 200 with an `error` body and no `choices`; the Activity checks for that.
54+
- **Retries are free when the first call succeeded.** The Activity sends `X-OpenRouter-Cache: true`, so if a Worker dies after OpenRouter answered but before Temporal recorded the result, the retried, byte-identical request is served from OpenRouter's response cache and billed at $0. Nothing per-attempt goes in the request body, so attempts stay identical.
55+
- **Heartbeats.** The Activity heartbeats so a dead Worker is detected after `heartbeat_timeout` (10s) rather than after the full `start_to_close_timeout`.
56+
57+
Each result carries the concrete model OpenRouter chose, OpenRouter's reported `usage.cost`, the generation id, and the cache status.
58+
59+
## What Temporal does and does not guarantee
60+
61+
Activities are at-least-once. If a Worker dies mid-call, the retry re-sends the request; within the cache TTL that retry costs nothing, but two identical requests in flight at the same time both miss the cache and both bill. Completed Activities are never re-run, so a restarted batch resumes at the first unfinished prompt.
62+
63+
OpenRouter decides which provider and model serve a request, in milliseconds (Auto Router, `models` fallback lists, provider preferences). Temporal decides what happens over time: waiting out a rate limit, surviving a Worker crash, pausing for hours until a human acts, and keeping the audit trail.
64+
65+
## Batch size
66+
67+
Each Activity adds a few events to the Workflow's Event History, and every answer is part of the Workflow result. These samples cap a batch at 100 prompts. For larger batches, use one Workflow per slice, or the pattern in [batch_sliding_window](../batch_sliding_window) with continue-as-new.
68+
69+
## Tests
70+
71+
The tests replace OpenRouter with a fake HTTP transport and the Activity with a fake, so they need no API key and make no network calls:
72+
73+
```bash
74+
uv run --group openrouter pytest tests/openrouter
75+
```

‎openrouter/__init__.py‎

Whitespace-only changes.

0 commit comments

Comments
 (0)