Skip to content

examples.md

Latest commit

 

History

History
348 lines (288 loc) · 14.4 KB

examples.md

File metadata and controls

348 lines (288 loc) · 14.4 KB
Raw

Examples

Every snippet below runs against cargo run -p gw-server with the embedded demo config (mock upstreams, zero egress) unless it says otherwise. The demo key is ak-demo-123.

Chat completion

curl -s localhost:8080/v1/chat/completions \
  -H 'authorization: Bearer ak-demo-123' -H 'content-type: application/json' \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"hello"}]}'
{"id":"chatcmpl-local-1","object":"chat.completion","model":"gpt-4o",
 "choices":[{"index":0,"message":{"role":"assistant","content":"..."},"finish_reason":"stop"}],
 "usage":{"prompt_tokens":5,"completion_tokens":10,"total_tokens":15}}

Streaming

curl -sN localhost:8080/v1/chat/completions \
  -H 'authorization: Bearer ak-demo-123' -H 'content-type: application/json' \
  -d '{"model":"gpt-4o","stream":true,"messages":[{"role":"user","content":"count to 3"}]}'
data: {"choices":[{"delta":{"content":"one"},"finish_reason":null}]}
data: {"choices":[{"delta":{"content":" two"},"finish_reason":null}]}
data: {"choices":[{"delta":{},"finish_reason":"stop"}],"usage":{...}}
data: [DONE]

The final data frame carries usage and finish_reason. Frames arrive as the upstream produces them only when security.dlp_redact is off; the embedded demo config ships with it on, so the stream is buffered and replayed post-redaction (see governance.md).

Anthropic messages

curl -sN localhost:8080/v1/messages \
  -H 'x-api-key: ak-demo-123' -H 'content-type: application/json' \
  -d '{"model":"claude-sonnet","stream":true,"max_tokens":128,
       "messages":[{"role":"user","content":"hi"}]}'
event: message_start
data: {"type":"message_start","message":{...}}
event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}
event: content_block_delta
data: {"type":"content_block_delta","delta":{"type":"text_delta","text":"..."}}
event: content_block_stop
data: {"type":"content_block_stop","index":0}
event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{...}}
event: message_stop
data: {"type":"message_stop"}

/v1/messages also works on OpenAI-protocol models — the gateway converts.

Tools

curl -s localhost:8080/v1/chat/completions \
  -H 'authorization: Bearer ak-demo-123' -H 'content-type: application/json' \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"weather in NYC?"}],
       "tools":[{"type":"function","function":{"name":"get_weather",
         "parameters":{"type":"object","properties":{"city":{"type":"string"}}}}}]}'

The response sets finish_reason:"tool_calls" with the call in choices[0].message.tool_calls; content carries any text the model emitted alongside and is omitted otherwise.

Embeddings, images, audio

curl -s localhost:8080/v1/embeddings \
  -H 'authorization: Bearer ak-demo-123' -H 'content-type: application/json' \
  -d '{"model":"text-embedding-3","input":"embed me"}'

curl -s localhost:8080/v1/images/generations \
  -H 'authorization: Bearer ak-demo-123' -H 'content-type: application/json' \
  -d '{"model":"dall-e-3","prompt":"a red cube","n":1}'

curl -s localhost:8080/v1/audio/speech \
  -H 'authorization: Bearer ak-demo-123' -H 'content-type: application/json' \
  -d '{"model":"tts-1","input":"hello"}'

Batch workflow

# 1. upload a JSONL file (one request per line)
FID=$(curl -s localhost:8080/v1/files \
  -H 'authorization: Bearer ak-demo-123' -H 'content-type: application/json' \
  -d '{"purpose":"batch","file":"{\"body\":{\"model\":\"gpt-4o\",\"messages\":[{\"role\":\"user\",\"content\":\"one\"}]}}\n{\"body\":{\"model\":\"gpt-4o\",\"messages\":[{\"role\":\"user\",\"content\":\"two\"}]}}"}' \
  | python3 -c 'import json,sys;print(json.load(sys.stdin)["id"])')

# 2. create a batch from the file
BID=$(curl -s localhost:8080/v1/batches \
  -H 'authorization: Bearer ak-demo-123' -H 'content-type: application/json' \
  -d "{\"input_file_id\":\"$FID\"}" \
  | python3 -c 'import json,sys;print(json.load(sys.stdin)["id"])')

# 3. poll for results
curl -s localhost:8080/v1/batches/$BID -H 'authorization: Bearer ak-demo-123'

Observability

curl -s localhost:8080/metrics | grep gateway_
export GW_ADMIN_TOKEN=change-me                    # names the global admin bearer (see conf/gateway.yaml)
curl -s -H "Authorization: Bearer $GW_ADMIN_TOKEN" \
  'localhost:8080/internal/ledger?limit=5'         # operator surface — global token + private network

Going live against a real provider

# my.yaml
listen: {host: 127.0.0.1, port: 8080}
access_keys:
  - {ak: ak-live, product: live, qps: 20, daily_token_quota: 10000000}
providers:
  - name: openai
    kind: openai
    api_key_env: OPENAI_API_KEY
    endpoint: "https://api.openai.com"   # or an OpenAI-compatible relay
models:
  - {name: gpt-4o-mini, provider: openai,
     input_price_per_1k_micros: 150, output_price_per_1k_micros: 600}
export OPENAI_API_KEY=sk-...
GW_CONFIG=my.yaml cargo run -p gw-server
curl -s localhost:8080/v1/chat/completions \
  -H 'authorization: Bearer ak-live' -H 'content-type: application/json' \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}'

Add a second provider (kind: anthropic, kind: deepseek, …) and more models: to route several vendors through one gateway. See Providers and Configuration.

Claude on AWS Bedrock

accounts:
  - {name: bedrock, provider: aws, endpoint: "https://bedrock-runtime.us-east-1.amazonaws.com",
     api_key_env: AWS_BEARER_TOKEN_BEDROCK, protocols: ["aws-anthropic", "aws-llama"]}
models:
  - {name: us.anthropic.claude-sonnet-4-5-20250929-v1:0, protocol: aws-anthropic, prompt_cache: true}
  - {name: us.meta.llama3-3-70b-instruct-v1:0, protocol: aws-llama}
export AWS_BEARER_TOKEN_BEDROCK=bedrock-api-key-...   # or AWS_ACCESS_KEY_ID + secret_key_env
curl -s localhost:8080/v1/chat/completions \
  -H 'authorization: Bearer ak-live' -H 'content-type: application/json' \
  -d '{"model":"us.anthropic.claude-sonnet-4-5-20250929-v1:0","stream":true,"reasoning_effort":"low",
       "messages":[{"role":"user","content":"What is 17*23?"}]}'

The same request shapes as any other model: reasoning comes back as reasoning_content + signed reasoning_details, tools and thinking replay through the loop, and /v1/messages speaks to it natively.

Grok (xAI)

providers:
  - {name: xai, kind: xai, api_key_env: XAI_API_KEY}
models:
  - {name: grok-4.6, provider: xai,
     input_price_per_1k_micros: 2000, output_price_per_1k_micros: 6000, token_rate: {read_cache: 0.25}}
  - {name: grok-imagine-image-2.0, provider: xai, protocol: image, unit_price_micros: 70000}
  - {name: grok-imagine-video-1.5, provider: xai, protocol: video, unit_price_micros: 100000}

The preset covers xAI's OpenAI-compatible surfaces (/v1/chat/completions, /v1/responses, /v1/images/generations), its async video (/v1/videos/generations answers {request_id}; poll GET /v1/videos/{id} until done — the first done bills the clip's seconds at the unit price and records xAI's cost_in_usd_ticks as the vendor cost) and its realtime voice socket (protocol: realtime, model grok-voice-latest, billed by the delivered output estimate since xAI reports no usage). reasoning_effort passes through verbatim — grok-4.6 takes low…xhigh, grok-4.3 also none, and a model answers 400 for a value it does not list. Anthropic clients reach Grok through /v1/messages (the gateway converts; xAI's own Anthropic-compatible endpoint is deprecated).

Price the chat models by their reasoning too: xAI's chat wire reports reasoning_tokens outside completion_tokens and adds them into total_tokens, and the gateway reads that arithmetic and bills them at the output rate. Every reply carries usage.cost_in_usd_ticks, so with list prices configured the ledger's cost_micros matches the vendor's own charge — the check scripts/live-matrix/live_matrix.py --group xai runs.

The same models are served by Bedrock and OpenRouter, both of which normalize that usage shape to the OpenAI one:

providers:
  - {name: openrouter, kind: openrouter, api_key_env: OPENROUTER_API_KEY}
accounts:
  # Converse: reasoning is encrypted, no prompt caching, effort as `reasoning_config`
  - {name: bedrock, provider: aws, endpoint: "https://bedrock-runtime.us-east-1.amazonaws.com",
     api_key_env: AWS_BEARER_TOKEN_BEDROCK, protocols: ["aws-converse"]}
  # the same account's OpenAI-compatible endpoint: effort, implicit cache reads, no reasoning prose
  - {name: bedrock-oai, provider: aws-oai, endpoint: "https://bedrock-runtime.us-east-1.amazonaws.com/openai",
     api_key_env: AWS_BEARER_TOKEN_BEDROCK, protocols: ["openai-chat"]}
models:
  - {name: x-ai/grok-4.3, provider: openrouter,
     input_price_per_1k_micros: 1250, output_price_per_1k_micros: 2500, token_rate: {read_cache: 0.16}}
  - {name: "us.xai.grok-4.6", provider: aws, protocol: aws-converse,
     input_price_per_1k_micros: 2200, output_price_per_1k_micros: 6600}
  - {name: "global.xai.grok-4.6", provider: aws-oai, protocol: openai-chat,
     input_price_per_1k_micros: 2000, output_price_per_1k_micros: 6000, token_rate: {read_cache: 0.25}}

Bedrock serves xai.grok-4.6 only through its us. and global. inference profiles. Converse returns the reasoning encrypted (a redacted_thinking block on /v1/messages, replayable in a tool loop), refuses cachePoint, and reports no cache reads — so leave read_cache off that model. The account's OpenAI-compatible endpoint is the fuller route: it takes reasoning_effort none…max, reports implicit cache reads as cached_tokens, and needs no preset — kind: openai plus the /openai base URL and the Bedrock API key. That caching is implicit and best-effort: an identical prefix repeated within seconds is read back on one run and missed on the next, so treat a cache hit as a discount that may not arrive, never as a planned price. OpenRouter reports usage.cost, so its rows carry a vendor cost as well.

Coding agents

Every coding agent that takes a base URL and a key works against the gateway; the gateway serves the exact wire each one speaks. What each client sends was captured from the real client and replays in the live matrix (agents group).

Claude Code — the Anthropic wire, so the gateway model must speak it natively or by conversion. Claude Code sends a system-block array with cache_control points, its tool set, thinking (with display: omitted), context_management, and its user id in metadata.user_id; it enables beta features through the anthropic-beta header, which the gateway forwards to Anthropic-wire upstreams (the header for kind: anthropic, the anthropic_beta body list for aws-anthropic). Point it at the gateway with the access key as the API key:

export ANTHROPIC_BASE_URL=https://gw.example.com
export ANTHROPIC_API_KEY=ak-...
claude --model claude-sonnet-5        # any configured Anthropic-wire model

ANTHROPIC_MODEL and ANTHROPIC_SMALL_FAST_MODEL pick the main and helper models when the flag is absent; both names must be configured on the gateway. The ledger attributes each turn to the JSON Claude Code puts in metadata.user_id unless the key has an owner.

Codex CLI — the native Responses wire with store: false, encrypted reasoning (include: ["reasoning.encrypted_content"]), function and custom tools, and prompt_cache_key. Serve it a reasoning model on protocol: responses — the surface is a native passthrough, so the model has to speak that wire:

models:
  - {name: gpt-5.4, provider: openai, protocol: responses}
# ~/.codex/config.toml
model = "gpt-5.4"
model_provider = "gateway"

[model_providers.gateway]
name = "gateway"
base_url = "https://gw.example.com/v1"
env_key = "GW_API_KEY"          # export GW_API_KEY=ak-...
wire_api = "responses"

Codex tries a WebSocket first and falls back to HTTPS on its own.

VS Code chat (GitHub Copilot) — Chat: Manage Language Models → Add Models → Custom Endpoint: pick the API type (Chat Completions, Responses or Anthropic Messages), give the full endpoint URL (https://gw.example.com/v1/chat/completions, /v1/responses or /v1/messages) and the access key; the model entries land in chatLanguageModels.json with "vendor": "customendpoint". No Copilot plan is needed for bring-your-own-key models.

opencode — an OpenAI-compatible provider block in opencode.json:

{
  "provider": {
    "gateway": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "gateway",
      "options": {"baseURL": "https://gw.example.com/v1", "apiKey": "{env:GW_API_KEY}"},
      "models": {"claude-sonnet-5": {"name": "claude-sonnet-5"}}
    }
  }
}

MCP servers through the gateway — any MCP client that speaks Streamable HTTP with a bearer header reaches the servers a key is entitled to:

claude mcp add --transport http tools https://gw.example.com/mcp/tools \
  --header "Authorization: Bearer ak-..."
codex mcp add tools --url https://gw.example.com/mcp/tools --bearer-token-env-var GW_API_KEY

The key's mcp_tools allowlist decides which tools the client is shown and may call; the gateway records each call in /admin/audit/events.

Cursor and other bring-your-own-key clients

Cursor's Models → API Keys lets you point its OpenAI, Anthropic and Google keys at another base URL. Requests are relayed by Cursor's servers, so the gateway must be reachable from the internet over HTTPS (a public host or a tunnel — never localhost).

  • OpenAI key + "Override OpenAI Base URL" → https://gw.example.com/v1 (Cursor appends /chat/completions), key = a gateway access key. Add each gateway model name under Model Names — any configured model, Claude and Gemini included: the gateway serves them on the OpenAI wire with streaming and tool calls, which is what Cursor's Agent/Ask modes send.
  • Anthropic key + base URL → https://gw.example.com; the access key rides as x-api-key, and /v1/messages serves every model natively or by conversion.

Tab completion and inline edit stay on Cursor's own models regardless of keys, and Cursor shows only the final answer (reasoning prose is dropped client-side). Verified on the gateway side by the live matrix (streamed chat with tools, x-api-key auth, /v1/models); driving the Cursor client itself was not part of that run.