diff --git a/README.md b/README.md index 69c1555..30646c8 100644 --- a/README.md +++ b/README.md @@ -37,14 +37,14 @@ Need a schema that isn't here? [Open an issue](https://github.com/livepeer/runne ## Examples -| Example | Goal | Registration | Mode | Transport | Pricing | -| ---------------------------------------------------- | ------------------------------------------------------------------------------------- | ------------ | ----------- | ----------------- | ------- | -| [`hello-world`](./hello-world) | The simplest app: one request, one response | dynamic | single-shot | HTTP (JSON) | fixed | -| [`tiles`](./tiles) | Capacity fan-out — one call per tile | dynamic | single-shot | HTTP (base64 PNG) | fixed | -| [`api-proxy`](./api-proxy) | Pass calls through to a hosted API — the operator holds the key, callers pay per call | static | single-shot | HTTP (JPEG bytes) | fixed | -| [`echo`](./echo) | Realtime video, transformed and echoed back | dynamic | persistent | trickle | hour | -| [`vllm`](./vllm) | Drop-in OpenAI API; the client stays unmodified | static | single-shot | HTTP + SSE | hour | -| [`realtime-transcription`](./realtime-transcription) | Audio up, transcripts back, on one socket | dynamic | persistent | WebSocket | hour | +| Example | Goal | Registration | Mode | Transport | Pricing | +| ---------------------------------------------------- | ---------------------------------------------------------------------------------------- | ------------ | ----------- | ----------------- | ------- | +| [`hello-world`](./hello-world) | The simplest app: one request, one response | dynamic | single-shot | HTTP (JSON) | fixed | +| [`tiles`](./tiles) | Capacity fan-out — one call per tile | dynamic | single-shot | HTTP (base64 PNG) | fixed | +| [`api-proxy`](./api-proxy) | Pass calls through to hosted APIs — the operator holds the key, one capability per model | static | single-shot | HTTP (JPEG bytes) | fixed | +| [`echo`](./echo) | Realtime video, transformed and echoed back | dynamic | persistent | trickle | hour | +| [`vllm`](./vllm) | Drop-in OpenAI API; the client stays unmodified | static | single-shot | HTTP + SSE | hour | +| [`realtime-transcription`](./realtime-transcription) | Audio up, transcripts back, on one socket | dynamic | persistent | WebSocket | hour | Start with `hello-world` (the smallest end-to-end path); the others each layer on one new idea. More will follow, including a full example that exercises every feature. Each is self-contained and runs **offchain** (free, no wallet); most also run **on-chain** (paid) — see each README. diff --git a/api-proxy/.env.example b/api-proxy/.env.example index 78e3d6a..e8173b8 100644 --- a/api-proxy/.env.example +++ b/api-proxy/.env.example @@ -22,9 +22,10 @@ ORCH_ETH_PASSWORD=your-operator-keystore-password # Registered orch = ticket recipient (-ethOrchAddr); empty = use the operating key. ORCH_ONCHAIN_ADDR=0xYourRegisteredOrchestrator -# The runner's price lives in runners.json (static runner): USD billed once per -# call (fixed pricing). Keep it under ~0.0019: the signer signs at most 100 +# Each runner's price lives in runners.json (static runner): USD billed once per +# call (fixed pricing). Keep each under ~0.0019: the signer signs at most 100 # tickets per payment and the demo orchestrator runs -ticketEV=1e10. -# Signer's max-price cap (payer side) is per billing unit, here one call, so it -# must exceed the runners.json price. +# Signer's max-price cap (payer side) is per billing unit, here one call, and it +# is one cap for every capability, so it must exceed the highest runners.json +# price (sd3, not flux). MAX_PRICE_PER_UNIT=0.000111USD diff --git a/api-proxy/README.md b/api-proxy/README.md index b6eb198..7178d04 100644 --- a/api-proxy/README.md +++ b/api-proxy/README.md @@ -1,50 +1,59 @@ # API-proxy app (a runner that is pure config) -The live runner can also **pass calls through to an API that runs somewhere else** — here the **Hugging Face text-to-image inference API**. This example's runner is a **stock nginx**: [nginx.conf.template](nginx.conf.template) forwards each call to one pinned model URL and injects the operator's token. There is **no app code at all** — the orchestrator operator offers a hosted model ([Stable Diffusion 3 medium](https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers) by default) as a paid capability with two config files. +The live runner can also **pass calls through to an API that runs somewhere else** — here the **Hugging Face text-to-image inference API**. This example's runner is a **stock nginx**: [nginx.conf.template](nginx.conf.template) forwards each call to its route's pinned model URL and injects the operator's token. There is **no app code at all** — the orchestrator operator offers two hosted models ([Stable Diffusion 3 medium](https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers) and [FLUX.1 schnell](https://huggingface.co/black-forest-labs/FLUX.1-schnell)) as **two separately priced capabilities**, with nothing but config. -| | | -| ------------ | -------------------------------------------- | -| App id | `livepeer-example/stable-diffusion-3-medium` | -| Runner mode | single-shot | -| Registration | static (orchestrator config + health poll) | -| Transport | HTTP (HF payload in, JPEG bytes out) | -| Pricing | fixed (one price per call) | -| Port | 8989 | +| | | +| ------------ | ------------------------------------------------------------------------------- | +| App ids | `livepeer-example/stable-diffusion-3-medium`, `livepeer-example/flux-1-schnell` | +| Runner mode | single-shot | +| Registration | static (orchestrator config + health poll) | +| Transport | HTTP (HF payload in, JPEG bytes out) | +| Pricing | fixed, per capability (0.0001 and 0.00004 USD per image) | +| Port | 8989 | Prerequisites (Docker, `uv`, and the [`livepeer-gateway` SDK](https://pypi.org/project/livepeer-gateway/)) and the shared on-chain/payment setup live in the [repo README](../README.md). The demo upstream additionally needs a **Hugging Face API token** (`HF_TOKEN`, from [huggingface.co → settings → tokens](https://huggingface.co/settings/tokens)) with inference-provider credits. ## How it's wired -The app is attached as a **static runner**: the orchestrator reads [runners.json](runners.json) via `-liveRunnerConfig` — app id, runner URL, single-shot mode, and the fixed price — and health-polls `/health` (an nginx `return 200`). The `/proxy` location proxies to the pinned model URL (`MODEL` in [compose.yml](compose.yml)) with `Authorization: Bearer ` added. The caller's body is the [Hugging Face text-to-image payload](https://huggingface.co/docs/inference-providers/tasks/text-to-image) forwarded verbatim — `{"inputs": ""}` — and the image comes back as **raw JPEG bytes**. The client calls it with `runner_selector` → `call_runner` ([client.py](client.py)) — discover, then one **single-shot** call per image, reading the bytes from `result.content`; the orchestrator reserves a session per call and releases it when the response returns. Grep `# Livepeer:` in client.py to see the exact calls. +The apps are attached as **static runners**: the orchestrator reads [runners.json](runners.json) via `-liveRunnerConfig` — app id, runner URL, single-shot mode, and the fixed price, one entry per capability — and health-polls `/health` (an nginx `return 200`). Each `/proxy` location proxies to its pinned model URL with `Authorization: Bearer ` added. The caller's body is the [Hugging Face text-to-image payload](https://huggingface.co/docs/inference-providers/tasks/text-to-image) forwarded verbatim — `{"inputs": ""}` — and the image comes back as **raw JPEG bytes**. The client calls it with `runner_selector` → `call_runner` ([client.py](client.py)) — discover, then one **single-shot** call per image, reading the bytes from `result.content`; the orchestrator reserves a session per call and releases it when the response returns. Grep `# Livepeer:` in client.py to see the exact calls. + +**A capability is a registration, not a container.** One nginx serves both: each `runners.json` entry points its app id at its own path (`http://app:8989/sd3`), and the orchestrator preserves that path when it forwards, so the two registrations land on different `location` blocks with different pinned models and different prices. `--app` on the client picks which one to call. + +The shared `/health` is deliberate: it reports that nginx is up and nothing more. [nginx.conf.template](nginx.conf.template) says why a real upstream check does not belong there. ## Offering an API as a capability — what this shows -Everything is operator-side config. `runners.json` names the capability and sets the **fixed per-image price**; the nginx config pins the model URL and holds the credential (`HF_TOKEN`, from `.env`). The pinned URL is also the security model: the operator's credential can only be spent on exactly the offered model. The config pins the method to `POST` and drops the caller's query string too, so the body is the only thing a caller controls: they choose nothing but the prompt, and never see an API key. They discover the capability and pay **per image through Livepeer**, while the operator pays the upstream and prices above the per-image upstream cost. +Everything is operator-side config. `runners.json` names each capability and sets its **fixed per-image price**; the nginx config pins the model URL and holds the credential (`HF_TOKEN`, from `.env`). The pinned URL is also the security model: the operator's credential can only be spent on exactly the models offered. The config pins the method to `POST` and drops the caller's query string too, so the body is the only thing a caller controls: they choose nothing but the prompt, and never see an API key. They discover a capability and pay **per image through Livepeer**, while the operator pays the upstream and prices above the per-image upstream cost. -The app id names the model, not the proxy, because that is what callers discover: they match it exactly, so it has to say what they get. Swapping `MODEL` means renaming the app id with it. +**A capability is a product, not a parameter.** FLUX.1 schnell is faster and cheaper than SD3 medium, so it is its own app id at its own price rather than a flag on one shared endpoint. Callers pick between them the same way they pick between orchestrators: discovery filters on `app`, matched exactly, so the app id has to say what you get. Swapping a model means renaming its app id with it, and offering a third is the same change once more: one `location` block, one `runners.json` entry. -Offering a second model is more config, not code: one more `runners.json` entry (its own app id and price) plus one more nginx service with a different `MODEL`. +**Each entry carries its own capacity.** That is the right shape here: the two capabilities do not contend, because the work happens upstream at Hugging Face and nginx is only forwarding bytes. Capacity is exposure control rather than a local limit, since payment is taken when the session is reserved: it caps how many paid calls can be in flight against the operator's credential, so size it to the upstream quota rather than to this machine. Runners that genuinely share a resource — several models resident on one GPU — are a different problem, since the orchestrator has no way to know that two registrations sit on the same card ([go-livepeer#4015](https://github.com/livepeer/go-livepeer/issues/4015)). **Fixed pricing** is the natural fit: one call is one bounded unit of work, so the runner bills one flat price per call instead of metering time. -> [!NOTE] -> Registration can also be **dynamic**: an operator tool can `register_runner` several API endpoints at runtime, each as its own priced capability, without touching the orchestrator config. See [livepeer/api-proxy](https://github.com/livepeer/api-proxy) for an example of dynamic endpoint registration, with key storage and request stats for orchestrator operators. +## Pinned here, dynamic in livepeer/api-proxy + +Both capabilities here are fixed at deploy time: two `location` blocks and two `runners.json` entries, changed by editing config and restarting. That is the whole point of a **static** runner, and it is the right trade when the offering is stable. + +Registration can also be **dynamic**. [livepeer/api-proxy](https://github.com/livepeer/api-proxy) is the same one-process-many-endpoints shape, except endpoints are added at runtime with a CLI or a dashboard: one `register_runner` each, each its own priced capability, with no orchestrator config to touch and no restart. It also stores the upstream keys encrypted and reports per-endpoint request stats, which is what an operator running more than a handful of these actually needs. ## Run offchain (free) ```sh cp .env.example .env # fill in HF_TOKEN; ignore the on-chain block docker compose up -d -curl -sk https://localhost:8935/discovery | jq '.[].runners[].app' # confirm livepeer-example/stable-diffusion-3-medium registered +curl -sk https://localhost:8935/discovery | jq '.[].runners[] | {app, price_info}' # both capabilities, each with its price uv run client.py --prompt "a watercolor painting of a llama writing code" +uv run client.py --app livepeer-example/flux-1-schnell \ + --prompt "a watercolor painting of a llama writing code" --output flux-out.jpg docker compose down ``` -`compose.yml` brings up an orchestrator (`-useLiveRunners -liveRunnerConfig`) and the nginx runner. The client sends one prompt through the orchestrator and writes `api-proxy-out.jpg`. +`compose.yml` brings up an orchestrator (`-useLiveRunners -liveRunnerConfig`) and the nginx runner, which registers as two capabilities. The client sends one prompt through the orchestrator per call and writes the image; `--app` chooses the model and `--output` keeps the two results apart. ## Run on-chain (paid) -Layer `compose.onchain.yml` to run the orchestrator on-chain with a remote signer paying each call — one fixed payment per image, at the price `runners.json` advertises. For the required RPC and wallets see [On-chain (paid) setup](../README.md#on-chain-paid-setup) in the repo README. +Layer `compose.onchain.yml` to run the orchestrator on-chain with a remote signer paying each call — one fixed payment per image, at the price `runners.json` advertises for the capability called. For the required RPC and wallets see [On-chain (paid) setup](../README.md#on-chain-paid-setup) in the repo README. ```sh cp .env.example .env # fill in HF_TOKEN, RPC, network, keystore paths, accounts @@ -55,4 +64,4 @@ uv run client.py --prompt "a watercolor painting of a llama writing code" \ docker compose -f compose.yml -f compose.onchain.yml down ``` -Each call is one paid single-shot session — the orchestrator reserves it, takes one fixed payment, and releases it when the response returns. +Each call is one paid single-shot session: the orchestrator reserves it, takes one fixed payment, and releases it when the response returns. `--app` picks which price you pay. The signer's `MAX_PRICE_PER_UNIT` is one cap across both capabilities, so it has to clear the **highest** price in `runners.json`. diff --git a/api-proxy/client.py b/api-proxy/client.py index 02524bd..86144bd 100644 --- a/api-proxy/client.py +++ b/api-proxy/client.py @@ -5,6 +5,10 @@ ({"inputs": ""}); the runner forwards it verbatim and the image comes back as raw JPEG bytes in `result.content`. +The operator offers two models, so `--app` picks which one to call. Discovery +matches app ids exactly, which is the whole reason each model has its own: it is +what a caller selects and pays for. + Livepeer integration (grep `# Livepeer:`): 1. runner_selector() — discover orchestrators advertising the app 2. call_runner() — call the app through the orchestrator; a non-JSON response @@ -25,7 +29,8 @@ from livepeer_gateway.selection import runner_selector DEFAULT_DISCOVERY = "https://localhost:8935/discovery" -APP_ID = "livepeer-example/stable-diffusion-3-medium" +DEFAULT_APP = "livepeer-example/stable-diffusion-3-medium" +FLUX_APP = "livepeer-example/flux-1-schnell" # the other one this demo offers DEFAULT_OUTPUT = "api-proxy-out.jpg" log = logging.getLogger("api-proxy-client") @@ -37,6 +42,11 @@ def _parse_args() -> argparse.Namespace: "--prompt", default="a watercolor painting of a llama writing code" ) parser.add_argument("--output", default=DEFAULT_OUTPUT, help="output image path") + parser.add_argument( + "--app", + default=DEFAULT_APP, + help=f"app id to call; this demo also offers {FLUX_APP}", + ) parser.add_argument("--discovery", default=DEFAULT_DISCOVERY) parser.add_argument( "--signer", default="", help="Remote signer base URL (on-chain/paid path)." @@ -52,10 +62,10 @@ async def main() -> None: try: cursor = await runner_selector( # Livepeer: 1 discovery_url=args.discovery, # omit if the signer does discovery itself - app=APP_ID, + app=args.app, ) runner = cursor.candidates[0] - log.info("app_url=%s", runner.url) + log.info("app=%s app_url=%s", args.app, runner.url) result = await call_runner( # Livepeer: 2 runner=runner, # discovery metadata tells call_runner the price unit diff --git a/api-proxy/compose.yml b/api-proxy/compose.yml index 420a937..51c0520 100644 --- a/api-proxy/compose.yml +++ b/api-proxy/compose.yml @@ -1,6 +1,7 @@ # End-to-end offchain demo: orchestrator + the api-proxy "app" — a stock nginx -# attached as a static runner via runners.json (-liveRunnerConfig). There is no -# app code at all: nginx.conf.template is the whole runner. +# attached as two static runners via runners.json (-liveRunnerConfig). There is +# no app code at all: nginx.conf.template is the whole runner, and it serves two +# capabilities from one process because a capability is a registration. # # Needs HF_TOKEN (the upstream credential nginx injects) in a local .env — copy # .env.example. Once up, call it from the host with the SDK: @@ -34,8 +35,5 @@ services: environment: # The operator-held upstream credential (huggingface.co → settings → tokens). - HF_TOKEN=${HF_TOKEN:?set HF_TOKEN in .env (copy .env.example)} - # The one model this runner offers; swap it here, and rename the app id - # in runners.json to match (the app id is what callers discover). - - MODEL=stabilityai/stable-diffusion-3-medium-diffusers volumes: - ./nginx.conf.template:/etc/nginx/templates/default.conf.template:ro diff --git a/api-proxy/nginx.conf.template b/api-proxy/nginx.conf.template index f6498ac..07b797d 100644 --- a/api-proxy/nginx.conf.template +++ b/api-proxy/nginx.conf.template @@ -1,10 +1,16 @@ -# The whole runner: a stock nginx forwards each call to one pinned Hugging Face -# model URL and injects the operator's token. The orchestrator health-polls -# /health and reverse-proxies caller requests to /proxy. HF_TOKEN and MODEL are -# substituted from the environment by the nginx image's template mechanism. +# The whole runner: a stock nginx pins one Hugging Face model per route and +# injects the operator's token. HF_TOKEN is the only value substituted from the +# environment, by the nginx image's template mechanism. +# +# runners.json points each app id at its own path (http://app:8989/sd3), which +# the orchestrator preserves when forwarding, so each lands on its own block. server { listen 8989; + # One endpoint for both capabilities. `return 200` only reports that nginx is + # up, never that a model or the token is good. Hugging Face exposes no + # readiness endpoint, only metadata, and the orchestrator polls this every 5s + # with a 3s timeout, so a real upstream check does not belong here. location = /health { default_type application/json; return 200 '{"status":"ok"}'; @@ -14,10 +20,10 @@ server { # be spent on exactly this model, callers choose nothing but the payload. # Method and query string are part of "nothing": nginx would forward both # upstream as the caller sent them, so pin the method and drop the args. - location = /proxy { + location = /sd3/proxy { limit_except POST { deny all; } set $args ''; - proxy_pass https://router.huggingface.co/hf-inference/models/${MODEL}; + proxy_pass https://router.huggingface.co/hf-inference/models/stabilityai/stable-diffusion-3-medium-diffusers; proxy_set_header Authorization "Bearer ${HF_TOKEN}"; # Hugging Face content-negotiates: with the SDK's Accept: # application/json it answers with a base64 PNG in a JSON string @@ -26,4 +32,15 @@ server { proxy_ssl_server_name on; proxy_read_timeout 120s; # a hosted diffusion model can take tens of seconds } + + # One more capability is one more location here plus one runners.json entry. + location = /flux/proxy { + limit_except POST { deny all; } + set $args ''; + proxy_pass https://router.huggingface.co/hf-inference/models/black-forest-labs/FLUX.1-schnell; + proxy_set_header Authorization "Bearer ${HF_TOKEN}"; + proxy_set_header Accept "image/jpeg"; + proxy_ssl_server_name on; + proxy_read_timeout 120s; + } } diff --git a/api-proxy/runners.json b/api-proxy/runners.json index 7f52f63..eaad64e 100644 --- a/api-proxy/runners.json +++ b/api-proxy/runners.json @@ -1,13 +1,22 @@ { "runners": [ { - "label": "api-proxy", + "label": "api-proxy-sd3", "app": "livepeer-example/stable-diffusion-3-medium", - "runner_url": "http://app:8989", - "health_url": "/health", + "runner_url": "http://app:8989/sd3", + "health_url": "http://app:8989/health", "mode": "single-shot", - "capacity": 1, + "capacity": 4, "price_info": { "price": 0.0001, "unit": "fixed" } + }, + { + "label": "api-proxy-flux", + "app": "livepeer-example/flux-1-schnell", + "runner_url": "http://app:8989/flux", + "health_url": "http://app:8989/health", + "mode": "single-shot", + "capacity": 4, + "price_info": { "price": 0.00004, "unit": "fixed" } } ] }