Skip to content

observability.md

Latest commit

 

History

History
102 lines (87 loc) · 5.53 KB

observability.md

File metadata and controls

102 lines (87 loc) · 5.53 KB
Raw

Observability

Metrics

GET /metrics renders the Prometheus registry.

Metric Type Labels
gateway_requests_total counter route, status
gateway_request_duration_seconds histogram route
gateway_node_duration_seconds histogram node (pipeline stage)
gateway_tokens_total counter kind (prompt/completion)
gateway_cache_hits_total counter —
gateway_ledger_write_failures_total counter —
gateway_upstream_connect_retries_total counter account
gateway_upstream_status_retries_total counter account, status
gateway_thinking_signature_review_total counter result (match/mismatch/miss/no_evidence)
gateway_thinking_signature_cache_events_total counter event
gateway_mcp_requests_total counter server (configured name), method (initialize / tools/list / tools/call / resources / prompts / ping / stream / other), result (HTTP status, denied, upstream_error, reply_unreadable, masked, blocked)
gateway_model_fallbacks_total counter from, to

gateway_requests_total is recorded by router middleware, so every response — including error statuses and the realtime WebSocket upgrade — is counted, which makes error-rate dashboards possible. All labels are bounded by the config, never by a caller: route templates, status codes, protocol/stage names, configured MCP server names and configured model names (gateway_model_fallbacks_total) — no per-key or per-user cardinality.

Traces

Every request runs inside one span named by its route template (/v1/chat/completions, /v1/messages, …). It carries http.request.method, http.route, http.response.status_code and, once the pipeline has served the request, the same fields as the access log: gw.request_id, gw.surface, gw.model, gw.protocol, gw.account, gw.tenant, gw.ak_id, gw.user_id, gw.prompt_tokens, gw.completion_tokens, gw.decisions. A streamed response keeps its span open until the last frame, so the span duration is the whole turn; a 5xx marks the span as an error.

The span exports over OTLP/HTTP (protobuf) when the standard OpenTelemetry environment names a collector — nothing else to configure:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318   # or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
export OTEL_SERVICE_NAME=gw                                # default when unset
export OTEL_TRACES_SAMPLER=parentbased_traceidratio        # default parentbased_always_on
export OTEL_TRACES_SAMPLER_ARG=0.1

OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_TIMEOUT apply as usual. An inbound W3C traceparent header joins the request to the caller's trace; the gateway's span is then a child of it. Without a collector the span is disabled at its call site and costs nothing per request; with one, span attributes are recorded in-process and batches leave on a background thread.

Access log

One structured line per successfully served request goes to stdout (via tracing; control level with RUST_LOG), carrying surface, request_id, ak_id (a stable SHA-256 fingerprint, never the bearer credential), product, user_id, model, protocol, account, prompt_tokens, completion_tokens, total_tokens, latency_ms, and decisions — the pipeline's routing trail for that request (quota fallback, variant pick, degrade, DLP/moderation outcomes). Errored requests are counted by gateway_requests_total{status} rather than logged. request_id joins the access log to the ledger row and the audit events for the same request.

Billing ledger

GET /internal/ledger?limit=N (global admin token) returns the most recent N billing records, oldest-first within the page; count is always the true total, independent of the page size. Records persist when a SQLite or Postgres store is configured and can be capped with storage.ledger_max_rows, though the cap is not hard: pruning spares rows not yet folded into the usage rollup, so the table can briefly exceed it under rollup lag. Each record carries request_id, the access key, product, tenant, user_id (effective end user), the requested model and the served_model (differs after a quota fallback), protocol, account, token counts, billed_units (characters / seconds / search units on the unit-priced surfaces; also summed in /admin/usage), charged cost_micros and vendor_cost_micros, created_at_epoch_secs, the PTU-spillover flag, and an estimated flag (set when counts came from an aborted stream rather than a vendor usage payload). Writes are idempotent by request_id; a transient store failure is held in a bounded repair queue and retried with backoff — eight attempts, about 13 s — before the batch is dropped and counted in gateway_ledger_write_failures_total. A full queue applies backpressure to settlement until the repair worker makes room, so overload alone never discards an accepted row. Per-user usage additionally rolls into durable minute buckets every minute, so GET /admin/usage/users stays correct after ledger_max_rows pruning (see Governance).

Audit trails

Three operator-facing audit surfaces, all under the gated /admin prefix and covered in Governance: GET /admin/audit/events (content-safety hits, no prompt text), GET /admin/audit/ops (admin-plane mutations with source IP), and GET /admin/usage/users (per-user cost). Content retention, when a tenant enables it, is read back via GET /admin/audit/content/{request_id}.