Skip to content

About

The Ruby client for Nitro Intelligence

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Nitro Intelligence

The entrypoint to everything AI.

This component aims to consolidate and standardize AI features implemented in the host application.

Public API

The gem's public API is what its API reference documents, and follows Semantic Versioning: it changes incompatibly only in a major release. The reference is generated from the source with YARD: run bundle exec yard doc and open doc/api/index.html, or browse it for any released version on RubyDoc.info. Anything marked @api private in the source is left out of it, may change in any release, and is not for callers. This README and ASSISTANTS.md are guides to using the gem, not a list of everything public.

Configuration

NitroIntelligence is configured via an initializer in the host application. All external dependencies (e.g. inference API credentials, observability settings) must be injected at boot time:

NitroIntelligence.configure do |config|
  # Standard Rails integrations
  config.logger         = Rails.logger           # Logger instance
  config.environment    = Rails.env              # e.g. "production", "test"
  config.cache_provider = Rails.cache            # ActiveSupport cache store

  # Inference (LLM) settings
  config.inference_api_key  = "..."              # API key for the inference service
  config.inference_base_url = "https://..."      # Base URL for the inference service (optional; defaults to the inference gateway)
  config.inference_max_retries = 0               # Retries the SDK makes itself (optional; defaults to 0 - see "Retries and Timeouts")
  config.inference_timeout     = 600             # Request timeout in seconds (optional; defaults to openai-ruby's)

  # Observability (Langfuse) settings
  config.observability_base_url = "https://..."  # Base URL for the observability service (optional; defaults to Cerebro)
  config.observability_projects = [              # Array of project credential hashes
    {
      "slug"       => "my-feature-project",
      "id"         => "project-id",
      "public_key" => "pk-...",
      "secret_key" => "sk-...",
      # "mask_content" => true,                  # Optional: keep prompts and responses out of Cerebro and the gateway logs, see "Masking Content"
    },
  ]

  # Nitro Intelligence Assistants settings (optional)
  config.assistants_config = {                   # See "Assistants" below
    "base_url" => "https://nip-assistants.example.com",
    "user_id" => "my-app",
    "definitions" => {
      "candidate-concierge" => {},
    }
  }

  # Model configuration
  config.model_config = {
    "default_audio_transcription_model" => "Qwen/Qwen3-ASR-1.7B",
    "default_text_model" => "Qwen/Qwen3.6-35B-A3B",
    "default_image_model" => "google/gemini-3.1-flash-image",
    "default_text_to_speech_model" => "openai/gpt-4o-mini-tts",
    "default_decision_model" => "typesafe/jev-1.14",
    "default_embedding_model" => "Qwen/Qwen3-Embedding-0.6B",
    "default_rerank_model" => "Qwen/Qwen3-Reranker-0.6B",
    "models" => [
      {
        "name" => "Qwen/Qwen3.6-35B-A3B",
        "type" => "text"
      },
      {
        "name" => "Qwen/Qwen3-ASR-1.7B",
        "type" => "audio_transcription"
      },
      {
        "name" => "google/gemini-3.1-flash-image",
        "type" => "image",
        "aspect_ratios" => ["1:1", "2:3", "3:2", "3:4", "4:3", "9:16", "16:9"],
        "resolutions" => ["512", "1K", "2K"],
        "omit_output_fields" => ["provider_specific_fields.thought_signatures"]
      },
      {
        "name" => "openai/gpt-4o-mini-tts",
        "type" => "text_to_speech",
        "default_voice" => "marin",
        "default_response_format" => "mp3",
        "voices" => ["echo", "nova", "marin", "cedar"],
        "response_formats" => ["mp3", "wav"]
      },
      {
        "name" => "typesafe/jev-1.14",
        "type" => "decision"
      },
      {
        "name" => "Qwen/Qwen3-Embedding-0.6B",
        "type" => "embedding",
        "query_prefix" => "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:"
      },
      {
        "name" => "Qwen/Qwen3-Reranker-0.6B",
        "type" => "rerank"
      }
    ]
  }
end

Configuration Keys

Key Type Default Description
logger Logger Logger.new($stdout) Logger used for diagnostic output
environment String "test" Runtime environment name
current_revision String "" The host application's revision, such as its commit, recorded on each trace in Cerebro as its release
cache_provider cache store NullCache ActiveSupport-compatible cache store
inference_api_key String "" API key for the LLM inference service
inference_base_url String "https://inference.powerhome.ai" Base URL for the LLM inference service. Defaults to the shared inference gateway, so only a host talking to a different one needs to set it
inference_max_retries Integer 0 Retries the SDK makes itself after a failed request. Off by default — see Retries and Timeouts
inference_timeout Numeric nil Request timeout in seconds. nil keeps openai-ruby's default of 600
observability_base_url String "https://cerebro.powerhome.ai" Base URL for the Langfuse observability service. Defaults to Cerebro, so only a host talking to a different one needs to set it
observability_projects Array<Hash> [] Langfuse project credentials (slug, id, public_key, secret_key). Add "mask_content" => true to keep a project's prompts and responses out of Cerebro and the gateway logs, see Masking Content
observability_user_id String "" User ID recorded on observed traces when a request passes no user_id parameter — see Users
assistants_config Hash {} Assistants to make addressable by key. base_url (String, default: "https://assistants.powerhome.ai") and user_id (String, default: "default-user") are shared by every entry; definitions (Hash) holds one entry per assistant, keyed by what it is looked up with, each able to override a shared value. Without definitions the hash is read as credentials for a single Assistants.new, which takes the same two defaults — see Assistants
model_config Hash {} Model defaults and per-model settings. Top-level keys: default_text_model, default_audio_transcription_model, default_image_model, default_text_to_speech_model, default_decision_model, default_embedding_model, default_rerank_model, and models (array of per-model hashes keyed by name and type, with type-specific options like aspect_ratios/resolutions for images or voices/response_formats for TTS, and for embeddings the optional, model-specific query_prefix and document_prefix: Qwen3-Embedding takes only query_prefix, E5 both, OpenAI's models neither, as Embeddings describes)

Basic Usage

OpenAI API/LLM Requests

A simple LLM call can be invoked as follows:

client = NitroIntelligence::Client.new
client.chat(message: "Why is the sky blue?")
content = result.choices.first&.message&.content

This component handles setting defaults for most things, such as model, host, keys, etc.

You may also use openai-ruby compatible syntax with this wrapper by passing the parameters keyword in. For example:

client = NitroIntelligence::Client.new
client.chat(parameters: { model: "Qwen/Qwen3.8-27B", messages: [{ role: "user", content: "Why is the sky blue?" }]})

Providing Parameters

Parameters such as 'max_tokens' and 'temperature' can be passed in under the parameters key.

client = NitroIntelligence::Client.new
client.chat(parameters: { model: "Qwen/Qwen3.8-27B", max_tokens: 1000, temperature: 0.7, messages: [{ role: "user", content: "Why is the sky blue?" }]})

For a full list of supported parameters, see the API reference here.

Retries and Timeouts

The client does not retry failed requests by default. openai-ruby on its own retries a 429, 408, 409 or 5xx twice within about two seconds, but none of those is worth repeating that quickly from here: the inference gateway has already failed over between providers before it answers, and a rate limit or exhausted budget it passes on will not clear within a second. Each retry only adds load, and for billable work such as image generation a retry after a timeout can pay twice. A caller that wants retries decides when, typically in a job with backoff, and can tell a rate limit from an exhausted budget first (see the gateway's error guidance).

To retry anyway, or to change the timeout, set them for every client in the configuration (inference_max_retries, inference_timeout), for one client with the same names, or for one request with openai-ruby's request_options. The narrower setting wins.

# For one client
client = NitroIntelligence::Client.new(inference_max_retries: 2, inference_timeout: 30)

# For one request, with any method
client.chat(message: "Why is the sky blue?", parameters: { request_options: { max_retries: 2, timeout: 30 } })
client.generate_image(message: "A bear installing a window", parameters: { request_options: { timeout: 120 } })

Audio Transcription

Nitro Intelligence can be used to transcribe audio from a file into text.

With observability enabled, the source audio is attached to the generation observation as input.

Basic examples of usage:

client = NitroIntelligence::Client.new
audio = File.open('sample.m4a', 'rb')
result = client.transcribe_audio(audio_file: audio)

result is a OpenAI::Models::Audio::Transcription object:

<OpenAI::Models::Audio::Transcription:0x2fdc8 {:text=>"Hello, how are you doing today?", :usage=>{:input_tokens=>28, :output_tokens=>10, :total_tokens=>38, :type=>:tokens, :input_token_details=>{:audio_tokens=>28, :text_tokens=>0}}}>

To use a prompt, simply initialize your client with the observability_project_slug and pass prompt_name into parameters:

client = NitroIntelligence::Client.new(observability_project_slug: "my-test-project")
audio = File.open('sample.m4a', 'rb')
result = client.transcribe_audio(audio_file: audio, parameters: {prompt_name: "Spanish Converter"})

puts result

# <OpenAI::Models::Audio::Transcription:0x2fd8c {:text=>"Hola, ¿cómo estás hoy?", :usage=>{:input_tokens=>37, :output_tokens=>9, :total_tokens=>46, :type=>:tokens, :input_token_details=>{:audio_tokens=>28, :text_tokens=>9}}}>

Text-to-Speech

Nitro Intelligence can be used to create spoken-word audio files from text.

With observability enabled, generated audio is labeled as output and attached to the generation observation.

Basic example of usage:

client = NitroIntelligence::Client.new
tts = client.text_to_speech(message: 'Hello, this is Power Home Remodeling Group.')

# tts is a StringIO object, we can write the file to disk
File.binwrite('tts.mp3', tts.string)

Supplying custom parameters:

client = NitroIntelligence::Client.new
tts = client.text_to_speech(
  message: 'Hello, this is Power Home Remodeling Group.',
  parameters: {
    voice: 'marin',
    response_format: 'mp3',
    instructions: 'Translate the English into Spanish before speaking.',
    speed: 1.25
  }
)

# tts is a StringIO object, we can write the file to disk
File.binwrite('tts.mp3', tts.string)

For a full list of supported parameters, see the API reference here. Note that voice and response_format are further constrained to the voices and response_formats listed for the chosen model in config.model_config.

Decisions

A decision model answers questions about some content with a probability for each answer you allow, rather than generated text. decide sends the content as state (a string, hash or array) and a hash of named questions to the gateway's System One route, POST /v1/systemone. Each question is noul (yes or no), choice (one of the options in criteria) or score (one of the ordered levels in criteria, the first being 0). See the Inference Gateway's decision model docs for when to use one and how far to trust its probabilities.

client = NitroIntelligence::Client.new
decision = client.decide(
  state: { subject: "Duplicate charge", message: "I was charged twice. Please help." },
  questions: {
    billing: { type: "noul", instructions: "Is this message about billing?" },
    team: {
      type: "choice",
      instructions: "Which team should handle this message?",
      criteria: {
        payments: "Charges, refunds and invoices",
        scheduling: "Appointments and visits",
        other: "Anything else, or not enough information to tell",
      },
    },
    urgency: {
      type: "score",
      instructions: "How urgent is this message?",
      criteria: ["Can wait", "Needs attention this week", "Needs attention today"],
    },
  }
)

decision.answers[:billing].noul          # => 0.98, the probability of yes
decision.answers[:team].choice           # => "payments"
decision.answers[:team].confidence       # => 0.91
decision.answers[:team].probabilities    # => {"payments" => 0.95, "scheduling" => 0.01, "other" => 0.04}
decision.answers[:urgency].score         # => 1.7, the expected level, which may fall between levels
decision.answers[:urgency].legend        # => {0 => "Can wait", 1 => "Needs attention this week", 2 => "Needs attention today"}
decision.answers[:urgency].probabilities # => {0 => 0.1, 1 => 0.1, 2 => 0.8}

answers is keyed by question name and accepts strings or symbols. Each answer is a NitroIntelligence::Decision::NoulAnswer, ChoiceAnswer or ScoreAnswer; an answer of any other type is returned as the gateway sent it. decision.usage holds the token counts, also by string or symbol, and decision.last_response the HTTP response, including the gateway's headers.

The model defaults to default_decision_model and must be listed in config.model_config with the type decision. parameters takes model, assets (images that state and the questions refer to, see the System One docs; the gateway refuses assets for a model it reaches through OpenRouter, which includes typesafe/jev-1.14), metadata, request_options and, with observability, trace_name, trace_seed, user_id, session_id and tags, described under Observability. Decisions take no managed prompt: prompt_name and prompt_variables are ignored, so put the instructions in questions. With observability enabled, each call is recorded in Cerebro as a generation with the state and questions as input, the answers as output, and the gateway's cost.

The client does not limit the request. To keep a feature portable to models NIP hosts itself, stay within the gateway's portable subset: up to 64 questions, 26 options per choice question and 10 levels per score question, with text state.

Embeddings

An embeddings model turns text into a vector, so text with similar meaning can be found by comparing vectors. Embeddings go through the gateway like any other request. The gateway must serve an embeddings model; see the Inference Gateway docs for what it serves.

Many embeddings models are trained to embed a search query differently from the documents it is searched against, so the client has a method for each. embed_documents embeds the documents for an index, and embed_query the query to search it with. Both return a NitroIntelligence::Embeddings with one vector per input, in input order:

# Documents, such as an index rebuild
embeddings = client.embed_documents(documents: pages.map(&:description), parameters: { dimensions: 512 })
embeddings.vectors # => [[0.012, -0.034, ...], ...], one per page

# A query to search them with
client.embed_query(query: "How do I request time off?", parameters: { dimensions: 512 }).vector # => [0.021, 0.008, ...]

embed_query also takes a list of queries, such as an evaluation set, and returns a vector for each in vectors. embed_documents also takes a single document.

parameters takes model, dimensions, batch_size, user, metadata, request_options and, with observability, trace_name, trace_seed, user_id, session_id and tags. The model defaults to default_embedding_model and must be listed in config.model_config with the type embedding.

  • dimensions shortens the vectors, for a model that supports it, as Qwen3-Embedding does. Embed queries and documents with the same model and the same dimensions, or the vectors cannot be compared.
  • batch_size caps how many inputs go in one request, 128 by default. A longer list is sent in several requests and returned as one result, recorded as one observation, so an index rebuild of a few thousand entries is one call.

A model's query_prefix and document_prefix in config.model_config are what it expects before a query's text and before a document's, in the form it was trained on. Qwen3-Embedding takes an instruction before each query and nothing before a document, as in the configuration above; E5 takes "query: " and "passage: ". Leave both out for a model that embeds queries and documents the same way, such as OpenAI's. Switching to another embeddings model only means changing its model_config entry, but vectors from different models cannot be compared, so it also means embedding your stored documents again.

embeddings.usage holds prompt_tokens and total_tokens summed over every request, and embeddings.responses the gateway's responses, one per batch, each with its last_response. With observability enabled, each call is recorded in Cerebro as an embedding observation named embed-query or embed-documents, with the input as the model received it, prefix included, the gateway's cost added up over every batch, and the number and size of the vectors as output. The vectors themselves are never recorded. If the gateway did not price every batch, the cost is left unset rather than undercounted. If a batch fails after others succeeded, the failed call still records what those cost before the error is raised.

Embeddings take no managed prompt: prompt_name and prompt_variables are ignored.

Reranking

A reranker scores how relevant each of a list of documents is to a query. It is more accurate than comparing vectors, and suits reordering the few dozen candidates a vector search returned. The gateway must serve a reranker; see the Inference Gateway docs for what it serves.

rerank takes a query and a list of documents and returns a NitroIntelligence::Ranking, most relevant first:

ranking = client.rerank(
  query: "How do I request time off?",
  documents: candidates.map(&:description),
  parameters: { top_n: 5 }
)
ranking.results.first.index           # => 3, the position in `documents`
ranking.results.first.relevance_score # => 0.97, from 0 to 1
candidates.values_at(*ranking.results.map(&:index))

parameters takes model, top_n (return only the most relevant this many, a positive integer, applied after sorting even when the provider ignores it), metadata, request_options and, with observability, trace_name, trace_seed, user_id, session_id, tags and trace_documents. The model defaults to default_rerank_model and must be listed in config.model_config with the type rerank. ranking.meta holds what the provider billed, and ranking.last_response the HTTP response, including the gateway's headers.

With observability enabled, each call is recorded in Cerebro as a generation, since Langfuse has no reranker type, with the query and documents as input, each document's index and score as output, the input tokens as usage, any search units the provider billed in the metadata, and the gateway's cost. Pass trace_documents: false when your own trace already records the candidates, such as an embedding call or a search step in the same trace, and only their number is recorded. Reranking takes no managed prompt: prompt_name and prompt_variables are ignored.

Image Editing and Generation

Nitro Intelligence can be used for image editing and generation

Basic examples of usage:

Image Generation

client = NitroIntelligence::Client.new
result = client.generate_image(message: "Create an image of a bear installing a window.")

result is a NitroIntelligence::ImageGeneration object, and the generated image can be accessed via result.generated_image which returns a NitroIntelligence::Image. Gemini image models that think can return their interim drafts alongside the final render; generated_image is always the final one.

With observability enabled, result.trace_id and result.observation_id identify the trace and generation observation. Both are nil without observability.

A NitroIntelligence::Image carries the file in byte_string, its file_extension (such as "png") and mime_type (such as "image/png"), and its width and height in pixels.

You may write the file to disk:

File.binwrite("my_generated_image.#{result.generated_image.file_extension}", result.generated_image.byte_string)

Image Editing and Uploading Reference Images

To edit an image, provide your source image as a byte string, along with any references you would like to include.

client = NitroIntelligence::Client.new
house = File.binread("./house.jpg")
siding = File.binread("./siding.png")
result = client.generate_image(message: "Replace the siding in the image of the house with the new siding I have provided.", target_image: house, reference_images: [siding])

Using Prompts

See Observability[## Observability] for more details. Basic usage looks like:

client = NitroIntelligence::Client.new(observability_project_slug: "sample-project-slug")
house = File.binread("./house.jpg")
siding = File.binread("./siding.png")
result = client.generate_image(target_image: house, reference_images: [siding], parameters: {prompt_name: "Sample Prompt Name"})

With observability enabled, the input and generated images are attached to the generation observation. Cerebro's Formatted observation preview displays them together, including in the tracing list's preview panel.

Image Configuration

You can specify parameters such as model to use, aspect ratio and resolution via the parameters key:

client = NitroIntelligence::Client.new
result = client.generate_image(message: "Create an image of a bear installing a window.", parameters: {aspect_ratio: "4:3", resolution: "512"})

Observability

Setup

If your feature is setup for observability, simply pass the observability project and desired prompt name in when invoking requests. It is also preferable to pass in a source and any other useful metadata for later debugging. For example:

client = NitroIntelligence::Client.new(observability_project_slug: "fake-feature-project")
client.chat(
  message: "Why is the sky blue?",
  parameters: {
    prompt_name: "My Prompt",
    # prompt_label: "debug",
    # prompt_version: "v2",
    metadata: {
      source: self.class,
    },
  }
)

If no prompt_label or prompt_version is provided, the 'production' prompt is used by default.

Exporting Completed Observations

Observations are exported in the background. Use the same shared project client for tracing and flushing when a job needs to export completed observations before submitting their IDs for human review:

project_client = NitroIntelligence.project_client_registry.fetch("my-feature-project")
raise "Observation export failed or timed out" unless project_client.flush(timeout: 30)

# Submit the completed observations' IDs to the annotation queue here.

flush waits only for this project's observations queued or in flight when called. Observations completed later do not extend the wait. The client remains usable, and active observations are not ended.

It returns true when those exports succeed or nothing needs exporting. It returns false if exporting fails or cannot start, an observation being flushed is dropped, the wait times out, or shutdown prevents completion. Calls made during or after shutdown return false. Earlier export failures are not reported.

The timeout defaults to 30 seconds; exports continue after a caller times out. With timeout: 0, a running client succeeds only if nothing is queued or in flight.

It does not flush queued scores or wait for evaluators, and successful export does not guarantee immediate visibility in Cerebro or annotation queues. Reserve NitroIntelligence.project_client_registry.shutdown_all for process teardown. See the ProjectClient#flush API reference for details.

Prompt Fallbacks

Pass prompt_fallback_name to name a second prompt to use when the one you asked for isn't there. The prompt in prompt_name is tried first; if it doesn't exist, or its lookup fails, the fallback is used instead:

client = NitroIntelligence::Client.new(observability_project_slug: "fake-feature-project")
client.chat(
  message: "Why is the sky blue?",
  parameters: {
    prompt_name: "My Prompt - Scheduling", # used when it exists
    prompt_fallback_name: "My Prompt",     # used when it doesn't
    # prompt_fallback_label: "debug",
    # prompt_fallback_version: "v2",
    prompt_variables: {
      appointment_id: "1234",
    },
  }
)

This is for shipping a variant of a prompt — per domain, per locale, per experiment — without having to create one for every caller. Build the specific name however your feature likes; the fallback is just the prompt you'd have asked for anyway.

In practice the caller composes the specific name from something it already knows. Summarising a call, for example: each queue holds a different kind of conversation, so a queue that needs its own wording can have its own prompt while the rest use the general one.

base = "call_summary"
queue = call.queue.slug # "sales", "support", "collections", ...

client.chat(
  parameters: {
    prompt_name: [base, queue].compact.join("."), # call_summary.collections
    prompt_fallback_name: base,
    prompt_variables: { transcript: call.transcript },
  }
)

The fallback is looked up at its own prompt_fallback_label and prompt_fallback_version, and inherits neither from the prompt you asked for first. They describe a different prompt: a version number is minted per prompt, so version 3 of one name is an unrelated revision of another, and a label the first prompt carries need not exist on the fallback at all.

That means an omitted prompt_fallback_label resolves the fallback the same way any unlabelled lookup does — at the 'production' label. If you want both prompts to come from the same label, name it for both.

Whichever prompt is used is the one linked to the trace and merged into your request. If the fallback is missing too, the error is raised rather than swallowed — a request that asked for a prompt shouldn't quietly run without one.

Custom Trace Names

To provide custom trace names to the observability platform, you can pass 'trace_name' in parameters. Example:

client = NitroIntelligence::Client.new(observability_project_slug: "fake-feature-project")
client.chat(
  message: "Why is the sky blue?",
  parameters: {
    prompt_name: "My Prompt",
    metadata: {
      source: self.class,
    },
    trace_name: "custom-trace-name"
  }
)

Custom Trace IDs

To generate a deterministic trace ID in the observability platform, you can pass trace_seed as a parameter. This is useful when you want a stable identifier that is derived from a specific domain value (e.g., document ID). The same trace_seed will produce the same trace ID, making it easier to correlate multiple events. Each seeded call creates a root observation in that trace, including when another observation is active. With Langfuse v4 event reads, trace input and output come from the most recently ingested root observation. Calls without a seed keep their active parent observation.

document_id = "document-123"
client = NitroIntelligence::Client.new(observability_project_slug: "fake-feature-project")
client.chat(
  message: "summarize the document",
  parameters: {
    trace_seed: document_id,
  }
)

Grouping Traces: Users, Sessions and Tags

user_id attributes traces to a user of your application, session_id groups related traces, and tags label them for filtering. All three are optional. session_id and tags are sent only when you set them.

client = NitroIntelligence::Client.new(observability_project_slug: "fake-feature-project")
client.chat(
  message: "why did this deploy fail?",
  parameters: {
    user_id: current_user.id.to_s,
    session_id: "deploy-9f2c1ab",
    tags: ["deployment-failure-analyzer"],
  }
)

Users

A trace that carries a user_id shows up under that user in Cerebro's Users view, which lists each user with their trace count, token usage and cost, and links through to their traces. The trace list can also be filtered by user.

When a request passes no user_id, the observability_user_id configuration key is used instead, so a host can attribute all of its traces to one identity (the application itself, say) and override it per request for the traces that belong to a person:

NitroIntelligence.configure do |config|
  config.observability_user_id = "nitro-web"
end

Use an identifier that is stable and not itself personal data — a database ID rather than an email address or name — since it is stored on every trace and shown in Cerebro. It must be a String of at most 200 characters; langfuse-rb drops any other value with a warning, so convert numeric IDs with to_s.

user_id is recorded only in Cerebro. It is not sent to the inference gateway, so to attribute gateway spend to a user as well, also put the identifier in metadata (see below).

Correlating With Application Logs and the Inference Gateway

Every observed request is correlated across three systems automatically:

  • Observability platform → inference gateway. The trace ID is sent to the gateway as x-litellm-trace-id, and metadata is sent as x-litellm-spend-logs-metadata (dropped if it exceeds 4KB).

    The two are independent. Metadata is sent whenever you set it, observed or not, so gateway spend can be attributed even without observability. The trace ID is sent only on the observed path - it comes from the observation being recorded, never from whatever span happens to be active, so a client built without an observability_project_slug sends none even inside a host application with its own OpenTelemetry instrumentation.

  • Inference gateway → observability platform. When a request fails, the gateway's own request identifier is read from the error response and recorded on the observation as litellm_call_id metadata.

  • Application users → observability platform. Pass user_id to see a user's traces, usage and cost together in Cerebro — see Users.

  • Application logs → observability platform. Put whatever your logs are keyed by into metadata — it lands on the trace and in the gateway's spend logs.

client.chat(
  message: "why did this deploy fail?",
  parameters: {
    trace_seed: deploy_url,
    metadata: {
      source: self.class,
      rails_request_id: request_id,
      job_id: job_id,
    },
  }
)

To log the trace ID from application code, derive it from the same seed with NitroIntelligence::Trace.create_id(seed:).

Failed Requests

A request that raises still produces a usable trace. The observation records:

  • the input that was sent (recorded before the request runs)
  • level: "ERROR" and a status_message carrying the exception class and message
  • litellm_call_id metadata when the gateway returned one

The exception is then re-raised, so this changes what is observed, not how callers handle failures. Image generation is the one exception to input recording: its input carries base64 payloads that are replaced with media references only on success, so it records no input up front.

Masking Content

A project can opt in to recording what happened without recording what was said. Set "mask_content" => true on its entry in observability_projects; projects that do not set it are unchanged.

For a masked project, every observed request (chat, image generation, audio transcription, text-to-speech, decisions, embeddings and reranking) is still traced, with its model, usage, cost, prompt link, tags, user, session, metadata and gateway call ID, but:

  • The input and output of the generation and of the trace are recorded as [masked]. The whole value is replaced, so reasoning, tool calls and anything else in a response are masked however the provider shaped it.
  • A failed request records the error class (for example OpenAI::Errors::BadRequestError) as its status message, without the error's own message, which can quote the request or the response.
  • No media is uploaded. Generated images, generated speech and transcribed audio never reach Cerebro. Callers still receive them as usual.
  • The request carries x-litellm-enable-message-redaction, so the inference gateway records redacted-by-litellm in place of the messages and the response in its own logs. It still records the model, usage, cost and spend-logs metadata, so gateway spend stays attributable.

Masking is not applied to what you choose to put on a trace yourself. Do not put prompt or response content in metadata, tags, session_id, user_id or trace names; metadata is the place for the identifiers that join a trace to your application's logs, and none of it is masked.

Two things this does not cover:

  • The gateway's redaction is by field, and misses some. It replaces the fields it knows about, such as content and reasoning_content, rather than the whole response, so a field a provider adds beyond those is still logged. Reasoning that a provider returns under provider_specific_fields is one such field today; see powerhome/nitro-intelligence#704.
  • Prompts you fetch from Cerebro are unaffected. Masking applies to traces, not to prompt management.

Tracing Application Steps

Each observed request is a trace of its own by default. A feature whose work is more than one model call, such as a search that embeds a question, queries an index, reranks the results and filters them by permission, can record that work as one trace with NitroIntelligence::Tracer: its own steps as observations, and the gem's requests nested among them.

tracer = NitroIntelligence::Tracer.new(observability_project_slug: "page-finder")
client = NitroIntelligence::Client.new(observability_project_slug: "page-finder")

pages = tracer.trace("Page Finder search", input: { question: }, user_id: current_user.id.to_s, session_id:) do |root_observation|
  query = client.chat(message: question) # recorded inside the trace

  hits = root_observation.observe("Elasticsearch", as_type: :retriever, input: { k: 50, boost: 2 }) do |step|
    search(query).tap { |hits| step.output = hits.map { |hit| { id: hit.page_id, score: hit.score } } }
  end

  allowed = root_observation.observe("Permission filter", input: { candidates: hits.size }) do |step|
    permitted(hits).tap { |pages| step.output = { allowed: pages.size } }
  end

  root_observation.output = allowed.map(&:page_id)
  @trace_id = root_observation.trace_id # to score the search when the user clicks a result
  allowed
end

Tracer#trace records its block as a trace with the given name, user, session, tags and metadata, and the environment and release the gem's own requests record. user_id defaults to observability_user_id, as it does for a request. It yields the trace's root observation: the input and output set on it are the trace's input and output too, while metadata set on it is recorded on that observation only. trace_seed: gives the trace the ID NitroIntelligence::Trace.create_id gives for the seed; otherwise the ID is random, and root_observation.trace_id reads it.

Observation#observe records a step inside the trace, or inside another step, around a block. as_type is a Langfuse observation type: :span by default, or :retriever, :tool, :agent, :chain, :evaluator, :guardrail, :embedding or :generation. Observation#event records a point in time, such as the results a user was shown. An exception raised inside a block is recorded on its step at level ERROR, and re-raised.

The gem's requests made inside the block are recorded as observations under the step that is open, not as traces of their own, when:

  • the client records in the same Cerebro project as the tracer. A request recorded in another project is a trace of its own there, as it would be outside the block.
  • the request passes no trace_seed. A seeded request is recorded in the seed's trace, as it is anywhere.

Such a request records its own observation only: the trace's name, input, output, user, session and tags are the ones the tracer was given, and the request's trace_name, user_id, session_id and tags are not recorded. Its metadata is still recorded on its observation and sent to the inference gateway.

A trace of its own started inside the block (a request to another project, a seeded request, or another Tracer#trace) neither takes the open trace's user, session, tags and metadata nor changes them. It takes only what the application propagated before opening the trace, such as with Langfuse.propagate_attributes, as it would outside the block.

A project that masks content records the input and output of every step and event as [masked], and a failure's error class without its message, as it does for the gem's requests. Metadata, names, user, session and tags are not masked.

Scoring

You can use NitroIntelligence::Reporter to evaluate existing traces. Calling NitroIntelligence::Reporter#score lets you attach metrics to a trace in the observability platform.

By Trace ID

reporter = NitroIntelligence::Reporter.new(observability_project_slug: "fake-feature-project")
reporter.score(name: "precision", value: 0.5, trace_id: "2377b0b38acf0f11b95504344fad6152")

By Trace Seed

Knowing the trace seed, you can use NitroIntelligence::Trace.create_id to get the trace ID.

document_id = "document-123"
reporter = NitroIntelligence::Reporter.new(observability_project_slug: "fake-feature-project")
trace_id = NitroIntelligence::Trace.create_id(seed: document_id)
reporter.score(name: "precision", value: 0.5, trace_id:)

An Observation, Later

A score can be recorded from a later request than the one that made the trace, such as a click on a search result, given the trace's ID. Pass observation_id: to score one observation in the trace, as Observation#id gives it, rather than the trace as a whole. comment: and metadata: are recorded with the score.

reporter = NitroIntelligence::Reporter.new(observability_project_slug: "page-finder")
reporter.score(trace_id: params[:trace_id], name: "clicked", value: true, data_type: :boolean, metadata: { position: 3 })

Using different data types for scoring

By default, scores submitted on a trace are numeric, but you can use the data_type keyword to change it to :boolean or :categorical. For example:

reporter.score(name: "precision", value: "Partially Correct", trace_id:, data_type: :categorical)

Numeric data is helpful for scores that happen on a sliding scale, like success rate or percent similarity. Boolean data is helpful to convey something that either passes or fails. Categorical data is helpful for scores that have pre-defined categories, like "Correct", "Partially Correct", and "Incorrect".

Dataset Items

NitroIntelligence::Reporter#create_dataset_item adds an item to one of the project's datasets, to evaluate prompts against later. It takes the item as Langfuse's POST /api/public/dataset-items does, and raises NitroIntelligence::Reporter::DatasetItemError, carrying the response's status and body, when Cerebro does not store it.

reporter = NitroIntelligence::Reporter.new(observability_project_slug: "fake-feature-project")
reporter.create_dataset_item({
  datasetName: "call-categorization",
  input: { transcript: },
  expectedOutput: "Rescheduled",
  sourceTraceId: trace_id,
})

Reaching Cerebro Directly

For what this gem does not wrap, such as a Langfuse API it has no method for, or a link to a trace, a project's details are available from its configuration:

project = NitroIntelligence.project_client_registry.fetch("fake-feature-project").project
project.id          # the project's id in Cerebro, as in its URLs
project.auth_token  # the Base64 credential for Cerebro's public API, without the "Basic " scheme

trace_url = "#{NitroIntelligence.config.observability_base_url}/project/#{project.id}/traces/#{trace_id}"
request["Authorization"] = "Basic #{project.auth_token}"

project.prompt(name, label:, version:) fetches one of the project's prompts, as client.prompt does.

NitroIntelligence.project_client_registry.fetch(slug) builds each project's client once per process, and raises NitroIntelligence::Observability::Project::NotFoundError for a slug observability_projects does not configure. NitroIntelligence::Observability::Project.find_by_slug(slug:) reads the same configuration without building a client, and returns nil instead. The gem never raises NitroIntelligence::Observability::ProjectClient::NotFoundError; it is public only because applications raise it themselves, and is to be retired (#182). Rescue Project::NotFoundError for a project that is not configured.

Traces and scores are sent to Cerebro in the background. Fetch each project in an initializer to build its client at boot, and shut them all down as the process exits so that the last ones are sent:

NitroIntelligence.config.observability_projects.each do |project|
  NitroIntelligence.project_client_registry.fetch(project["slug"])
end

at_exit { NitroIntelligence.project_client_registry.shutdown_all }

A server that loads the application and then forks its workers, such as Puma with preload_app! or Unicorn, runs initializers before the fork. Each worker then inherits the project clients but not the background thread that sends their scores, so a worker's scores wait until it shuts down, and are lost if it is killed. In such a server, fetch the projects in each worker after it forks instead, from Puma's on_worker_boot or Unicorn's after_fork, or leave each to be fetched on first use.

Prompts

Message Shape

A chat completion needs a user message in order for the model to answer. What you have to supply depends on the type of the prompt you name, and the two types behave differently.

Text prompts

A text prompt in Cerebro is equivalent to the system prompt. System prompts are used to pre-load initial context into the model needed for following call executions.

Pure text prompts require a user message to be supplied. For example:

#  Wrong - Raises ObservedChatPromptError - A prompt alone, provides a system message alone with nothing to answer
client.chat(parameters: { prompt_name: "Assistant" })

# Correct - The prompt supplies the system message, you supply the user message
client.chat(message: "Why is the sky blue?", parameters: { prompt_name: "Assistant" })
Chat prompts

A chat prompt in Cerebro holds a list of role-tagged messages rather than a single string, so it can carry its own user turn. Define one when the prompt structure should represent a conversation flow vs. a single block of static text, e.g. pre-modeling an interaction you want to continue:

[
  {
    "role": "system",
    "content": "You are a helpful support agent for an online bookstore. Be concise and friendly. Our return window is 30 days."
  },
  {
    "role": "user",
    "content": "Hi, do you ship to Canada?"
  },
  {
    "role": "assistant",
    "content": "Yes! We ship to Canada. Delivery usually takes 5–8 business days."
  },
  {
    "role": "user",
    "content": "{{question}}"
  }
]

That prompt is self-sufficient, so no additional message: is needed:

client.chat(parameters: { prompt_name: "Appointment Extractor", prompt_variables: { document: text } })

Any messages you do pass are appended after the prompt's own system message, so a chat prompt can also serve as a preamble to a live conversation.

A chat prompt containing only a system message has the same problem as a text prompt, and is refused in the same way. If you find yourself writing one, either add a user message to it in Cerebro or make it a text prompt and pass the turn from the caller via the message keyword.

Prompt Variables and Config

Prompts are often created with "variables". These variables can be supplied and compiled into the prompt. For example:

client = NitroIntelligence::Client.new(observability_project_slug: "fake-feature-project")
client.chat(
  message: "Where is the appointment?",
  parameters: {
    prompt_name: "My Prompt With Variables",
    prompt_variables: {
      appointment_id: "1234",
    },
    metadata: {
      source: self.class,
    },
  }
)

Prompts can be created with a "config" JSON object. This config stores structured data such as model parameters (like model name, temperature), function/tool parameters, or JSON schemas.

The prompt config is merged into your chat request by default. The prompt config object is merged into the "parameters" hash, overriding any existing keys.

Consider this prompt config:

{
  "model": "Qwen/Qwen3.6-35B-A3B"
}

Invoking this request would result in "Qwen/Qwen3.6-35B-A3B" being used as the model, even if supplied manually:

client = NitroIntelligence::Client.new(observability_project_slug: "fake-feature-project")
client.chat(
  message: "Where is the appointment?",
  parameters: {
    model: "Qwen/Qwen3.8-27B", # Will not be used, will be overridden by config "Qwen/Qwen3.6-35B-A3B"
    prompt_name: "My Prompt With Variables",
    prompt_variables: {
      appointment_id: "1234",
    },
    metadata: {
      source: self.class,
    },
  }
)

To disable the prompt config entirely, which may be useful for debugging/testing, you can supply the "prompt_config_disabled" keyword. For example:

client = NitroIntelligence::Client.new(observability_project_slug: "fake-feature-project")
client.chat(
  message: "Where is the appointment?",
  parameters: {
    model: "Qwen/Qwen3.8-27B", # This will now be used since "prompt_config_disabled" is true
    prompt_name: "My Prompt With Variables",
    prompt_variables: {
      appointment_id: "1234",
    },
    prompt_config_disabled: true,
    metadata: {
      source: self.class,
    },
  }
)

Reading a Prompt

To read a prompt without making a model call, for its config or to compile its text for a request of your own, fetch it from an observed client:

client = NitroIntelligence::Client.new(observability_project_slug: "page-finder")
prompt = client.prompt("Page Finder Search")      # the production label
prompt = client.prompt("Page Finder Search", label: "staging")
prompt = client.prompt("Page Finder Search", version: 4)

prompt.name    # => "Page Finder Search"
prompt.version # => 4
prompt.type    # => "text", or "chat"
prompt.config  # => { min_relevance: 0.1, candidates: 50 }, keys symbolized
prompt.compile(product: "Nitro") # the text, or the messages, with its variables filled in

Code that holds a project rather than a client can call project.prompt with the same arguments (see Reaching Cerebro Directly). Either uses the same cache as a request naming the prompt: five minutes for a label, indefinitely for a version, and the last prompt fetched for a label when Cerebro cannot be reached. To use one prompt's settings across several steps, fetch it once and keep the result, or pass its version to later lookups, so that a label moved in between cannot mix two versions. A missing prompt raises ObservabilityPromptNotFoundError and a failed lookup ObservabilityPromptError, as below. A client without observability has no project to fetch from, and raises NitroIntelligence::Observability::Project::NotFoundError.

Prompt Errors

A request naming a prompt raises when it cannot use one:

Error Raised when
NitroIntelligence::Observability::PromptStore::ObservabilityPromptNotFoundError The prompt does not exist, nor its fallback when prompt_fallback_name names one
NitroIntelligence::Observability::PromptStore::ObservabilityPromptError Cerebro fails to return the prompt; the message is its response body
NitroIntelligence::Client::Handlers::Observed::ChatHandler::ObservedChatPromptError #chat has no user message for the model to answer (see Message Shape)
NitroIntelligence::Client::Handlers::Observed::AudioTranscriptionHandler::ObservedAudioTranscriptionPromptError #transcribe_audio names a prompt that is not a text prompt
NitroIntelligence::Client::Handlers::Observed::TextToSpeechHandler::ObservedTextToSpeechPromptError #text_to_speech names a prompt that is not a text prompt

On an observed client whose project does not mask content, pass an open File as audio_file to #transcribe_audio. If audio_file has no path, as with a StringIO, the upload to Cerebro raises NitroIntelligence::Audio::AudioFileFormatError. By then, the gateway has already transcribed the audio and billed the request, but your app never receives the transcript. A masked project skips the upload, so the same input works there. Every method raises ArgumentError for a model, or a model setting such as a voice or aspect ratio, that model_config does not list.

Nitro Intelligence Assistants

NitroIntelligence::Assistants is Nitro Intelligence's lightweight SDK for working with hosted agent threads, runs, and human review flows. It is mainly used to initialize conversation threads, trigger agent runs, inspect what a paused run is waiting on (tool calls or a reply held for review), and resume paused threads after human reviews.

For the full guide, see ASSISTANTS.md. For the service this SDK talks to, see the Nitro Intelligence Assistants documentation.

Assistants

An assistant is addressed by the key it is configured under:

assistant = NitroIntelligence.assistants["candidate-concierge"]

result = assistant.run(thread_id: thread.id, messages: messages)
result.reply # => nil while the reply is held for review
assistant.review(thread_id: thread.id, reviewer_id: user.email,
                 interrupt_id: result.review.interrupt_id, reply: { "action" => "approve" })
assistant.thread_state(thread_id: thread.id)

run, review, and the deprecated await_run and review_tool_calls (see Upgrading to 4.0), supply the assistant's own id, so a caller never passes it -- passing assistant_id: to any of them raises ArgumentError rather than quietly sending the call to a different assistant, which is what a migrated caller carrying its old argument would otherwise do. Everything else is thread-scoped and delegates to the client untouched. Reach the client directly with assistant.client if you need it.

Configuration

Connection settings shared by every assistant sit at the top level; definitions holds one entry per assistant, keyed by what you look it up with:

config.assistants_config = {
  # Optional. Defaults to the shared Assistants deployment.
  "base_url" => "https://nip-assistants.example.com",
  "user_id" => "my-app",
  "definitions" => {
    "candidate-concierge" => {},
    "home-studio" => {
      # Overrides the shared value for this assistant only.
      "base_url" => "https://pr306.nip-assistants.example.com"
    }
  }
}

An entry may carry keys this gem has no use for — a graph id, or the Cerebro project an assistant reports to — and they are ignored, so one structure can serve both a deployment and the application reading it. That includes a name of its own: an entry's name is a human-readable label for the assistant's record, distinct from the key it is filed under. Assistant#key returns the latter.

Credentials

Each entry supplies its own api_key and assistant_id:

"definitions" => {
  "candidate-concierge" => {
    "api_key" => "...",
    "assistant_id" => "..."
  }
}

Where a host gets them is its own business — an environment variable its deployment mounts, a secrets store, a literal for local work. This gem reads no environment and assumes no naming convention, so nothing here is tied to one deployment's wiring.

A key that resolves without them raises Assistant::ConfigurationError, naming every field it is missing at once so a host resolving them from elsewhere can see which lookup failed. base_url is not among them: an entry that says nothing about where to reach its assistant gets https://assistants.powerhome.ai, so only a host talking to a different deployment — a review environment, a local server — sets it.

Without definitions

assistants_config lacking definitions is read as keyword arguments for a single Assistants client, and NitroIntelligence.assistants returns that client rather than a registry. This is the shape that predates lookup by key; a host still on it is left alone. base_url is optional here too, and a configuration without one reaches the same shared deployment.

About

The Ruby client for Nitro Intelligence

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages