feat(voice): add AssemblyAI streaming STT provider#1912
Open
cjol wants to merge 47 commits into
Open
Conversation
Spec for @cloudflare/voice-assemblyai — a type-compliant Streaming v3 STT provider for the Agents voice pipeline, mirroring the Deepgram/Telnyx example. Captures scope (STT-only, baseUrl-configurable for AI Gateway), the v3 protocol mapping, test plan, and deferred work (typed gateway helper, Workers AI self-host, on-prem pharma).
Apply design-review simplifications: - options down to apiKey/model/formatTurns/medical/keyterms/baseUrl/params - add `params` passthrough; drop endOfTurnConfidenceThreshold - hardcode encoding=pcm_s16le and sample_rate=16000 (pipeline-fixed) - drop `region` (use baseUrl for EU/gateway); document URLs in README - cut `prompt` from v1; close() no longer awaits Termination - single test file; add Known Limitations (no onError, model-determined language)
…ence
Verified streaming params against AssemblyAI's live API docs (via MCP):
- lock model to u3-rt-pro (the voice-agent model); drop the `model` option
- auth via Authorization header (raw key, no prefix), NOT a ?token= query param
- drop format_turns (u3-rt-pro finals are always formatted)
- fully typed surface, no passthrough: domain, keyterms, minTurnSilence,
maxTurnSilence, interruptionDelay, continuousPartials, baseUrl
- rename medical:boolean -> domain:"medical-v1"|(string&{})
- hardcode speech_model/sample_rate/encoding; conditional params only when set
- limitations: no onError, language model-determined, single model / 6 languages
Verified against AssemblyAI prompting + streaming API docs: - prompt is a connection param (no UpdateConfiguration round-trip needed); it is the u3-rt-pro differentiator and the language-guidance lever. Doc the "omit for the optimized default prompt" recommendation. - vadThreshold (vad_threshold): VAD silence-confidence threshold for noisy envs. - both appended only when set; language limitation softened (guided via prompt).
language_detection applies to u3-rt-pro (native multilingual code-switching) and returns language_code/language_confidence on Turn events. Since the Cloudflare transcript callbacks are text-only, pair the flag with a provider-specific onLanguageDetected(languageCode, languageConfidence) callback to surface it without changing the shared interface.
- Add a `speechModel` option (defaults to u3-rt-pro) with the `AssemblyAISpeechModel` type and model-aware validation that rejects u3-rt-pro-only params (prompt, continuousPartials, interruptionDelay) on universal-streaming models. - Default min/max turn silence to 400/1280 ms (above the server's 100/1000) so speakers can pause mid-thought without the turn being cut off; callers can still override. - Default continuous_partials on for u3-rt-pro for a live mid-turn transcript. - Log unexpected WebSocket closes (auth/session errors only arrive as close frames) and rewrite wss:// to https:// for Cloudflare's fetch-based upgrade.
- Streamline the intro, usage, options table, and how-it-works sections; document the new `speechModel` option and the 400/1280 ms turn-silence defaults. - Add an "AssemblyAI documentation" section linking the relevant streaming pages (Universal-3 Pro, turn detection & partials, message sequence, streaming API, endpoints & data zones, Medical Mode, keyterms) and a dashboard link for the API key.
Real-time voice agent running entirely in a Durable Object: AssemblyAI u3-rt-pro streaming STT, Workers AI LLM (glm-4.7-flash) with tools, and Workers AI MeloTTS for keyless TTS. Browser mic streams 16 kHz PCM over a plain WebSocket via the useVoiceAgent React hook, with barge-in, streaming responses, and conversation memory. Only ASSEMBLYAI_API_KEY is required.
Update the AssemblyAI streaming provider to the current model and API. Plugin (voice-providers/assemblyai): - Lock speech_model to universal-3-5-pro; drop the speechModel option and the old model-compat assertions (universal-3-5-pro is is_u3_pro server-side, so it supports prompt/agent_context/mode/etc). - Add connection params: mode, agentContext, previousContextNTurns, languageCode, voiceFocus, voiceFocusThreshold, validated against the server's ranges (1500-char caps, threshold requires voiceFocus, interruptionDelay 0-1000, previousContextNTurns 0-100). - Defer turn-silence / partials to mode instead of force-defaulting them. - Add session.updateAgentContext(text): sends an UpdateConfiguration with agent_context mid-session (pre-connect buffering, latest-only, 1500 cap). Framework (@cloudflare/voice): - Add optional TranscriberSession.updateAgentContext?(text) (no-op for providers without context carryover), a forwarder on AudioConnectionManager, and pipeline calls so withVoice feeds each spoken reply/greeting back to the transcriber as conversational context. Also update the example, provider/voice READMEs, and voice docs.
When previousContextNTurns is 0, the server discards agent_context (the carryover window is zero), so AssemblyAISession.updateAgentContext() now early-returns instead of sending an UpdateConfiguration the server would throw away. Contained to the plugin — the framework seam still calls updateAgentContext() unconditionally and the provider decides to no-op.
Rename prose references from "Universal-3.5 Pro Streaming" to the official "Universal 3.5 Pro Realtime" across the provider, example, READMEs, and voice docs. The wire value (speech_model=universal-3-5-pro) is unchanged.
- prompt and keyterms are NOT mutually exclusive on streaming u3.5-pro (no server validator enforces it) — correct the docs and the prompt JSDoc. - Raise MAX_PROMPT_CHARS 1500 -> 1750 to match the current API (connection_parameters.py); applies to prompt, agentContext, and the updateAgentContext() truncation. - Add descriptors for the Prompting/Keyterms, Conversation Context, and Voice Focus documentation links.
The "Endpoints & data zones" link pointed at the streaming API parameter reference; repoint it to /docs/streaming/endpoints-and-data-zones, which actually lists the regional WebSocket hosts the descriptor describes.
AssemblyAI requires 50-1000ms of audio per binary WebSocket message and terminates the session (3007) on the second undersized message. Browser mic chunks are 100ms and pass through untouched, but the Twilio/Telnyx adapters relay 20ms frames, which killed the session almost immediately. Accumulate sub-minimum frames until they cross 1600 bytes (50ms at 16kHz mono s16le), matching AssemblyAI's own LiveKit adapter. A sub-minimum tail is dropped at close().
The example pinned versions that had since moved on main, tripping sherif's multiple-dependency-versions check in CI. Resolved versions were already deduped to the workspace-highest, so only specifiers change.
The implementation plan and design spec were working documents for building the provider; docs/superpowers/ does not exist upstream and they fail oxfmt's format check. They remain in branch history.
…ction The server rejects out-of-range values by closing the session, which surfaces only as a logged close frame mid-call. Fail fast with a clear config error instead, consistent with the other validated options.
…ailures The server sends an Error message before closing the socket; previously only the close code was logged. Also include the HTTP status and body when the fetch upgrade returns no WebSocket (typically an auth failure).
Say 'server default' for previousContextNTurns instead of a number the docs don't pin down, source the keep-context-concise guidance from the Conversation Context page, and cite session-based billing rather than asserting the mechanics.
…g LLM errors
onCallStart queried cf_voice_messages with raw SQL, which throws 'no such
table' on a brand-new agent instance (the voice mixin creates the table
lazily) and silently killed the greeting. Use getConversationHistory(),
which ensures the schema. Also add a streamText onError logger — the AI
SDK otherwise swallows LLM failures, surfacing them only as an empty
reply ('No response generated').
The voice pipeline warns that textStream can join non-adjacent text parts incorrectly (visible as doubled tokens); the other voice examples already return result.fullStream.
glm-4.7-flash was intermittently hanging ~60s and returning 3043/504 from the inference upstream, leaving the agent silent. gpt-oss-20b (one of the voice-agent example's endorsed models) streams in ~1s and handles the tool-calling demo reliably.
…adapter MeloTTS was intermittently failing with 3043 inference errors, leaving the agent silent after transcription succeeded. The framework's default WorkersAITTS (@cf/deepgram/aura-1) is the same path the other voice examples use, and dropping the custom adapter simplifies the example.
…reply gpt-oss-20b emits reasoning tokens before text; with Workers AI's ~256 default cap, longer reasoning consumed the whole budget and the turn ended with finishReason=length and zero speakable text — surfacing as a silent 'No response generated'. Cap at 2048 and log turn transcripts and LLM finish stats so this failure mode is visible in the terminal.
Tested against the live stack: gpt-oss-20b works but its reasoning phase delays the first spoken token and can starve short caps; glm-4.7-flash was intermittently erroring upstream; llama-3.3-70b leaks tool calls as literal JSON text through workers-ai-provider. llama-4-scout executes tools correctly, emits no reasoning phase, and keeps replies concise.
…demo Replace the generic time/weather assistant with Luna Rossa, a restaurant reservation agent — a real use-case shaped around what the integration demonstrates: terse answers after agent questions (agent_context carryover), venue vocabulary (prompt + keyterms), reservations persisted in the DO's SQLite across calls, and availability/booking/lookup/cancel tools. The LLM is OpenAI gpt-4.1-mini via the AI SDK: every Workers AI model tested had a blocking defect in this multi-tool voice loop (glm-4.7-flash upstream errors, llama tool-call JSON leaks and hallucinated confirmation codes, gpt-oss reasoning-only empty replies, mistral token doubling). Verified end-to-end: 4-turn spoken booking conversation with real tool execution and a database-backed confirmation code.
Add an 'Under the hood' panel to the Luna Rossa client that renders debug_event messages broadcast by the agent: the exact agent_context values sent to AssemblyAI (observed by wrapping the transcriber's updateAgentContext), and every tool call and result from onStepFinish. Makes the integration's invisible machinery — context carryover and database-backed tools — visible during a live call.
Replace WorkersAITTS (aura-1) with a small in-example CartesiaTTS adapter — a natural, low-latency voice and a demonstration of bringing your own TTS vendor to the pipeline via the TTSProvider interface. Requests are serialized through a queue with one retry on 429, since Cartesia plans cap concurrent requests and the pipeline synthesizes sentences in parallel (an unqueued burst silently dropped a sentence mid-reply). With the LLM on OpenAI and TTS on Cartesia, nothing uses the AI binding anymore — drop it from wrangler.jsonc, so local dev needs no Cloudflare login, just the three vendor keys.
…tions Per-sentence synthesis over the REST endpoint made each sentence an independent generation, so the voice's pitch and pacing jumped audibly at sentence seams — Cartesia documents this exact failure mode and its fix: stream sentences of one reply into a shared WebSocket context (continuations), so the model extends one generation instead of restarting. The pipeline starts sentence syntheses eagerly, so the reply's first sentence opens the context and yields all of its audio while later sentences join it and yield nothing; a short idle timer finalizes the context and barge-in cancels it. WS audio is raw PCM, wrapped per-chunk in WAV headers for the client's decodeAudioData path (which also decodes exactly, avoiding MP3 padding at seams). One-shot utterances (the greeting) keep the simpler REST/MP3 path.
…pter Keep the example focused on the AssemblyAI integration: the built-in Workers AI TTS (aura-1) needs no extra vendor key or adapter code. The Cartesia WebSocket-continuations adapter lives in branch history (bb9cd02) as a reference for a future @cloudflare/voice-cartesia provider. MeloTTS was evaluated as the non-Deepgram alternative but currently fails ~half of requests upstream (3043).
Two robustness gaps from the final review: (1) if the socket never connects (e.g. a bad API key), feed() buffered mic audio unboundedly in the Durable Object — now capped at ~30s with a single logged warning; (2) WebSocket.send() throws in the gap between the server closing the socket and the close event firing — sends now route through a guarded helper instead of letting the exception escape into the audio pipeline.
… audio synthesize() read response.arrayBuffer() without checking response.ok, so an error body (e.g. the Workers AI free-tier 429 quota JSON) was sent to the client as audio bytes and failed to decode silently. Log the status and body and return null so the pipeline skips the sentence.
Identify traffic from this provider with AssemblyAI's structured user-agent format (AssemblyAI/1.0 (integration=Cloudflare-Agents)), matching how the official SDKs and other integrations attribute usage.
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| "$schema": "../../node_modules/wrangler/config-schema.json", | ||
| "name": "assemblyai-voice-agent", | ||
| "ai": { "binding": "AI", "remote": true }, | ||
| "compatibility_date": "2026-01-28", |
Contributor
There was a problem hiding this comment.
🟡 Example uses an outdated compatibility date, breaking the repo-wide convention
The example's Workers configuration uses an older compatibility date ("2026-01-28" at examples/assemblyai-voice-agent/wrangler.jsonc:5) instead of the repo-mandated "2026-06-11", so the example may miss runtime features or behavior changes that every other example relies on.
Impact: The example runs on an older Workers compatibility surface than all other examples in the repo.
AGENTS.md rule and repo-wide convention
The root AGENTS.md states: "All wrangler configs use compatibility_date: "2026-06-11" and compatibility_flags: ["nodejs_compat"]". The examples/AGENTS.md repeats: "compatibility_date: "2026-06-11", compatibility_flags: ["nodejs_compat"]". Every other example in the repo uses "2026-06-11" — this is the only one that deviates.
Suggested change
| "compatibility_date": "2026-01-28", | |
| "compatibility_date": "2026-06-11", |
Was this helpful? React with 👍 or 👎 to provide feedback.
| @@ -0,0 +1,26 @@ | |||
| { | |||
| "$schema": "../../node_modules/wrangler/config-schema.json", | |||
Contributor
There was a problem hiding this comment.
🟡 Schema path points to monorepo root instead of local node_modules, breaking standalone use
The wrangler config schema path uses a monorepo-root-relative path ("../../node_modules/wrangler/config-schema.json" at examples/assemblyai-voice-agent/wrangler.jsonc:2) instead of the mandated local path, so the schema reference breaks when the example is copied out and installed standalone.
Impact: IDE schema validation fails when the example is used outside the monorepo.
AGENTS.md rule and repo-wide convention
The examples/AGENTS.md states: 'Include "$schema": "./node_modules/wrangler/config-schema.json" — relative to the example root so it resolves both in the workspace and when the example is copied out and installed standalone'. Every other example uses "./node_modules/wrangler/config-schema.json".
Suggested change
| "$schema": "../../node_modules/wrangler/config-schema.json", | |
| "$schema": "./node_modules/wrangler/config-schema.json", |
Was this helpful? React with 👍 or 👎 to provide feedback.
| class names. Both paths are needed: the deep path resolves in the monorepo | ||
| (hoisted to root node_modules), the shallow one works standalone. */ | ||
| @source "../node_modules/@cloudflare/kumo/dist/**/*.{js,jsx,ts,tsx}"; | ||
| @source "../../../node_modules/@cloudflare/kumo/dist/**/*.{js,jsx,ts,tsx}"; |
Contributor
There was a problem hiding this comment.
🟡 CSS source path uses a monorepo-root-relative path that breaks standalone use
The stylesheet includes a monorepo-root-relative Kumo source path (@source "../../../node_modules/@cloudflare/kumo/..." at examples/assemblyai-voice-agent/src/styles.css:5) that the repo rules explicitly forbid, so Tailwind class extraction breaks when the example is copied out and installed standalone.
Impact: Kumo component styles are missing when the example is used outside the monorepo.
AGENTS.md rule and repo-wide convention
The examples/AGENTS.md states: "The @source path is relative to src/styles.css, so it points at the example's own node_modules (../node_modules). That resolves both in the pnpm workspace (each package gets its own node_modules with symlinks) and when the example is copied out and installed standalone — don't use a monorepo-root-relative path like ../../../node_modules." No other example in the repo uses this pattern.
Suggested change
| @source "../../../node_modules/@cloudflare/kumo/dist/**/*.{js,jsx,ts,tsx}"; |
Was this helpful? React with 👍 or 👎 to provide feedback.
stof
reviewed
Jul 10, 2026
| "url": "https://github.com/cloudflare/agents/issues" | ||
| }, | ||
| "peerDependencies": { | ||
| "@cloudflare/voice": "*" |
There was a problem hiding this comment.
I think this should define a proper constraint.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds
@cloudflare/voice-assemblyai, a streaming speech-to-text provider for@cloudflare/voice, based on the contributor branch linked in #1900. Closes #1900.Why
@cloudflare/voicealready supports pluggable STT providers, but users who want AssemblyAI streaming transcription need to write their own adapter.TranscriberSessionrather than coupling it to a specific provider. Existing providers can ignore it.Public API Surface
AssemblyAISTT@cloudflare/voice-assemblyai.AssemblyAISTTOptionsTranscriberSession.updateAgentContextCode Changes
voice-providers/assemblyai, following the existing voice provider package structure.onInterim,onUtterance, andonSpeechStartcallbacks.withVoice.examples/assemblyai-voice-agentreservation-agent demo.