Add Synthesia interactive-avatar plugin by swarnimkulkarni-synth · Pull Request #7216 · livekit/agents

swarnimkulkarni-synth · 2026-09-10T20:09:06Z

Summary

Adds livekit-plugins-synthesia, a plugin that attaches a Synthesia interactive avatar to a LiveKit voice agent:

synthesia.AvatarSession extends livekit.agents.voice.avatar.AvatarSession. Call start() before AgentSession.start() to dispatch the hosted avatar worker into the room and wire the agent's speech to it over a data stream.
swap_avatar() switches between up to five precomputed avatars mid-session.
Errors surface as a single SynthesiaError (an APIError) with a type: ErrorType field identifying the failure (auth, unknown avatar, quota, rate limit, concurrency limit, timeout, connection, etc.), plus retryable, retry_after, status, and request_id.
Lifecycle events (session end, unexpected avatar drop) are logged rather than emitted, since no other avatar plugin exposes a custom event API.

What's included

livekit-plugins/livekit-plugins-synthesia/ — the plugin package
Registered as a uv workspace member (pyproject.toml) and as a livekit-agents[synthesia] extra (livekit-agents/pyproject.toml), with uv.lock regenerated accordingly
tests/test_plugin_synthesia.py — a single combined unit-test module (plugin registration, config validation, error taxonomy, the HTTP client, the avatar session lifecycle, and a README-mirroring usage example), tagged pytest.mark.unit and pytest.mark.plugin("synthesia") per the existing test-category convention

Test plan

ruff check / ruff format --check clean
mypy (uv run mypy -p livekit.plugins.synthesia) clean under the repo's strict config
pytest tests/test_plugin_synthesia.py — 165 passed
Confirmed via pytest --list-categories that the new test module is picked up under both unit and plugin categories, so it runs in the standard make unit-tests CI gate
uv sync --all-extras --dev succeeds with the new package in the workspace

Adds livekit-plugins-synthesia: attach a Synthesia interactive avatar to a LiveKit voice agent, mirroring the bey/tavus/hedra avatar plugins. Registers the package as a uv workspace member and as a livekit-agents[synthesia] extra, and adds a combined unit-test module under tests/, following the plugin(name) test-category convention.

CLAassistant · 2026-09-10T20:09:28Z

All committers have signed the CLA.

devin-ai-integration

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 8 potential issues.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

devin-ai-integration · 2026-09-10T20:13:10Z

transport

+        audio = agent_session.output.audio
+        if isinstance(audio, DataStreamAudioOutput):
+            agent_session.output.audio = None
+            if hasattr(audio, "aclose"):
+                await audio.aclose()


🔴 Join timeout leaks audio transport

When wait_for_join times out, _discard_partial_start cannot close the real DataStreamAudioOutput. The transport has no aclose, leaving its participant-wait task and room listener alive.

Learn more

A connected room starts DataStreamAudioOutput._start_task immediately. That task waits indefinitely for the avatar participant and video track. If Synthesia never joins, wait_for_join times out and this cleanup detaches the transport, but the production transport has no aclose method. Its task and connection_state_changed listener therefore retain the room and transport after start() returns an error.

Example: Synthesia accepts a session request but its worker never joins. After 30 seconds, AvatarSession.start() raises TIMEOUT, while the transport still waits for that worker. Repeating the start creates another waiting task and listener.

Recommended fix: Add a real lifecycle close operation to DataStreamAudioOutput that unregisters its room listener, cancels _start_atask, closes any stream writer, cancels tracked tasks and timers, and removes registered RPC handlers. Store the created transport explicitly and always invoke that close operation during partial-start cleanup.

Was this helpful? React with 👍 or 👎 to provide feedback.

DataStreamAudioOutput has no aclose() at all today — confirmed in the framework source, which carries a maintainer TODO acknowledging exactly this ("this class should be encapsuled somewhere else to allow for a clean close"). Let me know if you feel differently about this.

devin-ai-integration · 2026-09-10T20:13:11Z

+            agent_session = self._agent_session
+            if agent_session is not None and isinstance(
+                agent_session.output.audio, DataStreamAudioOutput
+            ):
+                audio = agent_session.output.audio
+                agent_session.output.audio = None
+                if hasattr(audio, "aclose"):
+                    await audio.aclose()


🟡 Closed avatar sink remains installed

When aclose() runs, agent_session.output keeps the Synthesia sink installed. Clearing _audio_output leaves later agent speech routed to a closed or departed avatar.

Learn more

The avatar sink is part of the agent's active audio chain. Closing that sink does not detach it from the chain, and clearing _audio_output only removes the plugin's ownership reference. This occurs after normal close, an unexpected avatar departure, and startup failures after sink installation. The same stale reference remains when wrappers hold the sink through an audio proxy.

Example: An avatar joins, then disconnects while the AgentSession remains active. Teardown closes the Synthesia sink, but the next TTS response still traverses the existing output chain into that sink instead of using no output or the prior output.

Recommended fix: Retain enough chain state to detach or replace the exact Synthesia tail during both aclose() and _discard_partial_start(). Restore the displaced tail when possible; otherwise use a framework API that removes the tail while preserving wrappers. Add coverage that sends audio after teardown and verifies it cannot reach the Synthesia sink.

Was this helpful? React with 👍 or 👎 to provide feedback.

Same issue as above for close. We might need a framework level close method for this.

devin-ai-integration · 2026-09-10T20:13:14Z

+        except Exception as e:
+            raise SynthesiaError(f"avatar swap RPC failed: {e}", type=ErrorType.CONNECTION) from e


🟨 Avatar RPC failures expose provider data

A failed swap copies str(e) into SynthesiaError and preserves the original cause. Downstream logging can expose RPC payloads, identities, or request metadata.

Was this helpful? React with 👍 or 👎 to provide feedback.

body is a documented field on the base APIError class itself ("The API response body, if available"). This seems like an established repo-wide pattern. Happy to change if you feel strongly.

devin-ai-integration · 2026-09-10T20:13:15Z

+        avatar_id_result = response.get("avatar_id") if isinstance(response, dict) else None
+        if (
+            not isinstance(response, dict)
+            or response.get("error")
+            or not isinstance(avatar_id_result, str)
+        ):
+            detail = response.get("error") if isinstance(response, dict) else None
+            raise SynthesiaError(f"avatar swap failed: {detail or raw}")


🟨 Swap responses leak through exceptions

A rejected swap copies detail or the complete provider response into SynthesiaError. Automatic exception logging can expose customer-specific response content.

Was this helpful? React with 👍 or 👎 to provide feedback.

Same as above — body is a documented APIError field, and every other plugin attaches raw provider responses to it the same way. Not specific to this plugin's swap_avatar path. Again happy to change if you feel strongly.

devin-ai-integration · 2026-09-10T20:13:16Z

+        message = _body_message(body) or (
+            f"Synthesia request failed ({code or f'HTTP {resp.status}'})"
+        )
+        request_id = _body_request_id(body)
+        retry_after = (
+            _parse_retry_after(resp, body)
+            if error_type in (ErrorType.RATE_LIMITED, ErrorType.CONCURRENCY_LIMIT)
+            else None
+        )
+        return SynthesiaError(
+            message,
+            type=error_type,
+            retry_after=retry_after,
+            body=body,
+            status=resp.status,
+            request_id=request_id,


🟨 HTTP errors expose provider responses

HTTP failures copy backend details and the complete response into SynthesiaError. Automatic exception logging can expose echoed tokens or customer diagnostics.

Was this helpful? React with 👍 or 👎 to provide feedback.

Same as the other body= findings — this is the documented APIError.body contract, used identically across every plugin in the repo (deepgram, cartesia, etc.), not something unique to this HTTP client.

devin-ai-integration · 2026-09-10T20:13:17Z

+        raise SynthesiaError(
+            _exhausted_message(conn_options.max_retry + 1, last_status, last_body),
+            type=ErrorType.CONNECTION,
+            body=last_body,
+            status=last_status,
+            request_id=_body_request_id(last_body),
+        ) from last_exc


🟨 Retry failures expose request diagnostics

Retry exhaustion preserves response details, the complete body, and the transport cause. Exception logging can expose headers, URLs, or echoed request data.

Was this helpful? React with 👍 or 👎 to provide feedback.

Same as the other body= findings — retry exhaustion attaching the last response body/status/request_id to APIError.body matches the documented, repo-wide contract rather than being a plugin-specific issue.

…tion Match bey/tavus and use replace_audio_tail() to install the avatar's audio output, so wrappers AgentSession.start() adds (TranscriptSynchronizer, RecorderAudioOutput) survive. Teardown now closes the exact sink this session created, tracked in self._audio_output, rather than inferring ownership from isinstance(agent_session.output.audio, DataStreamAudioOutput). The old check broke once the sink was wrapped, and could not tell a caller's pre-existing DataStreamAudioOutput from this session's own.

…mple URL

swarnimkulkarni-synth · 2026-09-10T23:30:40Z

Verified against production Synthesia + LiveKit infrastructure

Ran an internal smoke test that drives the exact public path an integrator would use: construct AgentSession, attach synthesia.AvatarSession, call avatar.start(...) against a real LiveKit room, and confirm the hosted avatar worker joins and publishes video.

SMOKE TEST SUMMARY: 1/1 joined successfully
[PASS] joined + video published

Confirms auth, worker dispatch, room join, and video publish all work end-to-end against production, using the code currently in this PR.

…hesia/avatar.py Co-authored-by: Tina Nguyen <72938484+tinalenguyen@users.noreply.github.com>

get_job_context was referenced but never imported, and JobContext has no local_participant attribute — only local_participant_identity, which bey/tavus already use for the same reason: it doesn't depend on the room having finished connecting. Tests now fake the job context via an autouse fixture, since get_job_context() raises outside a real job entrypoint.

tinalenguyen

tested it and it works well for me, added some small docs comments!

tinalenguyen · 2026-09-11T20:30:02Z

+Documentation = "https://docs.synthesia.io"
+Website = "https://www.synthesia.io/"
+Source = "https://github.com/livekit/agents"


Suggested change

Documentation = "https://docs.synthesia.io"

Website = "https://www.synthesia.io/"

Source = "https://github.com/livekit/agents"

Documentation = "https://docs.livekit.io"

Website = "https://livekit.io/"

Source = "https://github.com/livekit/agents"

tinalenguyen · 2026-09-11T20:39:26Z

@@ -0,0 +1,56 @@
+# Synthesia plugin for LiveKit Agents
+
+Attach a Synthesia interactive avatar to a LiveKit voice agent. The avatar joins


Suggested change

Attach a Synthesia interactive avatar to a LiveKit voice agent. The avatar joins

Attach a [Synthesia](https://www.synthesia.io/) interactive avatar to a LiveKit voice agent. The avatar joins

tinalenguyen · 2026-09-11T20:40:11Z

+Attach a Synthesia interactive avatar to a LiveKit voice agent. The avatar joins
+the room and lip-syncs the agent's speech in real time.
+
+See the [Synthesia integration docs](https://docs.synthesia.io/reference/interactive-avatars) for more information.


Suggested change

See the [Synthesia integration docs](https://docs.synthesia.io/reference/interactive-avatars) for more information.

See the [Synthesia integration docs](https://docs.livekit.io/agents/models/avatar/plugins/synthesia/) for more information.

…ekit.io _mint_token now calls get_job_context(required=False) and falls back to room.local_participant.identity when there's no job context, so callers who construct and connect an rtc.Room directly (the base AvatarSession contract's supported standalone mode) can still start. Also point pyproject.toml's Documentation/Website and the README's docs link at docs.livekit.io, matching bey/tavus, and link the Synthesia mention in the README to synthesia.io.

devin-ai-integration

Devin Review found 1 new potential issue.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

devin-ai-integration · 2026-09-11T22:56:46Z

+        job_ctx = get_job_context(required=False)
+        agent_identity = (
+            job_ctx.local_participant_identity
+            if job_ctx is not None
+            else room.local_participant.identity
+        )


🟡 Disconnected standalone startup leaks SDK error

Without a job context, start() reads room.local_participant even when the room is disconnected. The RTC RuntimeError escapes instead of the plugin’s SynthesiaError.

Learn more

A LiveKit room creates its local participant during connection. The base avatar session accepts disconnected rooms because it can wait for a later connection, but token minting needs the agent identity immediately. Without a job context, that identity is only available from an already connected room. Accessing it earlier raises the RTC SDK's RuntimeError, and the broad startup cleanup handler re-raises that exception unchanged.

Example: A standalone script constructs rtc.Room() and calls avatar.start(session, room) before room.connect(...). Startup raises an RTC RuntimeError; callers expecting SynthesiaError cannot handle it through the plugin's error API.

Recommended fix: Check room.isconnected() before reading room.local_participant.identity. Raise SynthesiaError with the existing connection guidance when no job context and no connected room are available.

Suggested change

job_ctx = get_job_context(required=False)

agent_identity = (

job_ctx.local_participant_identity

if job_ctx is not None

else room.local_participant.identity

)

job_ctx = get_job_context(required=False)

if job_ctx is not None:

agent_identity = job_ctx.local_participant_identity

elif room.isconnected():

agent_identity = room.local_participant.identity

else:

raise SynthesiaError(

"connect the room before starting the avatar session outside a job context"

)

Was this helpful? React with 👍 or 👎 to provide feedback.

swarnimkulkarni-synth requested a review from a team as a code owner September 10, 2026 20:09

devin-ai-integration Bot reviewed Sep 10, 2026

View reviewed changes

swarnimkulkarni-synth added 3 commits September 10, 2026 14:09

Replace a real internal Synthesia domain in a test with a generic exa…

11a96b6

…mple URL

Point the docs link at the interactive-avatars reference page

47e37a8

tinalenguyen reviewed Sep 11, 2026

View reviewed changes

Comment thread livekit-plugins/livekit-plugins-synthesia/livekit/plugins/synthesia/avatar.py Outdated

Update livekit-plugins/livekit-plugins-synthesia/livekit/plugins/synt…

7309cb2

…hesia/avatar.py Co-authored-by: Tina Nguyen <72938484+tinalenguyen@users.noreply.github.com>

This comment was marked as resolved.

Sign in to view

tinalenguyen approved these changes Sep 11, 2026

View reviewed changes

devin-ai-integration Bot reviewed Sep 11, 2026

View reviewed changes

tinalenguyen merged commit d782aa9 into livekit:main Sep 11, 2026
16 checks passed

rosetta-livekit-bot Bot mentioned this pull request Sep 12, 2026

feat(synthesia): add interactive avatar plugin livekit/agents-js#2486

Open

5 tasks

		except Exception as e:
		raise SynthesiaError(f"avatar swap RPC failed: {e}", type=ErrorType.CONNECTION) from e

		@@ -0,0 +1,56 @@
		# Synthesia plugin for LiveKit Agents

		Attach a Synthesia interactive avatar to a LiveKit voice agent. The avatar joins

	Attach a Synthesia interactive avatar to a LiveKit voice agent. The avatar joins
	Attach a [Synthesia](https://www.synthesia.io/) interactive avatar to a LiveKit voice agent. The avatar joins

	See the [Synthesia integration docs](https://docs.synthesia.io/reference/interactive-avatars) for more information.
	See the [Synthesia integration docs](https://docs.livekit.io/agents/models/avatar/plugins/synthesia/) for more information.

Uh oh!

Conversation

swarnimkulkarni-synth commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's included

Test plan

Uh oh!

CLAassistant commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

swarnimkulkarni-synth Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

swarnimkulkarni-synth Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

devin-ai-integration Bot Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

swarnimkulkarni-synth Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

swarnimkulkarni-synth Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

swarnimkulkarni-synth Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

swarnimkulkarni-synth Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

swarnimkulkarni-synth commented Sep 10, 2026

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

tinalenguyen left a comment

Choose a reason for hiding this comment

Uh oh!

tinalenguyen Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

tinalenguyen Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

tinalenguyen Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

swarnimkulkarni-synth commented Sep 10, 2026 •

edited

Loading

CLAassistant commented Sep 10, 2026 •

edited

Loading

devin-ai-integration Bot left a comment •

edited

Loading

devin-ai-integration Bot Sep 10, 2026 •

edited

Loading

swarnimkulkarni-synth Sep 10, 2026 •

edited

Loading

devin-ai-integration Bot Sep 10, 2026 •

edited

Loading

swarnimkulkarni-synth Sep 10, 2026 •

edited

Loading

devin-ai-integration Bot Sep 10, 2026 •

edited

Loading

swarnimkulkarni-synth Sep 10, 2026 •

edited

Loading

devin-ai-integration Bot Sep 10, 2026 •

edited

Loading

swarnimkulkarni-synth Sep 10, 2026 •

edited

Loading