Skip to content

fix(google): allow generate_reply without a mutable chat context #7193

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Darshak03 wants to merge 1 commit into
base: main
Choose a base branch
from
+82 −16
Open

fix(google): allow generate_reply without a mutable chat context #7193

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 15 additions & 16 deletions livekit-plugins/livekit-plugins-google/livekit/plugins/google/realtime/realtime_api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -815,15 +815,6 @@ def generate_reply(
) -> asyncio.Future[llm.GenerationCreatedEvent]:
if is_given(tools):
logger.warning("per-response tools is not supported by Google Realtime API, ignoring")
if not self._realtime_model.capabilities.mutable_chat_context:
logger.warning(
f"generate_reply is not compatible with '{self._opts.model}' and will be ignored."
)
fut = asyncio.Future[llm.GenerationCreatedEvent]()
fut.set_exception(
llm.RealtimeError(f"generate_reply is not compatible with '{self._opts.model}'")
)
return fut
if self._pending_generation_fut and not self._pending_generation_fut.done():
logger.warning(
"generate_reply called while another generation is pending, cancelling previous."
Expand All @@ -845,13 +836,21 @@ def generate_reply(
)
self._in_user_activity = False

# Gemini requires the last message to end with user's turn
# so we need to add a placeholder user turn in order to trigger a new generation
turns = []
if is_given(instructions):
turns.append(types.Content(parts=[types.Part(text=instructions)], role="model"))
turns.append(types.Content(parts=[types.Part(text=".")], role="user"))
self._send_client_event(types.LiveClientContent(turns=turns, turn_complete=True))
if self._realtime_model.capabilities.mutable_chat_context:
# Gemini requires the last message to end with user's turn
# so we need to add a placeholder user turn in order to trigger a new generation
turns = []
if is_given(instructions):
turns.append(types.Content(parts=[types.Part(text=instructions)], role="model"))
turns.append(types.Content(parts=[types.Part(text=".")], role="user"))
self._send_client_event(types.LiveClientContent(turns=turns, turn_complete=True))
else:
# the session keeps its own history (see history_config in _build_connect_config)
# and rejects client turns, so the placeholder above is not accepted. realtime
# text input starts a generation without appending to the context.
self._send_client_event(
types.LiveClientRealtimeInput(text=instructions if is_given(instructions) else ".")
)
Comment on lines +851 to +853

@devin-ai-integration devin-ai-integration Bot Sep 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Text input disappears before generation

With Gemini 3.1, generate_reply(user_input=...) sends "." because instructions is absent. The immutable context path never forwards the user's text, so the model answers without it.

Prompt for agents 
Gemini 3.1 sessions cannot receive user_input through AgentActivity's existing update_chat_ctx flow. AgentActivity adds user_input to a copied context, but RealtimeSession._sync_chat_ctx intentionally skips non-tool context additions when mutable_chat_context is false. generate_reply then receives no user_input parameter and sends the fallback dot. Preserve the provider-neutral generate_reply(user_input=...) contract by adding an immutable-session input path that forwards the actual user text exactly once before generation. Keep per-turn instructions distinct from user content, and add an end-to-end unit test through AgentSession or AgentActivity rather than testing RealtimeSession.generate_reply alone.
Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@Darshak03 Darshak03 Sep 9, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The gap is real, but it is pre-existing and not in the code this PR touches, and this change strictly improves it.

Two corrections on the mechanism first. generate_reply has no user_input parameter — the abstract signature in llm/realtime.py is instructions, tool_choice, tools, and no plugin extends it. user_input is a private parameter of AgentActivity._realtime_reply_task, and it is never forwarded to generate_reply. The text is delivered a step earlier:

if user_input is not None:
    chat_ctx = self._rt_session.chat_ctx.copy()
    msg = chat_ctx.add_message(role="user", content=user_input)
    await self._rt_session.update_chat_ctx(chat_ctx)   # carries the text
...
generate_reply_fut = self._rt_session.generate_reply(...)  # only triggers

So the text is dropped by _sync_chat_ctx, which builds turns only when mutable_chat_context is true. That is where the gap lives, and it predates this PR.

On main today generate_reply raises RealtimeError for these models, so this path produces no turn at all. After this change the turn happens, with the user text still missing — a broken path becomes a partial one. Audio input is unaffected either way, since _realtime_generation_task sends audio straight to the model.

Worth flagging for whoever picks the remaining gap up: send_realtime_input(text=...) is itself a generation trigger — that is exactly why this PR uses it. Forwarding user text through it inside update_chat_ctx would start a turn, and generate_reply's "." would then start a second one. Closing the gap needs either the plugin holding the unsent user text so generate_reply can send it as the trigger, or letting generate_reply carry user content — neither of which belongs in this change.


def _on_timeout() -> None:
if not fut.done():
Expand Down
67 changes: 67 additions & 0 deletions tests/test_plugin_google_realtime.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -836,3 +836,70 @@ async def _connect(self: AsyncLive, **kwargs: object) -> AsyncIterator[_FakeLive
assert session._unsent_item_ids == set()
finally:
await session.aclose()


async def test_generate_reply_appends_a_turn_when_the_context_is_mutable(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Gemini needs the context to end on a user turn, so a placeholder one triggers the reply."""
async with _make_connected_session(monkeypatch) as session:
assert session._realtime_model.capabilities.mutable_chat_context
fut = session.generate_reply(instructions="say hi")

contents = [m for m in await _drain_sent(session) if isinstance(m, types.LiveClientContent)]
assert len(contents) == 1
assert contents[0].turn_complete is True
assert [(p.text, c.role) for c in contents[0].turns or [] for p in c.parts or []] == [
("say hi", "model"),
(".", "user"),
]
fut.cancel()


async def test_generate_reply_uses_realtime_text_when_the_context_is_immutable(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A session that owns its history rejects client turns; realtime text starts the turn.

Refusing generate_reply outright left these models with no way to open an
agent-initiated turn -- greetings, handoffs and post-tool prompts all go through it.
"""
async with _make_configured_session(
monkeypatch, model="gemini-3.1-flash-live-preview"
) as session:
assert not session._realtime_model.capabilities.mutable_chat_context
session._msg_ch = utils.aio.Chan[ClientEvents]()
session._active_session = object() # type: ignore[assignment]
try:
fut = session.generate_reply(instructions="say hi")
assert not fut.done(), "the reply is no longer refused up front"

sent = await _drain_sent(session)
assert not [m for m in sent if isinstance(m, types.LiveClientContent)]
inputs = [m for m in sent if isinstance(m, types.LiveClientRealtimeInput)]
assert [m.text for m in inputs] == ["say hi"]
fut.cancel()
finally:
session._active_session = None


async def test_generate_reply_without_instructions_nudges_an_immutable_session(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""`instructions` is often absent, so fall back to the same "." nudge the other path uses."""
async with _make_configured_session(
monkeypatch, model="gemini-3.1-flash-live-preview"
) as session:
session._msg_ch = utils.aio.Chan[ClientEvents]()
session._active_session = object() # type: ignore[assignment]
try:
fut = session.generate_reply()
inputs = [
m
for m in await _drain_sent(session)
if isinstance(m, types.LiveClientRealtimeInput)
]
assert [m.text for m in inputs] == ["."]
fut.cancel()
finally:
session._active_session = None