feat(core): add user transcription timeout

[chenghao-mou]

• add a new event when vad detects any speech, but stt fails to produce any transcript during that timeout

from livekit.agents import AgentSession, UserTranscriptionTimeoutEvent

  session = AgentSession(
      stt=...,
      llm=...,
      tts=...,
      # VAD heard speech but no transcript landed within 5s of the user stopping.
      # 5.0 is the default; set to None to disable.
      transcription_timeout=5.0,
  )

  @session.on("user_transcription_timeout")
  def _on_transcription_timeout(ev: UserTranscriptionTimeoutEvent) -> None:
      # ev.speech_duration   -> total VAD speech (s) this turn that produced no transcript
      # ev.vad_speech_started_at -> when the user first started speaking (epoch seconds)

      # ignore very short blips (coughs, door slams) that VAD picks up as "speech"
      if ev.speech_duration < 0.7:
          return

      session.generate_reply(
          instructions="Tell the user you didn't catch that and ask them to repeat it.",
      )

[devin-ai-integration]

Devin Review found 2 potential issues.

[devin-ai-integration]

🚩 New event not forwarded in SessionHost remote transport

The SessionHost in remote_session.py registers handlers for specific events (lines 371-379) and forwards them over the transport. The new user_transcription_timeout event is not registered or forwarded. This means remote sessions won't receive this event. This is likely acceptable as a first iteration (not all events need remote transport support immediately), but it's an inconsistency with the event being in EventTypes and AgentEvent.

(Refers to lines 366-379)

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🚩 Transcription timeout suppressed during agent speech with no re-arm

When _on_transcription_timeout fires while the agent is speaking (audio_recognition.py:1809), the callback returns silently without emitting the event and without scheduling a retry. This means if VAD detects user speech that produces no transcript, and the timeout happens to fire while the agent is still speaking (e.g. the user spoke over the agent near the end of its turn), the timeout event is permanently lost for that speech burst. The rationale is likely that speech detected during agent output is often echo/noise, but with AEC enabled it could be genuine user speech. Whether this is acceptable depends on the use case.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[chenghao-mou]

Since there is no user content, we should still emit the signal so the agent can check.

[longcw]

maybe skip this when stt is not set?

[davidzhao]

should we have a default handler with the above implementation? any downsides of having this work automatically?

[longcw]

should we have a default handler with the above implementation? any downsides of having this work automatically?

I am wondering how many false alarms it may have, it can by risky if there are some background noise.