PREFLIGHT_TRANSCRIPT/INTERIM_TRANSCRIPT should clear false interruption timer

[superMDguy]

Bug Description

Currently, the false-interruption timer doesn't get cleared until a turn is committed, which can often longer than the false interruption timeout in some scenarios.

Expected Behavior

A PREFLIGHT_TRANSCRIPT/INTERIM_TRANSCRIPT event should clear the false interruption timer.

Reproduction Steps

Interrupt the AI, stop in the middle of a sentence, and pause.
This causes the interim transcript to get totally lost, and the AI resumes.

Operating System

macOS Tahoe

Models Used

Deepgram FLux

Package Versions

livekit-agents==1.7.1

[devtechedge]

Working on a focused fix so PREFLIGHT_TRANSCRIPT / INTERIM_TRANSCRIPT clear the false-interruption timer (same as a committed turn), plus a regression test. Will open a PR linking this issue.

[longcw]

are you using turn_detection="stt", could you share a reproducible example?

it's intended to not use INTERIM_TRANSCRIPT to clear the timer since usually a noise (false interruption) can generate an interim transcript. If the VAD enabled, and turn detection is the default, the VAD should pause the timer when user is speaking.

[devtechedge]

Hi @longcw, thanks for the context.

To answer the repro question: I hit this with Deepgram FLux as the STT (same as reported by @superMDguy), with VAD enabled and turn detection left at the default.

The sequence is: the user interrupts the agent mid-sentence, VAD fires end_of_speech during the pause, and the false-interruption resume timer starts.

STT then keeps producing interim/preflight transcripts for the same utterance, but those do not clear the timer, so the agent resumes before the turn commits and the partial transcript is lost. @superMDguy, if your setup differs (e.g. turn_detection="stt"), please share it.

On the design point: I agree interim transcripts alone are not reliable evidence of real speech, since noise can generate them, so clearing the timer on any interim would be wrong.

PR #7199 narrows this: the timer is only cancelled while a false-interruption pause is already active (_paused_speech set) and the interim/preflight is non-empty, and it re-arms when speaking is False.

With default turn detection, VAD still controls the pause itself; the change only stops the resume timer from firing over an STT stream that is still producing text.

If you would rather keep INTERIM_TRANSCRIPT from clearing the timer entirely and rely on VAD alone, happy to rework the PR in whichever direction fits the intended design. Let me know.

[superMDguy]

Yes, I'm using turn_detection="stt", with VAD enabled for barge-in. The agent reached a voicemail which read "You have reached the voicemail for" then went silent. Flux never sent an EOT message, so when the false interruption timer fired, the interim transcript was discarded. I'm working on a minimal repro script to make it more clear.

Is sustained VAD supposed to cancel the false interruption timeout with turn_detection="stt"? That probably would have fixed this case.

[superMDguy]

Okay here's a repro script:

https://gist.github.com/superMDguy/383d85e70b970c2717d97dd6a9a0d359

In this case, the turn gets auto-committed by the shutdown fallback path, but the core issue of the agent resuming while the user was speaking occurs.

[dorukdumlu]

To answer the sustained VAD question: it does reset the timer, but only through _interrupt_by_audio_activity, and on_vad_inference_done only calls that for frames with speech_duration >= interruption.min_duration. Below that gate nothing touches the timer, and with turn_detection="stt" a real utterance can go by without a single qualifying VAD event, so the timer genuinely can fire mid-speech. We ran into two variants of that window in #7101, which is why the resume path there now guards against clearing a live turn's state.

On the design question I'd side with longcw that interims shouldn't clear the timer, noise produces them too. But there's a middle ground between clearing and ignoring: a non-empty interim while the pause is active could re-arm the timer the way finals already do, with a cap on total deferral so a noisy line can't hold the pause open forever. That fails in the right direction on both sides: real speech keeps deferring until its final lands, pure noise eventually resumes instead of leaving dead air.

Also worth knowing: Flux sending interims but never an EOT is being tracked from the pipecat side too (pipecat-ai/pipecat#5735), and Deepgram's own docs say Update messages should arrive roughly every 0.25s while audio flows. So part of this is a provider stall that no timer policy can fully absorb. The policy just picks the worst case, and bounded defer makes it "agent resumes" rather than "caller waits forever".

[devtechedge]

Thanks for digging in, @dorukdumlu.

One correction on the mechanism, since it moves where the gap actually is.

on_start_of_speech calls _cancel_false_interruption_timer() on every VAD START_OF_SPEECH, and that path never consults interruption.min_duration.

min_duration only gates whether on_vad_inference_done reaches _interrupt_by_audio_activity, which is the pause, not the timer.

A sub-threshold segment still resets the timer through SOS, so sustained speech under turn_detection="stt" does keep cancelling it.

The narrower window is where there is no VAD start-of-speech at all but STT is still emitting text, which is @superMDguy's case: the far end goes quiet, Flux keeps sending interims and never sends an EOT.

That window exists because the timer branch in on_interim_transcript is gated on self.vad is None, while on_final_transcript runs the same speaking is False re-arm with no such gate.

I agree with you and @longcw that interims should not clear the timer.

Re-arming instead of clearing fails in the right direction and matches what finals already do.

The bit I want @longcw to settle is the bound: without one, a provider stalled the way Flux is here keeps producing interims and can hold the pause open indefinitely, which is just a different kind of dead air.

#7199 is open on the clear-on-interim approach, which this thread has talked me out of.

I would rather rework it to re-arm-only on the VAD path than leave it on a design you do not want.

Say the word if you would rather fold this into #7101 instead, since both touch the same resume path.

[dorukdumlu]

You're right, the cancel is in on_start_of_speech (agent_activity.py:2389) and min_duration only gates the pause. I misread that, thanks.

Keep #7199 separate from #7101. Same lines, different decisions: when the resume fires vs what it discards. Re-arm with a cap sounds right to me. One data point for the bound: pipecat-ai/pipecat#5735 replayed a call recording straight into Flux and got interims with no EOT for the whole file at eot_threshold=0.85, so an unbounded re-arm holds the pause open until the caller hangs up.

[devtechedge]

Agreed on both counts, so #7199 stays separate from #7101 and the re-arm now carries a bound.

The cap is 3x false_interruption_timeout, tracked as a per-pause deferral budget.

Once that budget is spent, further interims stop moving the deadline and the armed timer fires, so the agent resumes instead of holding the pause open.

That is exactly the case your pipecat#5735 data point describes, where interims arrive with no EOT for the whole recording.

The budget resets whenever the pause is resolved, so a busy line gets a fresh allowance on each new interruption rather than inheriting the last one.

Interims still never clear the timer outright, only defer it, which keeps the noise case failing in the right direction.

For now the cap is a module constant rather than a new option in InterruptionOptions, since I did not want to add public API without maintainer agreement on the semantics.

Happy to promote it to a real option if longcw would rather have it configurable.