Voice AI Interrupt Handling: Fix the Latency Gap Killing Conversations

Barge-in is where most voice AI products die quietly. Users talk over the bot, the bot keeps going, the user hangs up. Teams spend months tuning ASR and TTS quality, then ship an experience that feels like calling a 2009 IVR system because nobody budgeted engineering time for interruption handling.

This is the specific problem I want to pull apart — not voice AI in general, but the interrupt loop: the 200–400ms window between a user starting to speak and the system actually stopping, re-orienting, and responding coherently. That window is where conversations feel natural or feel broken. There is no middle.

Why Interrupt Handling Is Harder Than It Looks

A voice agent has at minimum four components in the real-time path: VAD (voice activity detection), ASR (speech-to-text), LLM inference, and TTS synthesis. Each adds latency. The interrupt problem cuts across all four simultaneously.

When a user starts speaking mid-response, you need to:

1. Detect that speech started (VAD fires) — typically 20–80ms depending on the VAD model and chunk size
2. Decide whether it's an interrupt or background noise — this is the hard one
3. Halt TTS output mid-sentence without producing a jarring audio artifact
4. Cancel or suspend the in-flight LLM generation
5. Re-route partial ASR input into a new completion context that includes what the agent already said
6. Respond coherently to what the user actually said — not what you assumed they'd say

Step 2 is where the architecture usually breaks. Teams default to a simple energy threshold on the VAD. This works in a quiet studio. In the real world — a Dubai taxi, an open-plan office, a user on AirPods who breathes audibly — it produces either constant false interruptions or missed barge-ins. Both are fatal to conversation quality.

Building Mashreq Neo's conversational banking flows taught me the same lesson in a different medium: users in noisy real-world environments behave nothing like users in a quiet QA lab, and designing for the lab is designing for failure.

The Three Interrupt Failure Modes in Production

Failure Mode 1: Deaf agent. The agent ignores real interruptions because the barge-in threshold is set too high. Users speak louder, get frustrated, hang up. This is the most common failure in enterprise deployments because teams set conservative thresholds during quiet QA and never revisit them at real-world noise floors.

Failure Mode 2: Startled agent. The threshold is too low. A breath, a background word, a click triggers a barge-in. The agent stops mid-sentence for nothing, then has no new user input to work with, so it either replays or goes silent. Users experience this as the agent being "glitchy." It erodes trust fast.

Failure Mode 3: Coherence collapse. The interrupt fires correctly, TTS halts correctly, but the LLM context passed to the new turn is malformed. The agent was mid-explanation. The user asked a clarifying question. The new LLM call gets a context that says "agent said: 'The total amount due is'" with no completion, and the user said "wait, which account?" — and the model has no grounding for what "which account" refers to. The agent either hallucinates or asks the user to repeat the whole thing. This is the least obvious failure and the most expensive to debug.

A Decision Framework for Interrupt Architecture

Before writing a line of interrupt-handling code, answer these four questions. Your answers determine the architecture.

Question	If Yes →	If No →
Is your deployment environment acoustically controlled?	Simple energy-threshold VAD may be sufficient	You need a learned VAD model (Silero, WebRTC VAD with tuning, or Deepgram's endpoint detection)
Do users tend to have short, transactional turns?	Aggressive barge-in threshold, minimal context carry-over	Conservative threshold, full context window on interrupt
Is LLM latency ≤ 300ms (cached or fast model)?	You can re-run full inference on interrupt	You need speculative generation or a fallback stub response
Does your TTS provider support mid-stream cancellation?	Clean halt with no audio artifact	You need client-side audio buffer flushing as a workaround

The combination of "uncontrolled acoustics + long turns + slow LLM + no mid-stream TTS cancel" is the worst-case stack. Imagine a team bolting a modern LLM onto a legacy telephony platform without re-architecting the audio pipeline — this exact combination is what they get, and it degrades badly in ways that are hard to attribute to any single component.

The Context Carry-Over Problem (and How to Solve It)

Coherence collapse (Failure Mode 3 above) has a concrete fix, but most teams skip it because it requires instrumenting the TTS output stream.

The key insight: you must track what the agent actually said, not what it was supposed to say. These diverge on every interrupt.

Here's the pattern that works:

python
class AgentTurnTracker:
 def __init__(self):
 self.committed_text = "" # What TTS has actually played
 self.pending_text = "" # Queued but not yet played
 self.generation_cursor = 0 # Tokens streamed from LLM

 def on_tts_chunk_played(self, text_chunk: str):
 """Called by TTS client each time audio chunk completes playback."""
 self.committed_text += text_chunk

 def on_interrupt(self) -> str:
 """Returns the partial agent utterance for context injection."""
 # Only what was actually heard by the user
 return self.committed_text.strip()

 def build_interrupt_context(self, user_interrupt_text: str) -> list[dict]:
 partial_agent_turn = self.on_interrupt()
 return [
 {"role": "assistant", "content": partial_agent_turn + " [interrupted]"},
 {"role": "user", "content": user_interrupt_text}
 ]

The [interrupted] marker isn't decoration — it's a signal to the model that the prior turn was cut. With a well-designed system prompt, this lets the LLM acknowledge the interruption naturally rather than pretending the full response was delivered. "You interrupted me before I finished" is a fine human response. A voice agent that acts as if it completed a sentence it didn't is uncanny.

TTS providers that expose chunk-level playback callbacks (versus just stream completion) make this straightforward. If your TTS layer doesn't surface this, you have two options: estimate played text from audio duration and character rate (rough, but serviceable for short utterances), or switch to a provider that does. At real-time latency requirements, this is not a nice-to-have.

Speculative Generation: The Latency Hedge for Slow Models

If your LLM inference is sitting at 600–900ms to first token (realistic for large models without caching on a cold prompt), every interrupt creates a dead-air gap that feels broken. Users fill silence with "hello?" which triggers another interrupt. It spirals.

The fix is a speculative stub: a fast, cheap model (or a rule-based system) generates a bridging phrase — "Sure, let me address that" or "Good question" — while the full LLM call runs in parallel. The stub plays immediately; the real response follows. Total perceived latency drops to the stub's generation time (typically 50–150ms), not the full model's TTFT.

This is not a new pattern — it's what telephony systems have done with hold music for decades. The difference is that your stub needs to be contextually plausible, not generic. "Sure, let me address that" works for most clarifications. "Of course" works for confirmations. A small classifier on the interrupt text (question vs. objection vs. correction) gets you to the right stub bucket with minimal overhead.

For teams running multi-agent stacks, this connects directly to the broader agent handoff problem — designing explicit contracts for what state gets passed between turns is the same discipline applied to the voice interrupt context.

VAD Tuning: The Numbers That Actually Matter

Energy-based VAD has two knobs: speech onset threshold and end-of-speech silence duration. Here's what I use as starting points:

• Onset threshold: 15–20dB above ambient floor for controlled environments; move to a learned VAD model if ambient noise is variable
• End-of-speech silence: 400–600ms for conversational turn-taking; drop to 200–300ms for transactional queries ("pay bill", "check balance") where users expect speed
• Barge-in guard window: 150–200ms after agent speech starts — suppress VAD during this window to avoid the agent's own TTS output triggering a false barge-in through acoustic echo

The barge-in guard window is the most-missed configuration in teams new to voice. If you're on a WebRTC stack with proper echo cancellation, this is handled. If you're in a telephony environment without full AEC, you need to implement it manually by disabling VAD for the first N milliseconds of each agent utterance.

The Latency Budget: Where Your 400ms Goes

A real-time voice agent that feels natural needs end-to-end response latency under 600ms from end of user speech to start of agent speech. Here's a realistic breakdown for a well-architected system:

Component	Realistic Range	Notes
VAD onset detection	20–80ms	Lower with smaller chunk sizes; tradeoff with accuracy
ASR first result	80–200ms	Streaming ASR with partial results; Deepgram, AssemblyAI
LLM first token	150–400ms	With prompt caching; without cache, 400–900ms
TTS first audio chunk	80–200ms	Streaming TTS; ElevenLabs, Cartesia, Play.ht
Network / buffering	20–60ms	Keep infra co-located with model endpoints
Total	350–940ms	Left tail requires caching + fast models + co-location

The 350ms case is achievable but requires every component to be optimized simultaneously — cached prompts, a fast model like GPT-4o Realtime or a Cartesia-tier TTS, and infra co-located with the model provider. The 940ms case is what you get when you connect disparate services across regions with no caching. Most production systems sit somewhere in the 500–700ms range, which is acceptable but not natural.

For context: human conversational response latency is 200–300ms. You're not going to beat human biology. But under 600ms, users stop consciously noticing the gap. Over 800ms consistently, they start filling it with words — and now you have a VAD problem again.

What to Actually Do

If you're building or auditing a voice AI system right now, here's the concrete sequence:

1. Instrument your TTS playback. If you can't tell what your agent actually said before an interrupt, you cannot build coherent multi-turn conversations. Add chunk-level tracking before touching anything else.

2. Run your VAD in your actual deployment environment, not a quiet office. Record 10 minutes of ambient audio from the real context (call center floor, mobile app in transit, browser in open-plan office) and replay it through your VAD pipeline. Measure false interrupt rate and missed barge-in rate. Set thresholds against real data.

3. Implement the interrupt context pattern. Use the [interrupted] marker in your message history. Test with a dozen realistic interrupt scenarios — mid-sentence, mid-list, mid-number-read. Check that the model responds coherently to each.

4. Measure your actual latency budget by component. Add spans to every stage of the pipeline. You cannot optimize what you haven't measured. The common surprise: ASR is rarely the bottleneck; LLM TTFT and TTS first chunk are usually where time is lost.

5. Add the speculative stub if your LLM TTFT exceeds 400ms. This is a one-day implementation that makes a perceptible difference in perceived responsiveness.

Voice AI that handles interrupts well doesn't feel like AI — it feels like talking to someone competent. That's the bar. Everything else is just a demo.