You are adding a voice interface to your assistant. What changes architecturally when the latency budget is conversational?

Question

Accepted Answer

The budget change is qualitative: chat tolerates 3 to 5 seconds, voice feels broken past about a second of silence, so the target is first audio in under roughly 800ms, and that means rebuilding the pipeline around overlap. Everything streams and everything overlaps: speech-to-text is streaming, so transcription of the user is finishing roughly as they stop speaking, not starting; endpointing, detecting they are done, matters enormously because a lazy 700ms silence threshold spends most of the budget before we begin, so a fast semantic endpointer is worth real engineering; generation starts on the finalised transcript immediately and streams; and synthesis speaks the first sentence while the second is still generating, sentence-level chunked TTS, so first-audio depends only on time-to-first-sentence, not total response length. Then the pipeline itself slims, because a 4-second text stack cannot hide inside a 1-second budget: a fast model for the conversational layer, escalating to a stronger one only when the turn needs it; context trimmed hard, long prompts also slow time-to-first-token; retrieval pre-warmed or run speculatively while the user is still talking, betting on the likely topic; and responses designed short, voice answers are 1 to 3 sentences, which conveniently also cuts generation time. Where the use case allows, native speech-to-speech models remove the STT and TTS hops entirely and preserve prosody, at the cost of harder integration with tools and RAG, so the pragmatic 2026 architecture is often hybrid: speech-to-speech for the conversational skin, a text pipeline behind it for tool-using turns, with a filler, "let me check that", covering the tool latency honestly. The conversational machinery is product-critical: barge-in that stops synthesis instantly and records what was actually heard; recovery turns for misrecognition; and p95 first-audio per turn type, because the tail is what makes a voice product feel haunted. The mental shift: in chat you optimise a response, in voice you engineer a conversation, and silence is the error state.

Answer

The assistant needs a 2-second tool call mid-turn. How do you keep the conversation alive?

Acknowledge immediately with a natural filler generated before the tool call, stream the real answer the moment the result lands, and cap tool time with a spoken fallback. Dead air is the failure, not the latency itself.

Answer

How does barge-in actually work end to end?

Echo-cancelled always-on listening detects user speech over playback, synthesis halts within maybe 100ms, generation cancels, and the context must record where playback stopped, otherwise the model believes the user heard content that was cut off.

You are adding a voice interface to your assistant. What changes architecturally when the latency budget is conversational?

You are adding a voice interface to your assistant. What changes architecturally when the latency budget is conversational?

The follow-ups they ask next

What the interviewer is listening for

What sinks the answer

If you genuinely do not know

Keep going with performance

Knowing the answer is not the same as recalling it under pressure