Users say your AI feature feels slow. What are the levers for making an LLM-backed feature feel fast?
What they are really testing: Latency is the most user-visible property of these systems. The key insight being probed: perceived latency is dominated by time-to-first-token, and streaming changes the experience more than most model-side optimisations.
A real interview question
Users say your AI feature feels slow. What are the levers for making an LLM-backed feature feel fast?
What most people say
drag me
“I would switch to a faster model or a smaller one so responses come back quicker.”
One lever, picked blind. Without splitting first-token from total time or knowing where the pipeline spends its time, a model swap may trade quality away for a latency the UX change of streaming would have fixed free.
The follow-ups they ask next
Why does streaming not reduce total latency but still matter so much?
It moves feedback to first-token time, users read while the rest generates, so waiting and reading overlap. Perceived latency is about feedback, not completion.
Your agent takes 20 seconds across 6 tool calls. Options?
Parallelise independent calls, collapse steps into fewer tool round-trips, use a faster model for intermediate reasoning, stream progress so the wait is legible, and question whether all 6 steps earn their latency.
What the interviewer is listening for
- Separates time-to-first-token from total time immediately
- Streams first, then optimises, and traces the whole pipeline
- Knows output length is the generation-time driver and designs for brevity
What sinks the answer
- Only lever offered is swapping models
- No pipeline trace, guesses where time goes
- Optimises average latency and ignores the tail
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“Split [time to first token from total time], users feel the first. Then: [stream by default], [cap output length since generation scales with it], [parallelise and prune pipeline steps], [small models for easy steps], [prefix and response caching], and measure [p95, not average].”
Keep going with performance
Junior
Beyond the model itself, what does the engineering around a production LLM call look like, timeouts, retries, fallbacks?
Senior
You are adding a voice interface to your assistant. What changes architecturally when the latency budget is conversational?
Junior
How do you decide how to chunk documents for a RAG system, and what goes wrong with naive chunking?
Junior
Your LLM feature must return JSON that downstream code parses. How do you make that reliable?
Junior
What is prompt injection, and why is it a bigger deal once your model can use tools?
Junior
Your LLM feature works in the demo. What will it cost in production, and where do the surprises come from?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free