Overnight, p95 latency on your AI assistant tripled from 4 to 12 seconds. No deploy happened. Diagnose it.
What they are really testing: Whether they can decompose an LLM pipeline latency budget and know the no-deploy suspects: provider degradation, silent fallbacks, input growth, and retry amplification. The tell is asking where in the pipeline the time went before proposing anything.
A real interview question
Overnight, p95 latency on your AI assistant tripled from 4 to 12 seconds. No deploy happened. Diagnose it.
What most people say
drag me
“Probably the provider is slow, I would check their status page and switch to a backup model if it continues.”
One hypothesis, tested by a status page that famously lags real incidents, and a mitigation with no evidence it targets the cause. If the real driver is our own retry storm or input growth, the backup model inherits it.
The follow-ups they ask next
Traces show time-to-first-token is flat but total time tripled. What does that tell you?
Generation lengthened: output tokens grew. Something is making answers longer, a prompt change upstream, a document format change, or users pasting bigger inputs, check output token counts per request.
Everything looks flat per-stage but p95 still tripled. What did you miss?
Queueing between stages, the time is in the gaps, not the spans. Check queue depth and scheduler delay, and whether concurrency limits or rate-limit backpressure built up a wait that traces attribute to no stage.
What the interviewer is listening for
- Decomposes the pipeline and localises before hypothesising
- Knows the silent-fallback and retry-amplification traps
- Mitigates for users in parallel and closes with the missing per-stage alert
What sinks the answer
- Single hypothesis, tested only by the provider status page
- Never looks at token counts or model distribution in traces
- No user-facing mitigation while the diagnosis runs
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“Localise first: [per-stage timings name the suspect class]. Then the no-deploy suspects: [provider TTFT], [silent fallback in the model distribution], [input and output token growth], [retry amplification], [queue depth]. Mitigate in parallel [failover, tighter timeouts, capped output] and add [per-stage latency alerts] after.”
Keep going with troubleshooting
Junior
An alert fires: 30 percent of your model API calls are failing with 429 rate-limit errors. Users see errors. What do you do?
Mid
Users report your RAG assistant is giving wrong answers. Walk me through how you debug it.
Mid
You are paged: the overnight batch agent has been running 6 hours instead of 20 minutes and has spent 400 dollars. Walk me through your response.
Senior
Support reports the AI assistant got noticeably worse this week. Nothing was deployed. Walk me through the investigation.
Senior
Your evals pass at 92 percent, but production complaints keep coming. Users say the assistant is failing at things your suite says it does well. Debug the gap.
Mid
What are hybrid search and reranking, and when does a RAG system actually need them?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free