Support reports the AI assistant got noticeably worse this week. Nothing was deployed. Walk me through the investigation.
What they are really testing: The signature LLM incident: quality regression with no deploy. It tests whether they know the invisible change surfaces, provider model updates, index drift, traffic shift, and whether they can turn "feels worse" into something diagnosable.
A real interview question
Support reports the AI assistant got noticeably worse this week. Nothing was deployed. Walk me through the investigation.
What most people say
drag me
“Since nothing was deployed on our side, I would suspect the provider changed something and open a ticket with them while we monitor.”
It guesses the most famous cause and outsources the investigation. No examples collected, no timeline, no check of the index or traffic mix, and a provider ticket with no evidence gets the response it deserves.
The follow-ups they ask next
The provider confirms a model update. You cannot roll their change back. Options?
Pin to a dated model version if offered, adapt prompts to the new behaviour and re-eval, shift affected routes to an alternative model through the gateway, and prioritise by which features regressed.
How would you have caught this without support noticing first?
A daily canary eval against production config with alerting on score drops, plus monitored user-feedback rates. Silent-change detection is a running probe, not a hope.
What the interviewer is listening for
- Quantifies the regression and pins the start time before hypothesising
- Enumerates non-deploy change surfaces: provider, index, traffic, fallbacks
- Ends by building the alarm that was missing, not just fixing the instance
What sinks the answer
- Blames the provider by default with no evidence gathered
- Forgets the index and traffic mix as change surfaces entirely
- Fixes the instance and leaves detection exactly as blind as before
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“Turn [feels worse] into [examples, metrics, and a start time]. Nothing deployed still leaves [provider model updates, index drift, traffic shift, silent fallbacks]. Decide with [the canary eval and before-after trace diffs on the same queries], then [fix the cause] and [build the alert that would have caught it in hours].”
Keep going with troubleshooting
Junior
An alert fires: 30 percent of your model API calls are failing with 429 rate-limit errors. Users see errors. What do you do?
Mid
Users report your RAG assistant is giving wrong answers. Walk me through how you debug it.
Mid
You are paged: the overnight batch agent has been running 6 hours instead of 20 minutes and has spent 400 dollars. Walk me through your response.
Mid
Overnight, p95 latency on your AI assistant tripled from 4 to 12 seconds. No deploy happened. Diagnose it.
Senior
Your evals pass at 92 percent, but production complaints keep coming. Users say the assistant is failing at things your suite says it does well. Debug the gap.
Senior
Design the model-serving layer for a company with 15 LLM features. One gateway or per-team integrations, and what lives in it?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free