Users report your RAG assistant is giving wrong answers. Walk me through how you debug it.
What they are really testing: The single most common real AI engineering task. The signal is decomposition: retrieval and generation fail differently, and someone who inspects the retrieved chunks before touching the prompt has done this before.
A real interview question
Users report your RAG assistant is giving wrong answers. Walk me through how you debug it.
What most people say
drag me
“I would improve the prompt to tell the model to be more accurate and only use the provided context.”
It jumps to the generation side without ever checking what was retrieved, and in most wrong-answer reports the model faithfully answered from wrong or missing chunks, so the prompt edit fixes nothing.
The follow-ups they ask next
The right chunk is retrieved at position 8 of 10 but the answer is still wrong. What is happening?
Position and dilution: the model attends poorly to material buried mid-context among 9 distractors. Rerank to put the best chunks first and send fewer of them.
How do you measure retrieval quality on its own?
A labelled set of query to relevant-passage pairs, then recall at k and MRR. It isolates the retriever from the generator so you know which to fix.
What the interviewer is listening for
- Reads the retrieved chunks before editing any prompt
- Names the query-phrasing mismatch and chunking as usual suspects
- Converts every confirmed failure into a regression eval case
What sinks the answer
- Starts and ends with prompt tweaks
- Never inspects traces or retrieved content
- No per-stage metrics, debugs the same class of bug repeatedly
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“Collect [real failing queries and full traces], then ask [was the right passage retrieved]. Missing: fix [chunking, phrasing mismatch, stale index, filters]. Present: fix [grounding, context placement, reranking]. Then [every failure becomes an eval case] so it stays fixed.”
Keep going with troubleshooting
Junior
An alert fires: 30 percent of your model API calls are failing with 429 rate-limit errors. Users see errors. What do you do?
Mid
You are paged: the overnight batch agent has been running 6 hours instead of 20 minutes and has spent 400 dollars. Walk me through your response.
Mid
Overnight, p95 latency on your AI assistant tripled from 4 to 12 seconds. No deploy happened. Diagnose it.
Senior
Support reports the AI assistant got noticeably worse this week. Nothing was deployed. Walk me through the investigation.
Senior
Your evals pass at 92 percent, but production complaints keep coming. Users say the assistant is failing at things your suite says it does well. Debug the gap.
Mid
What are hybrid search and reranking, and when does a RAG system actually need them?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free