Your evals pass at 92 percent, but production complaints keep coming. Users say the assistant is failing at things your suite says it does well. Debug the gap.
What they are really testing: The maturity question about measurement itself. Evals are a proxy, and proxies rot: distribution drift, grader blindness, and context differences between the harness and production. Senior engineers debug the instrument, not just the system.
A real interview question
Your evals pass at 92 percent, but production complaints keep coming. Users say the assistant is failing at things your suite says it does well. Debug the gap.
What most people say
drag me
“The eval set is probably outdated, I would add the failing cases to it and retrain or improve the prompts until the new cases pass.”
Patching cases treats the symptom and skips the diagnosis: why did the suite miss them, is the whole distribution stale, does the harness even reproduce production conditions, is the grader blind. Without those answers the gap reopens next month.
The follow-ups they ask next
The harness passes the exact failing input. Walk me through reproducing the production failure.
Replay with full production context: same conversation history, same retrieved chunks from the trace, same model version and parameters. Add each element until it fails, the element that flips it names the gap.
How do you weight the eval set against traffic without drowning in rare-but-critical cases?
Two-axis weighting: volume slices keep the aggregate honest, and a separate high-stakes suite with its own gate protects the rare-but-expensive cases from being averaged away.
What the interviewer is listening for
- Runs the failing production inputs through the harness as the first experiment
- Distinguishes coverage gaps from harness-fidelity gaps from grader blindness
- Builds the production-to-eval feedback loop rather than patching cases
What sinks the answer
- Adds the cases and moves on with no diagnosis of the miss
- Never questions whether the harness reproduces production context
- Treats the judge as ground truth even against user complaints
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“Debug the instrument: [run the real failing inputs through the harness]. Harness fails too: [coverage drift, re-weight the set to live traffic]. Harness passes: [fidelity gap, add production context: history, retrieval, version]. Judge disagrees with users: [calibrate the rubric on the complaint set]. Then [a monthly production-to-eval loop] so 92 stays a number about the present.”
Keep going with troubleshooting
Junior
An alert fires: 30 percent of your model API calls are failing with 429 rate-limit errors. Users see errors. What do you do?
Mid
Users report your RAG assistant is giving wrong answers. Walk me through how you debug it.
Mid
You are paged: the overnight batch agent has been running 6 hours instead of 20 minutes and has spent 400 dollars. Walk me through your response.
Mid
Overnight, p95 latency on your AI assistant tripled from 4 to 12 seconds. No deploy happened. Diagnose it.
Senior
Support reports the AI assistant got noticeably worse this week. Nothing was deployed. Walk me through the investigation.
Senior
Design the model-serving layer for a company with 15 LLM features. One gateway or per-team integrations, and what lives in it?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free