You changed a prompt and the feature "seems better". How do you know it actually is?
What they are really testing: The habit that separates AI engineering from vibes. Non-determinism plus anecdotes is how teams ship regressions, and this checks for the regression-test instinct applied to model behaviour.
A real interview question
You changed a prompt and the feature "seems better". How do you know it actually is?
What most people say
drag me
“I would test it with a bunch of examples and compare the outputs side by side to see which looks better.”
Eyeballing a handful of outputs is exactly how a fix for one case silently breaks five others. No fixed set, no grading criteria, no baseline: the conclusion is a mood, not a measurement.
The follow-ups they ask next
How much do you trust the LLM judge?
As far as its measured agreement with humans on a spot-checked sample, recalibrated when the rubric or model changes. It is a scalable approximation, not ground truth.
Model outputs vary run to run. How do evals stay meaningful?
Temperature 0 where possible, multiple runs per case where not, and thresholds with slack: gate on meaningful deltas, not single-case noise.
What the interviewer is listening for
- Reflex is a fixed measured set, not side-by-side eyeballing
- Grading method chosen per task type, judge calibrated against humans
- Failures feed the suite, and it gates changes in CI
What sinks the answer
- Ships prompt changes on anecdotal comparison
- One aggregate score, never looks at per-case flips
- Uses an LLM judge without ever checking it against humans
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“Replace [seems better] with [a fixed eval set of 50 to 100 real cases including past failures], graded by [exact checks, assertions, or a calibrated LLM judge], run [on every change against a baseline], watching for [cases that flipped from pass to fail], not just the aggregate.”
Keep going with evals
Junior
How do you decide how to chunk documents for a RAG system, and what goes wrong with naive chunking?
Junior
Your LLM feature must return JSON that downstream code parses. How do you make that reliable?
Junior
What is prompt injection, and why is it a bigger deal once your model can use tools?
Junior
Your LLM feature works in the demo. What will it cost in production, and where do the surprises come from?
Junior
Users say your AI feature feels slow. What are the levers for making an LLM-backed feature feel fast?
Junior
How does tool calling actually work under the hood, and what makes a tool definition good versus bad?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free