Why do language models hallucinate, and why can you not simply prompt them to stop?
What they are really testing: Whether you understand the mechanism well enough to design around it, or whether you think hallucination is a bug some better prompt will fix. The mechanism dictates the mitigations, which is what the role actually involves.
A real interview question
Why do language models hallucinate, and why can you not simply prompt them to stop?
What most people say
drag me
“Models sometimes make things up, so you add "do not make things up" to the system prompt.”
It treats a statistical property as a behaviour problem. The instruction shifts tone, not truthfulness, and an interviewer hears that you have never had to actually fix this in a product.
The follow-ups they ask next
Does retrieval eliminate hallucination?
No. The model can still contradict or embellish the supplied context, and it hallucinates when retrieval returns nothing relevant. It reduces the rate, so you still measure.
How would you measure your hallucination rate?
A fixed eval set with known answers, grade model outputs against them, either exact-match for factual fields or an LLM judge with spot-checked agreement. Track the rate per release.
What the interviewer is listening for
- Explains the mechanism, next-token prediction with no truth check
- Knows prompting alone barely helps and says why
- Names grounding, the explicit "I do not know" out, and measurement
What sinks the answer
- Believes a strong enough prompt fixes it
- Cannot explain why wrong answers sound confident
- No idea how to measure the rate
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“Models [predict plausible next tokens with no fact check], so wrong answers come out [as fluent as right ones]. You cannot prompt it away because [the instruction changes style, not the fact source]. The real fixes are [retrieval grounding with citations], [permission to say I do not know], and [measuring the rate on a known-answer eval set].”
Keep going with llm fundamentals
Foundation
What is a token, and why does it matter that models bill and limit by tokens rather than words?
Foundation
A model advertises a 200k context window. What can you actually rely on it for, and what not?
Foundation
What does temperature do, and how would you set it for a customer-facing extraction API versus a marketing copy generator?
Junior
The model has no memory between API calls. How do chat products remember the conversation, and what goes wrong as it grows?
Foundation
What is an embedding, and what does "similar" actually mean when you search with one?
Foundation
Explain RAG to me, and tell me what problem it solves that a bigger model does not.
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free