Explain RAG to me, and tell me what problem it solves that a bigger model does not.
What they are really testing: RAG is the most common architecture you will inherit. The second half of the question filters people who can define it from people who know why it exists, knowledge freshness, private data and verifiability are not solved by scale.
A real interview question
Explain RAG to me, and tell me what problem it solves that a bigger model does not.
What most people say
drag me
“RAG is retrieval augmented generation, you fetch documents and put them in the prompt so the model has more context.”
Expands the acronym and stops. It never answers the actual question, what this solves that model scale does not, which is where the engineering judgment lives.
The follow-ups they ask next
When is RAG the wrong tool?
When the goal is behaviour or style rather than knowledge, teaching a format or tone, that is fine-tuning or prompting territory. Also when the corpus is tiny enough to fit in the prompt every time.
Where do RAG systems usually fail in practice?
Retrieval, not generation: bad chunking, embedding mismatch between query and document phrasing, or the answer living across multiple chunks that never get retrieved together.
What the interviewer is listening for
- Answers the "that a bigger model does not" half explicitly
- Names freshness, private data and citations as the three wins
- Knows retrieval, not generation, is where RAG usually breaks
What sinks the answer
- Defines the acronym and stops
- Thinks long context windows or bigger models make RAG obsolete
- Has never considered where the failures happen
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“RAG is [retrieve relevant passages, then generate grounded in them]. It solves what scale cannot: [knowledge freshness via index updates], [private data the model never trained on], and [citable answers]. The cost is a new failure surface, [retrieval quality], which is where most real RAG bugs live.”
Keep going with rag & retrieval
Foundation
What is an embedding, and what does "similar" actually mean when you search with one?
Foundation
Do you actually need a dedicated vector database to ship a RAG feature? Walk me through how you would decide.
Junior
How do you decide how to chunk documents for a RAG system, and what goes wrong with naive chunking?
Mid
What are hybrid search and reranking, and when does a RAG system actually need them?
Foundation
What is a token, and why does it matter that models bill and limit by tokens rather than words?
Foundation
Why do language models hallucinate, and why can you not simply prompt them to stop?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free