How do you decide how to chunk documents for a RAG system, and what goes wrong with naive chunking?
What they are really testing: Chunking is the highest-leverage, least glamorous decision in RAG, and the classic source of "the model is bad" complaints that are actually retrieval bugs. This checks you have debugged one, not just diagrammed one.
A real interview question
How do you decide how to chunk documents for a RAG system, and what goes wrong with naive chunking?
What most people say
drag me
“I would split the documents into 500-token chunks with some overlap, that is the standard approach.”
A copied default presented as a decision. It ignores document structure entirely, and 500 tokens is exactly the kind of setting that should come out of measurement, not folklore.
The follow-ups they ask next
A user asks a question whose answer spans three chunks. What happens and what do you do?
Top-k may fetch only one piece, so the model answers from a fragment. Mitigate with retrieval that pulls neighbouring chunks, parent-document retrieval, or restructuring so answers cohere.
How would you chunk a 200-row pricing table?
Not by tokens. Keep it whole if feasible, or split by row groups with the header repeated on every piece, plus a natural-language summary chunk pointing to it.
What the interviewer is listening for
- Reasons about the precision-context trade, not a magic number
- Splits on document structure and keeps chunks self-describing
- Has a retrieval eval and tunes chunking against it
What sinks the answer
- One fixed size for everything, defended as standard
- No metadata attached to chunks
- Never measured retrieval quality directly
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“Chunking trades [retrieval precision against usable context]. Split on [structure: headings, sections, paragraphs], keep chunks [self-describing with title and section prepended], add [10 to 15 percent overlap], and tune with [a retrieval eval of 50 known-answer questions], not folklore sizes.”
Keep going with rag & retrieval
Foundation
What is an embedding, and what does "similar" actually mean when you search with one?
Foundation
Explain RAG to me, and tell me what problem it solves that a bigger model does not.
Foundation
Do you actually need a dedicated vector database to ship a RAG feature? Walk me through how you would decide.
Mid
What are hybrid search and reranking, and when does a RAG system actually need them?
Junior
Your LLM feature must return JSON that downstream code parses. How do you make that reliable?
Junior
What is prompt injection, and why is it a bigger deal once your model can use tools?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free