A model advertises a 200k context window. What can you actually rely on it for, and what not?
What they are really testing: Whether you know the difference between what fits and what works. Long-context marketing numbers are a trap juniors fall into, and the "lost in the middle" behaviour is the classic production surprise.
A real interview question
A model advertises a 200k context window. What can you actually rely on it for, and what not?
What most people say
drag me
“It means I can put 200k tokens in, so I could pass entire documents instead of doing RAG.”
Believing the headline number is exactly the mistake this question hunts for. It ignores degraded mid-context recall, and it ignores that sending 200k tokens on every request is a cost and latency disaster.
The follow-ups they ask next
When is dumping a whole document the right call despite the cost?
When the task needs global understanding, summarisation, cross-referencing, contract review, and chunk-level retrieval would lose the thread. Then cache aggressively.
What is prompt caching and how does it change this economics?
Providers charge much less, often 10x less, for repeated prefix tokens they have cached. A stable long prefix plus a small changing suffix makes long context far cheaper.
What the interviewer is listening for
- Distinguishes what fits from what is reliably used
- Knows the lost-in-the-middle effect and positions content accordingly
- Reaches for cost and latency numbers unprompted
What sinks the answer
- Treats the advertised window as fully usable
- Concludes long context makes RAG obsolete
- Never mentions cost or latency of full-window calls
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“The window is [capacity, not uniform attention]: recall is [best at the start and end, worst in the middle]. So I use it for [headroom], but still [retrieve the relevant slice and place it early], because sending everything is [slower, far more expensive, and often less accurate].”
Keep going with llm fundamentals
Foundation
What is a token, and why does it matter that models bill and limit by tokens rather than words?
Foundation
Why do language models hallucinate, and why can you not simply prompt them to stop?
Foundation
What does temperature do, and how would you set it for a customer-facing extraction API versus a marketing copy generator?
Junior
The model has no memory between API calls. How do chat products remember the conversation, and what goes wrong as it grows?
Foundation
What is an embedding, and what does "similar" actually mean when you search with one?
Foundation
Explain RAG to me, and tell me what problem it solves that a bigger model does not.
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free