The model has no memory between API calls. How do chat products remember the conversation, and what goes wrong as it grows?
What they are really testing: Statelessness is the mechanical fact everything else builds on. This checks you know memory is an application-layer illusion you construct, and that you have thought about the cost and quality decay of long histories.
A real interview question
The model has no memory between API calls. How do chat products remember the conversation, and what goes wrong as it grows?
What most people say
drag me
“The chat keeps the history and the model uses it to remember what was said earlier in the session.”
Describes the illusion from the outside. Who keeps the history, what gets sent, and what happens at turn 60 when the window or the budget runs out is the part the engineer owns, and it is absent.
The follow-ups they ask next
The user says "as I told you earlier" but that turn was summarised away. What now?
This is the retrieval case: search the stored full transcript for the referenced content and reinject it, rather than relying on what survived summarisation.
What belongs in pinned memory versus the running summary?
Pin durable facts and constraints, name, preferences, decisions. Summarise narrative flow. Pins are small and always present, summaries are lossy by design.
What the interviewer is listening for
- Says plainly the model is stateless and the app constructs memory
- Quantifies the cost growth and knows attention decays before the window fills
- Names window plus summary plus retrieval as the standard architecture
What sinks the answer
- Believes the model or provider remembers between calls
- Plan is to send the full transcript forever
- Never tested behaviour on long conversations
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“Models are [stateless per call], so memory is [the app resending curated context]. Full transcripts fail on [cost growth per turn] and [attention decay]. The pattern: [recent turns verbatim], [running summary of older ones], [pinned durable facts], and [retrieval over stored history] for anything beyond.”
Keep going with llm fundamentals
Foundation
What is a token, and why does it matter that models bill and limit by tokens rather than words?
Foundation
Why do language models hallucinate, and why can you not simply prompt them to stop?
Foundation
A model advertises a 200k context window. What can you actually rely on it for, and what not?
Foundation
What does temperature do, and how would you set it for a customer-facing extraction API versus a marketing copy generator?
Junior
How do you decide how to chunk documents for a RAG system, and what goes wrong with naive chunking?
Junior
Your LLM feature must return JSON that downstream code parses. How do you make that reliable?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free