Your LLM feature works in the demo. What will it cost in production, and where do the surprises come from?
What they are really testing: Whether you can do the unit-economics arithmetic before launch and whether you know the classic multipliers, history growth, retries, verbose outputs, that make real bills 5 to 10x the naive estimate.
A real interview question
Your LLM feature works in the demo. What will it cost in production, and where do the surprises come from?
What most people say
drag me
“I would look at the price per million tokens and multiply by our expected usage to get a monthly estimate.”
The naive estimate is the one that is wrong by 5 to 10x, because it prices single calls while production sends history per turn, retrieved context per request, and suffers retries, none of which appeared in the demo.
The follow-ups they ask next
Why does conversation cost grow so fast with turns?
Each turn resends all prior turns as input, so input tokens grow linearly per turn and total conversation cost roughly quadratically until you cap or summarise history.
Marketing wants a cost-per-conversation number. What do you measure?
Distribution, not average: log tokens per request, aggregate per conversation, report median and p95, because agentic or long-chat outliers dominate the mean.
What the interviewer is listening for
- Estimates from a real production-shaped request
- Names history resending and output pricing as the multipliers
- Instruments cost per request before launch, not after the first bill
What sinks the answer
- Prices a single naive call and multiplies
- Does not know output tokens cost more than input
- No per-feature cost attribution planned
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“Estimate from [a real request: system prompt, context, history, realistic answer], not the pricing page. The surprises are [history resent every turn], [output tokens at 4 to 5x input price], and [retries]. Defences: [log cost per request from day 1], [cap history and output], [cache the stable prefix], [right-size the model].”
Keep going with cost
Mid
Your LLM bill doubled month over month with flat user growth. How do you find the cause and get it down?
Junior
How do you decide how to chunk documents for a RAG system, and what goes wrong with naive chunking?
Junior
Your LLM feature must return JSON that downstream code parses. How do you make that reliable?
Junior
What is prompt injection, and why is it a bigger deal once your model can use tools?
Junior
Users say your AI feature feels slow. What are the levers for making an LLM-backed feature feel fast?
Junior
How does tool calling actually work under the hood, and what makes a tool definition good versus bad?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free