Your LLM bill doubled month over month with flat user growth. How do you find the cause and get it down?
What they are really testing: A realistic on-call-adjacent scenario testing attribution discipline. Flat users with doubling spend means per-request or per-user consumption changed, and finding it requires the per-feature instrumentation the weak answer lacks.
A real interview question
Your LLM bill doubled month over month with flat user growth. How do you find the cause and get it down?
What most people say
drag me
“I would switch to a cheaper model and add caching to bring the spend down quickly.”
Optimising before diagnosing. If the driver is an agent retry loop or a history bug, the cheap model halves a bill that should have dropped 90 percent, and the actual regression ships again next sprint unnoticed.
The follow-ups they ask next
How does prefix caching actually save money?
Providers cache the processed prefix and charge cached tokens at a steep discount, around 90 percent off. It requires the stable content, system prompt, tool defs, to be a byte-identical prefix, so prompt structure determines savings.
When is semantic response caching safe, and when is it a correctness bug?
Safe for identical or near-identical queries over stable knowledge with no per-user context. Dangerous when answers are personalised, permissioned or time-sensitive, cache keys must include everything that changes the answer.
What the interviewer is listening for
- Attributes spend by feature, model, token direction and user before touching anything
- Names concrete regressors: history growth, k changes, retry and agent loops
- Installs budget alerts so the next anomaly surfaces in a day, not a month
What sinks the answer
- Jumps to a cheaper model with no diagnosis
- No per-feature attribution and no plan to add it
- Treats cost as finance business rather than an engineering signal
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“Flat users with doubled spend means [consumption changed, find what]: break the bill by [feature, model, input versus output, user distribution], check [history growth, context changes, retries, agent loops, model mix], fix the driver, then [prefix caching, right-sized models, response caching, batch APIs] and [per-feature budget alerts] so it never surprises again.”
Keep going with cost
Junior
Your LLM feature works in the demo. What will it cost in production, and where do the surprises come from?
Mid
Users report your RAG assistant is giving wrong answers. Walk me through how you debug it.
Mid
What are hybrid search and reranking, and when does a RAG system actually need them?
Mid
What actually is an agent, mechanically, and what stops one from looping forever or going off the rails?
Mid
What problem does MCP, the Model Context Protocol, solve, and what should you check before plugging a third-party MCP server into your assistant?
Mid
Design the guardrails for a customer-facing AI assistant. What layers do you put around the model?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free