Beyond the model itself, what does the engineering around a production LLM call look like, timeouts, retries, fallbacks?
What they are really testing: Whether they treat a model API as what it is, an unreliable remote dependency, and wrap it in the same resilience engineering as any other. Demo code has none of this, production code is mostly this.
A real interview question
Beyond the model itself, what does the engineering around a production LLM call look like, timeouts, retries, fallbacks?
What most people say
drag me
“I call the API in a try-catch and show an error message if it fails, maybe retry once.”
It handles the exception, not the reliability problem. No backoff means retry storms during provider incidents, no timeout means hung requests, no fallback means every provider blip is a full feature outage.
The follow-ups they ask next
Why is jitter in the backoff important?
Synchronized retries from many clients re-spike the recovering service. Randomised delays spread the herd.
What changes about retries mid-agent-run?
Steps may have side effects, so you need idempotency keys or checkpointing to resume, retrying the failed step only, never replaying completed actions.
What the interviewer is listening for
- Backoff with jitter, retry-after respected, retries capped and budgeted
- Fallback chain thought through: model, provider, cache, honest error
- Mentions circuit breaking and per-feature limits unprompted
What sinks the answer
- Naive immediate retries with no backoff
- No timeout on calls at all
- Single provider, no degradation path, spinner-forever UX
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“Treat the API as [an unreliable dependency]: [explicit timeouts including first-token], [backoff with jitter capped at 2 to 3 tries within a latency budget], [no blind retries past side effects], then a fallback chain: [smaller model or second provider, cache, honest failure UX], with [circuit breaking] so incidents shed load.”
Keep going with performance
Junior
Users say your AI feature feels slow. What are the levers for making an LLM-backed feature feel fast?
Senior
You are adding a voice interface to your assistant. What changes architecturally when the latency budget is conversational?
Junior
How do you decide how to chunk documents for a RAG system, and what goes wrong with naive chunking?
Junior
Your LLM feature must return JSON that downstream code parses. How do you make that reliable?
Junior
What is prompt injection, and why is it a bigger deal once your model can use tools?
Junior
Your LLM feature works in the demo. What will it cost in production, and where do the surprises come from?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free