An alert fires: 30 percent of your model API calls are failing with 429 rate-limit errors. Users see errors. What do you do?
What they are really testing: A rite-of-passage incident for anyone shipping on LLM APIs. Tests whether they know 429s are a demand-shape problem, that naive retries make them worse, and that the fix has an immediate, structural and preventative layer.
A real interview question
An alert fires: 30 percent of your model API calls are failing with 429 rate-limit errors. Users see errors. What do you do?
What most people say
drag me
“I would add retries with exponential backoff so failed requests eventually succeed, and request a higher rate limit from the provider.”
Retries are already the amplifier: at 30 percent failure, every retry adds demand to a saturated quota, and without finding what consumed the quota, a higher limit is a bigger bucket for the same leak, possibly a runaway loop now burning more money.
The follow-ups they ask next
Why jitter in the backoff specifically?
Without it, all failed clients retry in synchronized waves that re-spike the quota at the same instant. Randomised delays spread the retry load flat.
The runaway turns out to be a customer integration, not your code. Now what?
Per-client rate limits and quotas at your API boundary, so one tenant exhausts their allocation, not the shared pool, plus contact and possibly a temporary block. Multi-tenant fairness is your job, not the provider.
What the interviewer is listening for
- Recognises retries as the amplifier and caps them first
- Sheds background traffic to protect interactive users immediately
- Hunts the demand change, then separates lanes and alerts on utilisation, not failure
What sinks the answer
- Adds retries into a saturated quota
- Requests a higher limit without finding what consumed the current one
- No distinction between interactive and background traffic
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“Order: [cap retries, backoff with jitter, honour retry-after], [pause background consumers so users get the quota], [find the demand change, often a runaway loop or batch job], then structurally [separate lanes, client-side limits at 80 percent, priority queue] and [alert on quota utilisation, not user-visible failure].”
Keep going with troubleshooting
Mid
Users report your RAG assistant is giving wrong answers. Walk me through how you debug it.
Mid
You are paged: the overnight batch agent has been running 6 hours instead of 20 minutes and has spent 400 dollars. Walk me through your response.
Mid
Overnight, p95 latency on your AI assistant tripled from 4 to 12 seconds. No deploy happened. Diagnose it.
Senior
Support reports the AI assistant got noticeably worse this week. Nothing was deployed. Walk me through the investigation.
Senior
Your evals pass at 92 percent, but production complaints keep coming. Users say the assistant is failing at things your suite says it does well. Debug the gap.
Junior
How do you decide how to chunk documents for a RAG system, and what goes wrong with naive chunking?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free