About 2% of requests return 500s. The rest are fine. Nothing is obviously broken. How do you find it?
What they are really testing: Partial failure is harder than total failure because averages hide it. They want to see you find what is different about the failing 2% rather than staring at aggregate dashboards.
A real interview question
About 2% of requests return 500s. The rest are fine. Nothing is obviously broken. How do you find it?
What most people say
drag me
“2% is within acceptable error budget, so I would monitor it and see if it gets worse.”
It rationalises a signal instead of investigating it. Partial failures are usually early warnings of something that will become total, and 2% of requests is a real number of real users experiencing a broken product right now.
The follow-ups they ask next
All failures come from one pod. Do you just delete it?
Take it out of rotation first to stop the bleeding, but capture logs, heap state and metrics before killing it, or you lose the cause and it recurs next week on a different pod.
Failures correlate with one customer. What does that suggest?
Data-dependent behaviour: unusual volume, a shape your code does not handle, or their traffic pattern hitting a limit. Reproduce with their data shape in staging rather than guessing.
What the interviewer is listening for
- Slices by dimension systematically
- Suspects a single bad replica early
- Refuses to rationalise partial failure
What sinks the answer
- Accepts it as within budget
- Stares at aggregates
- Kills the bad instance without capturing evidence
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“I find [what is different about the failing 2%] by slicing on [instance, region, version, endpoint, customer, payload shape]. 2% is [suspiciously close to one bad replica]. Also consider [data-dependent paths and connection pool exhaustion, which fail a small share first]. Then [get one full failing trace]. I would not [accept it as within budget].”
Keep going with troubleshooting
Junior
A developer says the build passes locally but fails in CI. How do you debug that?
Junior
A production server is failing writes. Disk is 100% full. Walk me through what you do.
Junior
At midnight everything started failing with TLS errors. What happened and what do you do?
Mid
A pod is in CrashLoopBackOff after a deploy. Walk me through your debugging, command by command.
Mid
The deploy reports success, pods are Running, but users get 502s. Where do you look?
Mid
A Kubernetes node goes NotReady and pods are stuck Terminating. What is happening and what do you do?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free