p99 latency tripled overnight with no deploy and no traffic change. How do you investigate?
What they are really testing: Whether you can reason about slow degradation with no obvious trigger. Removing the deploy explanation forces you to think about accumulating state, which is where the real skill shows.
A real interview question
p99 latency tripled overnight with no deploy and no traffic change. How do you investigate?
What most people say
drag me
“I would scale up the service to handle the load better.”
It treats a symptom with capacity without knowing the cause, and if the bottleneck is a shared database or a lock, adding instances makes it worse by increasing concurrency against the same contended resource.
The follow-ups they ask next
Tracing shows one database query went from 5ms to 900ms. What do you check?
The query plan first: it may have flipped to a table scan as row counts changed or statistics went stale. Then lock contention, replication lag, and whether an index was dropped or invalidated.
Only 1 of your 6 instances is slow. What does that suggest?
Something instance-local: a noisy neighbour, a degraded disk, a stuck thread pool, memory pressure causing GC pauses, or an uneven shard assignment. Remove it from rotation and see if p99 recovers.
What the interviewer is listening for
- Distinguishes step change from gradual ramp
- Thinks about accumulating state
- Compares traces rather than reading dashboards
What sinks the answer
- Scales first
- Only looks at their own service
- Cannot explain why p99 differs from average
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“No deploy points at [accumulating state or something outside my service]. I check [the shape: step change means an event, gradual ramp means something filling], then [trace to find which span grew], then [data growth, index thresholds, cache hit rate, queue depth, disk]. Also [a dependency may have deployed]. Scaling [can make contention worse].”
Keep going with troubleshooting
Junior
A developer says the build passes locally but fails in CI. How do you debug that?
Junior
A production server is failing writes. Disk is 100% full. Walk me through what you do.
Junior
At midnight everything started failing with TLS errors. What happened and what do you do?
Mid
A pod is in CrashLoopBackOff after a deploy. Walk me through your debugging, command by command.
Mid
The deploy reports success, pods are Running, but users get 502s. Where do you look?
Mid
A Kubernetes node goes NotReady and pods are stuck Terminating. What is happening and what do you do?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free