The network team applied a change last night. Since then roughly five percent of connections between your web subnet and your data subnet time out, the other ninety five percent are perfectly fine. It is intermittent and nobody can reproduce it on demand. Walk me through how you attack this.
What they are really testing: They want to see if you can debug a partial, non-reproducible failure without flailing, and whether you know that an intermittent network fault usually means one node or one rule is different from the rest.
A real interview question
The network team applied a change last night. Since then roughly five percent of connections between your web subnet and your data subnet time out, the other ninety five percent are perfectly fine. It is intermittent and nobody can reproduce it on demand. Walk me through how you attack this.
What most people say
drag me
“Five percent is quite low so I would add retries in the application and keep an eye on it to see if it gets worse.”
Retries hide a real fault, the five percent is a signal that something is genuinely different about one path, and burying it means the next change makes it a hundred percent.
The follow-ups they ask next
Flow logs show nothing denied, yet connections still time out. Where do you look next?
Look beyond NSGs: user defined routes, an appliance in the path, SNAT port exhaustion, or the destination refusing connections.
The application team wants to add retries and move on. How do you respond?
Agree retries buy time, but insist the fault is found, because five percent today is a hundred percent after the next change.
How do you make this class of change safe next time?
Network changes go through infrastructure as code with a plan diff, a canary subnet, and a synthetic connectivity test in the pipeline.
What the interviewer is listening for
- Treats five percent as a clue rather than acceptable noise
- Actively converts an intermittent fault into a deterministic one
- Knows subnet and NIC level rules combine into effective rules
- Leaves behind a synthetic probe so the next drift is caught early
What sinks the answer
- Papers over the failure with retries and moves on
- Asks what the change was supposed to do instead of reading flow logs
- Checks only one side of the connection
- Chases three hypotheses at once and cannot say which was disproved
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“I have not debugged this exact NSG issue, but here is how I would reason about a five percent failure: find what is different about that five percent, because intermittent almost always means one path is not like the others.”
Keep going with troubleshooting
Junior
You get paged at 2am. Checkout latency has gone from 200ms to 4 seconds and it started right after the 11pm deploy. Walk me through what you do.
Junior
It is 3am and you get paged: the primary application server is at 100% disk usage and the app is throwing write errors. You are the only one online. Walk me through exactly what you do.
Junior
You push a new version of a service and within two minutes the pods are all showing CrashLoopBackOff. The team is watching you share your screen. Walk me through exactly what you do.
Junior
A developer pings you: the checkout service cannot reach the payments service, the calls just hang and time out. You have never worked on either service. Walk me through what you do.
Junior
A developer messages you: 'My app suddenly gets Access Denied writing to the S3 bucket. Nothing changed.' It worked yesterday. Walk me through what you do.
Mid
Your load balancer shows 5xx errors jumping from 0.1 percent to 12 percent of requests over ten minutes. There was no deploy. What do you do?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free