You own a system of many services and an incident took too long to diagnose. Design an observability strategy across the whole estate.
What they are really testing: Whether you can go beyond "add more dashboards" to a coherent strategy across logs, metrics, and traces, tied to SLOs and alerting that pages on symptoms rather than causes. They want the diagnostic workflow, not a tool list.
A real interview question
You own a system of many services and an incident took too long to diagnose. Design an observability strategy across the whole estate.
What most people say
drag me
“We will add CloudWatch dashboards and set alerts on CPU and memory for every service.”
Dashboards plus resource-level alerts give you a wall of graphs and a flood of pages that do not tell you whether users are affected. There is no tracing to follow a request across services, no correlation, and no SLO, so the next incident is just as slow to diagnose.
The follow-ups they ask next
Your team is drowning in alerts and starting to ignore the pager. How do you fix it?
Alert fatigue usually comes from cause-based, per-resource alerts. Move to symptom/SLO-based alerting that pages only when users are actually affected, route the rest to dashboards or tickets, and tune thresholds against real burn rate. Fewer, higher-signal pages restore trust in the pager.
Tracing every request is blowing the budget. What do you do?
Sample. Tail-based sampling keeps the interesting traces (errors, high latency) and drops a fraction of the healthy ones, so you keep diagnostic value at a fraction of the cost. The trade-off to state is that you lose full fidelity on the boring majority, which is usually an acceptable price.
What the interviewer is listening for
- Frames observability around diagnostic questions
- Uses metrics, logs, and traces for their distinct jobs
- Correlates via a propagated trace id
- Alerts on SLOs/symptoms and names the cost trade-off of sampling
What sinks the answer
- Equates observability with more dashboards
- No distributed tracing across services
- Alerts only on resource metrics like CPU
- Ignores the cost of telemetry and tracing
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“I anchor on [is it broken, where, why]. I use [metrics for trends, logs for detail, traces to follow a request], glued by [a trace id propagated and stamped everywhere]. I define [SLOs] and alert on [symptoms/SLO burn, not CPU]. The trade-off: telemetry costs [cardinality + retention], so I [sample, keeping errors/slow requests at full fidelity].”
Keep going with architecture & design
Junior
How does putting a load balancer in front of your app improve availability, and what does it NOT solve?
Junior
What is autoscaling, and how does it decide when to add or remove instances?
Senior
Design a highly available web application on AWS. Walk me through the architecture and your trade-offs.
Senior
Design a disaster recovery strategy for a revenue-critical workload on AWS. How do you pick between backup-and-restore, pilot light, warm standby, and active-active?
Senior
Design a scalable async processing pipeline for spiky, long-running work (think video transcoding or report generation). Walk me through the queue-based architecture and its failure modes.
Senior
Design a CI/CD pipeline that lets a team deploy many times a day safely. How do you make production changes low-risk with progressive delivery?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free