Design an observability stack for a system where nobody can currently answer why a request was slow.
What they are really testing: Whether you can connect the three signals into an investigation workflow. Listing metrics, logs and traces is table stakes; connecting them with correlation IDs is the actual answer.
A real interview question
Design an observability stack for a system where nobody can currently answer why a request was slow.
What most people say
drag me
“Install Prometheus, Grafana, and the ELK stack.”
It is a shopping list, not a design. It leaves out tracing, which is the only signal that answers the question asked, and says nothing about how an engineer moves from an alert to a root cause.
The follow-ups they ask next
Sampling at 2% means you miss the slow request someone complains about. How do you handle that?
Tail-based sampling: buffer the spans and decide after the request completes, keeping all errors and anything over a latency threshold while sampling the boring successes.
What makes metrics expensive unexpectedly?
Cardinality. A label with unbounded values like user ID or request path with IDs multiplies time series, and cost scales with series count rather than request volume.
What the interviewer is listening for
- Starts from the unanswered question
- Correlates logs to traces via trace ID
- Raises sampling and cardinality cost
What sinks the answer
- Lists tools only
- Omits tracing entirely
- No cost awareness
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“The gap is [tracing], so [propagate a trace ID from the edge across services]. Metrics [detect and alert on user-visible symptoms], traces [localise where the time went], logs [carry the trace ID so you jump straight to the right lines]. Control cost with [tail sampling that keeps errors and slow requests, plus retention tiers and cardinality limits].”
Keep going with system design
Senior
Design a CI/CD system for 15 microservices owned by 4 teams deploying several times a day.
Senior
Design secret management for 40 services across 3 environments, with an audit requirement.
Senior
Design a system giving every pull request its own preview environment, for a team of 30.
Principal
Design multi-region failover for a system that currently runs in one region, targeting 15 minutes recovery.
Senior
Leadership asks you to prove the platform investment is working. What do you measure?
Senior
You inherit 200 Jenkins jobs with no documentation and are asked to migrate to a modern CI system. How do you approach it?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free