Design a log and metrics ingestion pipeline for a platform with about 200 services. Engineers need to search logs and alert on metrics.
What they are really testing: Whether you separate the two very different workloads, logs and metrics, instead of dumping both into one expensive store.
A real interview question
Design a log and metrics ingestion pipeline for a platform with about 200 services. Engineers need to search logs and alert on metrics.
What most people say
drag me
“I would send everything to Elasticsearch and build Kibana dashboards on top of it.”
It names one product instead of sizing the load, and it treats metrics and logs as one problem, which is exactly how teams end up with a storage bill nobody can explain.
The follow-ups they ask next
Log volume triples overnight because someone turned on debug logging. What breaks first?
Name the ingestion buffer filling and the index tier throttling. Talk about per service rate limits and sampling as the control.
Why not just use CloudWatch Logs for all of it?
Say you would for a small estate. At 2 terabytes a day the per gigabyte ingestion charge dominates, so give the rough math rather than a taste opinion.
How do you keep the metrics path alive when the logs path is on fire?
Separate agents, separate transport, separate quotas. Alerting must never depend on the log store.
What the interviewer is listening for
- Sizes the load in gigabytes per day before naming a single service
- Separates metrics from logs and gives each a different retention
- Uses a local agent buffer as the anti data loss mechanism
- Names the hot versus cold storage trade-off in cost terms
What sinks the answer
- Puts everything in one search cluster and never mentions cost
- Cannot say roughly how much data the system handles
- Assumes the ingestion path never fails
- Lists five AWS services without connecting them into a flow
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“I have not run a pipeline at this size, but here is how I would reason about it. First I would ask how many gigabytes a day and how far back people actually search, because that decides almost everything else.”
Keep going with system design
Mid
Design an image upload and processing service for a marketplace app. Users upload photos from their phone, and we need thumbnails and a moderation check before the listing goes live. Walk me through it.
Senior
Design an event-driven order processing system for an e-commerce site. Payment, inventory, and shipping are separate services.
Senior
Design an analytics pipeline that produces a daily report for the leadership team by 8am. The data comes from a production database and a stream of app events.
Senior
Design a system that must never lose a message. It ingests payment notifications from a partner and each one must be processed exactly once.
Senior
We have about 50 product teams and every one of them has rolled their own pipeline. Design a CI/CD platform that all 50 can use. Take me through it.
Senior
We are moving from one shared cloud account to a proper multi-account setup. Design the landing zone.
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free