Design a system that must never lose a message. It ingests payment notifications from a partner and each one must be processed exactly once.
What they are really testing: Whether you know that exactly once does not exist end to end, and whether you can build at least once delivery plus idempotency instead of promising magic.
A real interview question
Design a system that must never lose a message. It ingests payment notifications from a partner and each one must be processed exactly once.
What most people say
drag me
“I would use Kafka with exactly once semantics enabled, so messages cannot be lost or duplicated.”
It hands the whole problem to a config flag. Kafka's exactly once applies within Kafka, not across a partner HTTP call and a database write, and an interviewer will push straight into that gap.
The follow-ups they ask next
Your consumer crashes right after the database commit but before the ack. What happens?
The message is redelivered. Say so calmly and point at the unique key on the message id, which makes the replay a no-op.
How do you prove you have not lost anything?
Reconciliation. Compare the partner's daily count against your processed count and alert on any gap, rather than trusting the broker.
The dead letter queue has 400 messages. What is your process?
Triage by error class, fix the bug, then replay from the DLQ. This requires the payloads to be stored raw, which is why you kept them.
What the interviewer is listening for
- Says out loud that exactly once end to end is not achievable
- Acknowledges only after a durable commit, in both the producer and the consumer
- Uses a business key for idempotency at the sink
- Builds a reconciliation check to prove no loss rather than assuming it
What sinks the answer
- Treats exactly once as a checkbox on the broker
- Acknowledges the message before the work is durably done
- Has a dead letter queue that nobody watches
- Cannot say what losing one message actually costs
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“I have not designed a zero loss pipeline myself, but here is how I would reason about it. I would walk every handoff where a message could disappear and ask what happens if the process dies right there.”
Keep going with system design
Mid
Design a log and metrics ingestion pipeline for a platform with about 200 services. Engineers need to search logs and alert on metrics.
Mid
Design an image upload and processing service for a marketplace app. Users upload photos from their phone, and we need thumbnails and a moderation check before the listing goes live. Walk me through it.
Senior
Design an event-driven order processing system for an e-commerce site. Payment, inventory, and shipping are separate services.
Senior
Design an analytics pipeline that produces a daily report for the leadership team by 8am. The data comes from a production database and a stream of app events.
Senior
We have about 50 product teams and every one of them has rolled their own pipeline. Design a CI/CD platform that all 50 can use. Take me through it.
Senior
We are moving from one shared cloud account to a proper multi-account setup. Design the landing zone.
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free