Design a scalable async processing pipeline for spiky, long-running work (think video transcoding or report generation). Walk me through the queue-based architecture and its failure modes.

Question

Accepted Answer

First I clarify the shape: how spiky the load is, whether a job takes 10 seconds or 40 minutes, whether ordering matters, and whether the caller needs the result synchronously. Then I decouple with a queue. The producer enqueues a job and returns a job id in under 100 milliseconds, and a worker fleet pulls from the queue and autoscales on queue depth, say a target of 10 messages per worker, so a spike to 50,000 jobs piles up harmlessly in the queue instead of timing out the API. The part that matters most is failure handling. Delivery is at-least-once, so workers must be idempotent via an idempotency key, otherwise a redelivered job transcodes the same video twice. I set the visibility timeout longer than the worst realistic job, so if p99 runtime is 8 minutes I set it around 12, and a crashed worker releases its job automatically. I add a dead-letter queue after 5 failed receives so one poison message cannot wedge the fleet, and I alarm on DLQ depth above 0 and on oldest-message age crossing something like 15 minutes, which is the real signal that consumers cannot keep up. Trade-offs I would name: I default to at-least-once plus idempotency rather than chasing exactly-once, because true exactly-once is expensive and rarely needed, and I only use FIFO ordering if the work genuinely requires it, since strict ordering caps throughput hard. The caller gets the result by polling the job id or via a push notification.

Answer

One malformed job keeps crashing your workers and they keep retrying it. What happens to the rest of the queue?

Without a dead-letter queue, the poison message is redelivered forever and can starve or wedge the consumer fleet. The fix is a maxReceiveCount that routes the message to a DLQ after N attempts, plus an alarm on DLQ depth so a human investigates the bad job instead of it silently burning capacity.

Answer

Why not just chase exactly-once delivery and skip the idempotency work?

True exactly-once across a distributed boundary is very hard and usually means at-least-once delivery plus idempotent consumers under the hood anyway. Building idempotent workers is cheaper, more robust, and protects you even when the broker redelivers, so the senior move is at-least-once plus idempotency, not a quest for exactly-once.

Design a scalable async processing pipeline for spiky, long-running work (think video transcoding or report generation). Walk me through the queue-based architecture and its failure modes.

Design a scalable async processing pipeline for spiky, long-running work (think video transcoding or report generation). Walk me through the queue-based architecture and its failure modes.

The follow-ups they ask next

What the interviewer is listening for

What sinks the answer

If you genuinely do not know

Keep going with architecture & design

Knowing the answer is not the same as recalling it under pressure