Event-Driven Architecture on the Cloud

When one slow call takes down everything

You ship an order service. It takes a payment, then calls the email service, then the analytics service, then the warehouse service, one after another, in the same request. It works great in the demo. Then the email provider has a bad afternoon, its API hangs for 30 seconds, and suddenly customers can't place orders at all. Nobody is buying email. They're buying products. But because checkout calls email *synchronously*, a hiccup in a non-critical service became an outage in your most critical one.

The fix isn't a faster email service. It's changing the *shape* of the conversation. Instead of checkout calling everyone and waiting, checkout announces "an order was placed" and walks away. Whoever cares, email, analytics, the warehouse, reacts on their own time. That announcement is an event, and designing around it is event-driven architecture.

The mental model: a newsroom, not a phone tree

Event-driven architecture means services communicate by emitting and reacting to events, facts about what happened, instead of directly commanding each other.

Synchronous calls are a phone tree. To tell five teams something, you phone each one, wait for them to pick up, and you're stuck on the line until the last call ends. If one person doesn't answer, you're frozen mid-tree.

Event-driven is a newsroom. A reporter publishes a story. They don't know, or care, who reads it. Subscribers (the sports desk, the weather team, an archive bot) each pick it up and do their own thing, at their own pace. The reporter's job is done the moment the story is filed. Add a tenth subscriber tomorrow and the reporter's code never changes.

The picture: one event, many reactions

Here's the order flow rebuilt around an event. Checkout publishes once to a topic; the topic fans the event out to every interested consumer. Checkout finishes in milliseconds and is completely unaffected if a downstream consumer is slow or down.

1. 1

   Customer clicks Buy

   Checkout charges the card and persists the order. That part is still synchronous, it must succeed before we promise anything.

2. 2

   Checkout publishes one event

   It sends a single OrderPlaced message to the topic with the order id and details, then returns 200 to the customer. Total added latency: a few milliseconds.

3. 3

   The topic fans out

   The messaging service delivers a copy to every subscriber, email, warehouse, analytics. Checkout has no idea who they are.

4. 4

   Consumers react independently

   Email sends a receipt. The warehouse reserves stock. Analytics records the sale. If email is down, the warehouse and analytics are unaffected, and email retries later.

Queues vs pub/sub vs streams

"Messaging" is three different patterns wearing the same coat. Picking the wrong one is the most common early mistake, so anchor on these distinctions before you reach for a service.

• Queue, one message, one worker. The message is consumed and gone. Use it to *distribute work*: ten workers pull from one queue and each grabs different items.
• Pub/Sub, one message, every subscriber gets a copy. Use it to *broadcast a fact* so multiple systems can react (this is fan-out).
• Stream, an append-only log you can replay. Messages aren't deleted on read; consumers track their own position. Use it for *ordered history, replay, and analytics*.

Publishing and consuming an event

Concretely, publishing is a one-liner and consuming is a small loop. Here's the SNS → SQS fan-out from the diagram, in Python with boto3. First, checkout publishes the event:

import json
import boto3

sns = boto3.client("sns")
TOPIC_ARN = "arn:aws:sns:eu-west-1:123456789012:OrderPlaced"

def publish_order_placed(order_id: str, total: float, email: str) -> None:
    event = {
        "type": "OrderPlaced",
        "order_id": order_id,
        "total": total,
        "customer_email": email,
    }
    sns.publish(
        TopicArn=TOPIC_ARN,
        Message=json.dumps(event),
        # idempotency: a stable id lets consumers dedupe
        MessageAttributes={
            "event_id": {"DataType": "String", "StringValue": order_id},
        },
    )
    # checkout returns to the customer right here, no waiting on consumers

Each consumer owns an SQS queue subscribed to that topic. The email worker just polls its queue, does its job, and deletes the message to acknowledge it:

import json
import boto3

sqs = boto3.client("sqs")
QUEUE_URL = "https://sqs.eu-west-1.amazonaws.com/123456789012/email-queue"

while True:
    resp = sqs.receive_message(
        QueueUrl=QUEUE_URL,
        MaxNumberOfMessages=10,
        WaitTimeSeconds=20,  # long polling, cheaper, less spin
    )
    for msg in resp.get("Messages", []):
        envelope = json.loads(msg["Body"])        # SNS wraps the payload
        event = json.loads(envelope["Message"])   # our actual event

        if not already_processed(event["order_id"]):
            send_receipt(event["customer_email"], event["order_id"])
            mark_processed(event["order_id"])

        # delete = acknowledge. Only do this AFTER the work succeeded.
        sqs.delete_message(QueueUrl=QUEUE_URL, ReceiptHandle=msg["ReceiptHandle"])

Notice the already_processed / mark_processed guard, and that we delete the message only *after* the work succeeds. That's not optional decoration, it's the heart of running this safely. Here's why.

Delivery guarantees: why idempotency is non-negotiable

Almost every managed messaging service gives you at-least-once delivery. Read that carefully: *at least* once. Not exactly once. The same event can, and eventually will, be delivered to your consumer more than once. This isn't a bug; it's the honest trade-off that makes the system reliable.

It happens for boring, unavoidable reasons. A consumer processes a message, but its acknowledgement gets lost on the network. The broker never hears "done," so after a visibility timeout it redelivers, and now you've sent two receipt emails for one order. The number of guarantees in plain terms:

• At-most-once, fire and forget. Fast, but you can silently lose messages. Rarely what you want.
• At-least-once, never lost, sometimes duplicated. The realistic default for SQS, SNS, Pub/Sub, and Kinesis.
• Exactly-once, the dream. A few services offer it in narrow conditions (Kafka transactions, SQS FIFO dedup windows), but it's limited and costs throughput. Don't architect around assuming it.

The same instinct covers the *other* failure: a message your consumer can never process (bad data, a permanent bug). Without a backstop it gets redelivered forever, a "poison pill." Configure a dead-letter queue (DLQ) so a message that fails N times moves aside for inspection instead of blocking the line.

Common mistakes that cost hours

1. Treating at-least-once as exactly-once. No dedup guard means double-charged cards and duplicate emails the first time a network blip causes a redelivery. Make consumers idempotent from day one.
2. Acknowledging before the work is done. Delete or ack the message only *after* processing succeeds. Ack first and crash, and the event is gone forever.
3. No dead-letter queue. A single malformed message retries endlessly, drowns your logs, and can stall the whole queue. Always wire a DLQ with a sane retry count.
4. Putting commands in events. An event states a fact (OrderPlaced), not an instruction (SendEmail). If the producer is telling a specific consumer what to do, you've just rebuilt a synchronous call with extra steps and lost the decoupling.
5. Reaching for a stream when you needed a queue. Kafka/Kinesis are powerful and operationally heavy. If you just need background jobs, a plain queue (SQS) is simpler, cheaper, and enough.
6. Forgetting ordering isn't free. Standard queues and pub/sub don't guarantee order. If OrderShipped can arrive before OrderPlaced, either use a FIFO/partitioned option or make consumers tolerant of out-of-order events.

When event-driven beats a synchronous call

Event-driven isn't free, you trade the simplicity of a function call for eventual consistency, harder debugging, and new infrastructure. So don't make *everything* an event. Reach for it when the trade pays off:

• Multiple consumers care about the same thing, fan-out beats calling each one yourself.
• The reaction can happen later, receipts, analytics, and indexing don't need to finish before you answer the user.
• You want failure isolation, a down consumer shouldn't take down the producer.
• Load is spiky, a queue absorbs bursts and lets workers drain at their own pace.

Keep it synchronous when the caller genuinely needs the answer *now* to continue, reading a user's profile to render a page, or checking inventory before confirming a price. A request that can't proceed without the response shouldn't be fire-and-forget.

Where to go next

You now have the vocabulary and the mental model. The fastest way to make it stick is to wire a real producer and consumer, then design a system where events change how it scales.

• Go deeper on the worker side: Async Processing: Queues & Workers covers retries, visibility timeouts, and worker pools in detail.
• See how decoupling shapes growth: Scalability Principles shows why queues and fan-out are the backbone of systems that scale.
• Ready to build the rest of the stack? The Cloud Engineer path walks you from networking to compute to event-driven systems, level by level.