Apache Kafka and Event Streaming, From First Principles

Why everyone reaches for Kafka eventually

You start with one service that writes to a database. Then analytics wants a copy of every order. Then search wants to index those orders. Then billing, then the fraud team, then a data lake. Soon your one service is making six synchronous calls on every write, and when one of those six is down, the order fails. You added coupling you never wanted.

Kafka flips this around. The service writes one event to a log, and says nothing about who reads it. Analytics, search, billing, and fraud each read that same log at their own pace. Add a seventh consumer next year and nobody changes the producer. That decoupling, plus the ability to replay history, is the whole pitch.

The core idea: a log, not a mailbox

Kafka is a distributed, append-only commit log: writers append records to the end, readers track their own position, and nothing is deleted just because someone read it.

That single sentence is the whole system. Every confusing Kafka behavior makes sense once you internalize that it is a log you read by position, not a mailbox you drain by removing messages.

The newspaper image fixes the most common beginner mistake: assuming a message 'disappears' after you read it. In Kafka it does not. Ten teams can read the same record; each just remembers where they are.

Topics, partitions, and offsets

A topic is a named log, like orders. But a single log on a single machine cannot scale, so Kafka splits each topic into partitions, independent ordered logs that can live on different brokers. A topic with 12 partitions is really 12 little logs that together form orders.

Within one partition, every record has a monotonically increasing offset (0, 1, 2, 3...). The offset is the record's address. A consumer's progress is literally just 'I have processed orders-3 up to offset 41,209.' Commit that number and you can crash, restart, and resume exactly where you left off.

• Topic, the logical stream (orders, payments, clicks).
• Partition, one ordered shard of the topic; the unit of parallelism and ordering.
• Offset, a record's position within a partition; how consumers track progress.
• Broker, a Kafka server hosting some partitions; a cluster is many brokers.
• Replica, a copy of a partition on another broker for fault tolerance.

The picture: producers, partitions, consumer groups

1. 1

   Producer picks a partition

   If the record has a key, Kafka hashes it: hash(key) % partitionCount. Same key always lands on the same partition, which preserves order for that key.

2. 2

   Broker appends and assigns an offset

   The record is written to the end of that partition's log and replicated to follower brokers before it is considered committed.

3. 3

   Consumer group rebalances

   Kafka assigns each partition to exactly one consumer in the group. Group 'billing' has 2 consumers for 3 partitions, so one consumer owns 2.

4. 4

   Consumers read and commit offsets

   Each consumer pulls batches from its partitions, processes them, then commits the offset it reached. On restart it resumes from the last commit.

5. 5

   Other groups read the same data independently

   Group 'search' has its own offsets, so it can be hours behind or replay from zero without affecting 'billing'.

The load-bearing rule: parallelism is capped by partition count. A group can never have more *active* consumers than the topic has partitions, extra consumers sit idle. Want to process orders with 20 workers? You need at least 20 partitions.

Kafka log vs a traditional queue

Both move messages between services, but the semantics differ in ways that decide your architecture. The table below is the comparison that actually matters when you choose.

Code: a keyed producer and a consuming group

Here is a producer publishing order events with customerId as the key. The key is the whole game: it pins every event for one customer to one partition, so they are processed in order.

import { Kafka, Partitioners } from 'kafkajs';

const kafka = new Kafka({
  clientId: 'order-service',
  brokers: ['broker-1:9092', 'broker-2:9092'],
});

const producer = kafka.producer({
  // Turn on the idempotent producer: no duplicate appends on retry.
  idempotent: true,
  maxInFlightRequests: 1,
  createPartitioner: Partitioners.DefaultPartitioner,
});

export async function publishOrder(order: {
  id: string;
  customerId: string;
  totalCents: number;
}) {
  await producer.connect();
  await producer.send({
    topic: 'orders',
    // acks: -1 means wait for all in-sync replicas (durable).
    acks: -1,
    messages: [
      {
        // KEY controls the partition -> ordering per customer.
        key: order.customerId,
        value: JSON.stringify(order),
        headers: { eventType: 'OrderPlaced' },
      },
    ],
  });
}

Now the consumer side. Note fromBeginning (replay vs only-new), the per-message processing, and, critically, that we commit offsets after the work succeeds, not before.

import { Kafka } from 'kafkajs';

const kafka = new Kafka({ clientId: 'billing', brokers: ['broker-1:9092'] });

// groupId defines the consumer group. Run this process N times and the
// N instances split the partitions between them automatically.
const consumer = kafka.consumer({ groupId: 'billing-service' });

async function run() {
  await consumer.connect();
  await consumer.subscribe({ topic: 'orders', fromBeginning: false });

  await consumer.run({
    // Disable auto-commit so WE decide when an offset is safe to record.
    autoCommit: false,
    eachMessage: async ({ topic, partition, message }) => {
      const order = JSON.parse(message.value!.toString());

      // 1) Do the real work first. If this throws, we do NOT commit,
      //    so the message is redelivered after restart (at-least-once).
      await chargeCustomer(order);

      // 2) Only now record progress: last processed offset + 1.
      await consumer.commitOffsets([
        { topic, partition, offset: (Number(message.offset) + 1).toString() },
      ]);
    },
  });
}

async function chargeCustomer(order: { id: string; totalCents: number }) {
  // Make this idempotent (keyed by order.id) so a redelivery is harmless.
}

run().catch((err) => {
  console.error('consumer crashed', err);
  process.exit(1);
});

Delivery semantics: at-least-once to exactly-once

There are three delivery guarantees, and Kafka can give you any of them depending on how you configure producers, consumers, and commits.

• At-most-once, commit the offset *before* processing. If you crash mid-work, the message is skipped. Fast, lossy. Rarely what you want.
• At-least-once, process *then* commit (the default sane choice, shown above). A crash between work and commit causes redelivery, so consumers must be idempotent.
• Exactly-once (effectively-once), no duplicates and no loss, achieved with idempotent producers plus transactions.

True exactly-once across a network is impossible, but Kafka gets you the practical equivalent. The idempotent producer (idempotent: true) tags each record with a producer id and sequence number, so a retried send is deduplicated by the broker, no double appends. Transactions then let you read, process, and write-plus-commit-offset as one atomic unit (the read-process-write loop), so a stream processor never emits a partial result.

const producer = kafka.producer({
  transactionalId: 'enrich-orders-v1', // stable id ties the transaction state
  idempotent: true,
});
await producer.connect();

// Read from 'orders', write to 'orders-enriched', and commit the source
// offset all-or-nothing. Downstream sees the output only if the offset
// commit also succeeds.
const txn = await producer.transaction();
try {
  await txn.send({ topic: 'orders-enriched', messages: enriched });
  await txn.sendOffsets({
    consumerGroupId: 'enricher',
    topics: offsetsToCommit,
  });
  await txn.commit();
} catch (err) {
  await txn.abort(); // nothing downstream sees a partial write
  throw err;
}

Retention and compaction: how long the log lives

Because consuming does not delete, something else has to reclaim disk. Kafka offers two cleanup policies, and choosing the right one per topic is part of the design.

• Delete (time/size retention), keep records for retention.ms (e.g. 7 days) or up to retention.bytes, then drop the oldest segments. Right for event streams where old events stop mattering.
• Compact, keep at least the *latest* value for every key, garbage-collecting superseded versions. The log becomes a changelog / snapshot you can replay to reconstruct current state. Right for 'latest known config per device' or materializing a table.

# An event stream: keep 7 days, then expire.
kafka-topics.sh --create --topic orders \
  --partitions 12 --replication-factor 3 \
  --config retention.ms=604800000 \
  --config cleanup.policy=delete

# A compacted changelog: keep the latest value per key forever.
kafka-topics.sh --create --topic device-config \
  --partitions 6 --replication-factor 3 \
  --config cleanup.policy=compact \
  --config min.cleanable.dirty.ratio=0.2

Consumer lag: the one metric to watch

If you monitor exactly one thing about a Kafka pipeline, monitor consumer lag: the difference between the latest offset written to a partition (the log-end offset) and the offset a consumer group has committed. Lag of 0 means caught up. Lag that climbs steadily means consumers cannot keep up with producers, and your data is getting staler by the second.

kafka-consumer-groups.sh --bootstrap-server broker-1:9092 \
  --describe --group billing-service

# GROUP           TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
# billing-service orders  0          41209           41212           3
# billing-service orders  1          38880           52001           13121   <-- hot/slow partition
# billing-service orders  2          50410           50410           0

Alert on lag, not on throughput. Throughput looks healthy right up until the moment you fall behind; lag tells you the truth about whether consumers are winning the race against producers.

Kafka vs SQS / RabbitMQ: when to choose which

Kafka is powerful and operationally heavy. Most teams do not need it on day one. Choose by what the workload actually demands.

• Reach for SQS / RabbitMQ when: you have a task queue (resize this image, send this email), each message has exactly one owner, you do not need replay, and you want a managed, near-zero-ops broker. SQS especially is the boring correct default for offloading work.
• Reach for Kafka when: multiple independent consumers need the same stream (fan-out), you need to replay history (reprocess after a bug, bootstrap a new service from the past), you need per-key ordering at high throughput, or you are doing real stream processing / event sourcing.
• Throughput: Kafka sustains millions of messages/sec on modest clusters because it is sequential disk I/O; queues are fine into the tens of thousands but rarely the same order of magnitude.
• Ops cost: managed Kafka (MSK, Confluent Cloud) reduces but does not erase the burden of partitions, retention, and consumer groups. SQS has essentially none.
• Honest default: if you cannot name a consumer that needs replay or a second team that needs the same events, you probably want a queue. Revisit when fan-out or reprocessing shows up.

Stream processing in one breath

Once events flow through Kafka, you often want to transform, join, and aggregate them continuously, not in nightly batches. That is stream processing: a long-running program that reads one or more topics, maintains state, and writes derived topics.

The mental model: a stream is an unbounded table of changes, and a table is the latest snapshot of a stream. Counting orders per minute, joining clicks to users, or detecting three failed logins in 60 seconds are all stream jobs. Frameworks (Kafka Streams, Flink, ksqlDB) give you windowing, exactly-once via the transactional read-process-write loop, and state stores backed by compacted changelog topics.

StreamsBuilder builder = new StreamsBuilder();

builder.stream("orders", Consumed.with(Serdes.String(), orderSerde))
    // group by key (customerId already keys the records)
    .groupByKey()
    // a 1-minute tumbling window of activity per customer
    .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(1)))
    .count()
    .toStream()
    .map((windowedKey, count) ->
        KeyValue.pair(windowedKey.key(), count))
    // derived topic other services can consume
    .to("orders-per-minute", Produced.with(Serdes.String(), Serdes.Long()));

You do not need a framework to start, a plain consumer that aggregates in memory and writes a result topic is real stream processing. Reach for Kafka Streams or Flink when you need fault-tolerant state, exactly-once, and windowing without hand-rolling them.

Common mistakes that cost hours

1. Too few partitions. You cannot scale a group beyond its partition count, and increasing partitions later rehashes keys and breaks existing per-key ordering. Provision generously up front (2-3x peak consumers).
2. Hot keys. Keying everything by a low-cardinality value (country, region, a giant tenant) jams most traffic onto one partition. One consumer melts while the rest idle. Key by something high-cardinality (customerId, orderId).
3. Auto-commit before processing. With autoCommit: true, kafkajs commits on a timer regardless of whether your handler finished. A crash then silently skips messages (at-most-once by accident). Commit after the work succeeds.
4. Assuming global ordering. Order holds only within a partition. Cross-partition events have no guaranteed order, design so it only matters per key.
5. Non-idempotent consumers. At-least-once means redelivery happens. If your handler charges a card or sends an email without an idempotency key, retries double-bill. Make side effects idempotent.
6. RF=1 in production. One broker dies and that partition's data is gone. Use replication-factor 3 with min.insync.replicas=2.

Takeaways

Where to go next

Kafka is one piece of a broader async toolkit. Build the surrounding mental models so you reach for the right tool, and design consumers that survive redelivery.

• async processing, queues, and workers, the foundation: why you push work off the request path at all.
• event-driven architecture on the cloud, how producers, brokers, and consumers compose into a system.
• idempotency and exactly-once, make at-least-once safe, which every Kafka consumer needs.
• Practice the surrounding ops in the networking lab and Docker lab to run a local broker and producer/consumer pair.

Start small: one topic, a sensible partition count, a keyed producer, and a consumer group that commits after it works. Add transactions and stream processing only when a real requirement asks for them. The log will be waiting either way.