Reliability of Stateful & Data Systems

The stateless services failed over fine. The database didn't.

It's 3am. An availability zone drops. Your load balancer notices in seconds and pulls the bad targets. Your stateless API pods reschedule onto healthy nodes, pass their health checks, and start serving traffic again. The dashboards go green. You almost go back to sleep.

Then the error rate spikes again. The API is up, but every write is failing. The primary database lived in the zone that just died. The replica is there, ready, but nothing promoted it, because nobody had tested the failover in eight months and the automation referenced a security group that no longer existed. You spend the next 40 minutes promoting a replica by hand, praying its replication lag was small, and wondering exactly how many seconds of writes you just lost.

This is the recurring lesson of on-call: stateless tiers heal themselves; stateful tiers do not. A web server has no memory, so any copy is as good as any other. A database *is* the memory. You can't just spin up another one, it has to be the right one, with the right data, promoted in the right order, and the cost of getting that wrong is measured in lost transactions, not lost requests.

Stateless is easy. State is where reliability is won or lost.

Stateless services are easy to make reliable, any instance is interchangeable. State is hard, because every copy is a claim about the truth, and reliability is the discipline of keeping those claims in agreement while machines fail underneath them.

Reliability for stateless workloads is a numbers game: run more replicas, spread them across zones, let the orchestrator replace dead ones. There is no "wrong" instance. Reliability for stateful workloads is a *truth* game. The instant you have two copies of data, you have a question, which one is authoritative?, and every failure mode is some version of that question being answered wrong: a stale replica promoted, two nodes both believing they're the primary, a backup that restores to a moment that never actually existed.

The picture: primary, replicas, failover, and backups

A reliable data store has two completely separate safety systems, and conflating them is the root of most data-loss incidents. Replication keeps live copies for availability, if the primary dies, a replica takes over fast. Backups + point-in-time recovery (PITR) keep independent historical copies for durability, if data is corrupted, deleted, or a bad migration runs, you can rewind to a moment before the damage. Replication protects against a machine dying. Backups protect against the *truth itself* being wrong.

1. 1

   The application reads and writes the primary

   All writes go to exactly one node, the primary, the single source of truth. Reads may be routed to replicas to spread load, but every write funnels through one place so there's an unambiguous order of events.

2. 2

   The primary streams its log to replicas

   Every change is recorded in the write-ahead log (WAL). Replicas replay that stream to stay current. A synchronous replica acknowledges before the primary confirms the write (zero loss, higher latency); an async replica trails behind (lower latency, a small window of loss).

3. 3

   The same log is archived to durable storage

   The WAL is also shipped to object storage and paired with a periodic base backup. Together they let you replay the database forward to any second in time, that's PITR. This path is independent of the replicas.

4. 4

   A controller watches health and promotes on failure

   When the primary stops responding, the failover controller picks the most up-to-date replica, promotes it to primary, and repoints the application. The whole game is doing this quickly, exactly once, and never to a stale node.

Steps to make a data store reliable

There's a natural order here. Each step assumes the one before it actually works, and "actually works" means tested, not configured.

1. 1

   Define your RPO and RTO first

   RPO (recovery point objective) is how much data you can afford to lose; RTO (recovery time objective) is how long you can be down. "Zero" for either is extremely expensive, pick honest numbers per dataset. Everything below is just the machinery to hit them. See [Disaster Recovery: RTO, RPO & DR Strategies](/blog/disaster-recovery-rto-rpo).

2. 2

   Add at least one replica in another failure domain

   Another availability zone at minimum. A replica in the same rack as the primary protects against a disk failure but not a zone outage. Match the replica's resources to the primary, an undersized replica falls behind and is useless as a failover target.

3. 3

   Set up backups + PITR, separate from replication

   A base backup plus continuous WAL archiving to storage that is isolated from the primary's account/region. A replica that faithfully copies a `DROP TABLE` is not a backup.

4. 4

   Automate failover, with a guard against split-brain

   Use a consensus-based controller (Patroni, the managed provider's HA, or built-in like RDS Multi-AZ) that fences the old primary so it can't keep accepting writes after a new one is elected.

5. 5

   Test failover and restore on a schedule

   Run a real failover and a real restore-to-PITR in a non-prod environment regularly. An untested backup is a hope, not a recovery plan. Measure the actual RTO/RPO you achieve and compare it to your target.

6. 6

   Watch replication lag as a first-class signal

   Replica lag is the size of your data-loss window if the primary dies *right now*. Alert on it. Lag silently growing is one of the most common ways a "safe" setup turns into a 30-second data loss during failover.

Replication vs backup vs snapshot, they protect against different things

These three are constantly confused, and the confusion is what causes data loss. They are not redundant; you need all three. The key question for any copy is: *what failure does it protect against, and how much data does it lose?*

Verify it: failover health and backup restore

Two things must be continuously true: the failover target is healthy and not lagging, and the backups can actually be restored. Here's a health check a controller can poll to decide whether a replica is safe to promote, it refuses any replica whose lag exceeds your RPO.

#!/usr/bin/env bash
# Decide whether a Postgres replica is safe to promote.
# Exit 0 = healthy & within RPO; non-zero = do NOT promote.
set -euo pipefail

REPLICA_HOST="${1:?usage: $0 <replica-host>}"
MAX_LAG_SECONDS="${MAX_LAG_SECONDS:-5}"   # your RPO budget

# 1. Is it actually in recovery (a real replica, not a stale ex-primary)?
in_recovery=$(psql -h "$REPLICA_HOST" -tAc "SELECT pg_is_in_recovery();")
if [ "$in_recovery" != "t" ]; then
  echo "FAIL: $REPLICA_HOST is not a replica (split-brain risk)"; exit 2
fi

# 2. How far behind is it, in seconds of wall-clock time?
lag=$(psql -h "$REPLICA_HOST" -tAc \
  "SELECT COALESCE(EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp())), 1e9);")
lag=${lag%.*}

if [ "$lag" -gt "$MAX_LAG_SECONDS" ]; then
  echo "FAIL: lag ${lag}s exceeds RPO ${MAX_LAG_SECONDS}s, promoting loses data"; exit 3
fi

echo "OK: $REPLICA_HOST in recovery, lag ${lag}s (<= ${MAX_LAG_SECONDS}s), safe to promote"

Backups are worthless until you've restored one. Wire a restore drill into a scheduled job that restores to a throwaway instance, runs an integrity query, and fails loudly if the data isn't there. This is the SQL the drill runs after restoring to a target point in time:

-- Run against a database freshly restored from backup + PITR.
-- Fails loudly if the restore is empty, stale, or structurally broken.

-- 1. Core tables exist and are non-empty.
DO $$
DECLARE n bigint;
BEGIN
  SELECT count(*) INTO n FROM orders;
  IF n = 0 THEN RAISE EXCEPTION 'restore check failed: orders table is empty'; END IF;
END $$;

-- 2. The restore reached the expected point in time (data is fresh, not days old).
SELECT
  max(created_at)                         AS newest_row,
  now() - max(created_at)                 AS staleness
FROM orders
HAVING now() - max(created_at) < interval '24 hours';
-- If this returns zero rows, the restore is too old, alert.

-- 3. Referential integrity survived the restore (no orphaned children).
SELECT count(*) AS orphaned_line_items
FROM order_items oi
LEFT JOIN orders o ON o.id = oi.order_id
WHERE o.id IS NULL;
-- Expect 0. Anything else means a torn / partial restore.

Consistency under failover, and the split-brain trap

Failover is where consistency guarantees go to die. The promotion itself is a distributed-systems problem: you're changing the source of truth while machines may be partitioned, slow, or briefly back from the dead. Two failure modes dominate.

Lost writes (the async gap)

If you promote an async replica, every transaction the primary committed but hadn't yet streamed is simply gone. The replica never saw it, so the new primary's history has a hole. This is exactly your replication lag, realized as data loss. Synchronous replication closes this gap, the primary waits for the replica to acknowledge before confirming a write, at the cost of write latency and an availability trade-off (if the sync replica is down, writes block). Choosing sync vs async per dataset is choosing your RPO. For the deeper rules on isolation and what "committed" even means, see Database Transactions & Consistency.

Split-brain (two primaries)

Split-brain is the nightmare: a network partition makes the controller think the primary is dead, so it promotes a replica, but the old primary is actually alive and still taking writes from clients that can still reach it. Now you have two primaries accepting conflicting writes to the same logical dataset. When the partition heals, there is no correct way to merge them; someone's data is getting overwritten or thrown away. This is silent, permanent corruption, and it's far worse than an outage.

This is why consensus-based systems (Raft/Paxos under the hood of etcd, Patroni's DCS, managed Multi-AZ) require a *majority* to elect a primary: a partition that can't reach the majority is forbidden from promoting, which makes two simultaneous primaries impossible by construction.

Operating stateful workloads on Kubernetes

Kubernetes was built for the easy case, stateless pods you can kill and reschedule anywhere. Running databases on it is doable but every default works against you, because every default assumes interchangeability.

• Use a StatefulSet, never a Deployment. StatefulSets give each pod a stable identity (db-0, db-1), stable storage that follows the pod, and ordered startup, all of which a database needs and a Deployment deliberately doesn't provide.
• Pin storage with the right reclaim policy. A PersistentVolumeClaim must bind to a volume in the same zone as the pod, and the StorageClass reclaim policy must be Retain so a deleted PVC doesn't delete your data with it.
• Prefer a battle-tested Operator over hand-rolling. Operators (CloudNativePG, Zalando/Crunchy Postgres, Vitess) encode failover, fencing, backup, and PITR as controllers that understand the database. Reimplementing that with raw manifests is how you build split-brain by hand.
• Set PodDisruptionBudgets and anti-affinity. Without a PDB, a routine node drain can evict your primary and a replica at once. Anti-affinity keeps replicas on different nodes and zones so one node loss isn't two-thirds of your cluster.
• Mind that the scheduler will happily reschedule your primary. A pod eviction that's invisible for a stateless service is a failover event for a database. Treat node maintenance as a controlled failover, not a background detail.

The honest take: a managed database (RDS, Cloud SQL, Aurora) hands you tested failover, automated backups, and PITR for a price. Run databases on Kubernetes when you have a real reason, multi-cloud portability, scale, or cost at volume, and the operational maturity to back it, not by default.

Common mistakes that cost hours (and data)

1. Untested failover. The automation looks fine in the console and references a security group, IAM role, or DNS record that quietly drifted away months ago. You discover this at 3am during the only failover that matters. Run game days.
2. Treating replication as a backup. Replicas copy DROP TABLE and bad migrations instantly and faithfully. Without an independent PITR backup, a single bad command erases every copy you have.
3. Ignoring replication lag. Lag is your data-loss window made visible. A replica that's secretly 4 minutes behind turns a "zero-loss" failover into a four-minute hole. Alert on lag against your RPO.
4. No PITR, only nightly snapshots. A snapshot loses everything since the last snapshot. If a corruption lands at 4pm and your last snapshot was midnight, you've lost 16 hours. WAL archiving lets you rewind to the second before the damage.
5. No fencing, so split-brain on every partition. A promote-on-failure script with no quorum or fencing will eventually run two primaries during a network blip and silently corrupt data, worse than the outage it was meant to prevent.
6. Running stateful pods like stateless ones. Deployments instead of StatefulSets, no PDB, Delete reclaim policy, no anti-affinity, each one turns a routine Kubernetes operation into a data incident.

Takeaways

Where to go next

Stateful reliability sits at the intersection of distributed systems, transactions, and disaster recovery. These pair naturally with this article:

• Database Transactions & Consistency, what "committed" really means, isolation levels, and why consistency is the thing failover threatens.
• Disaster Recovery: RTO, RPO & DR Strategies, turning the RPO/RTO numbers in this article into a full DR strategy and runbook.
• Reliability & Resilience: Designing for Failure, the broader patterns (redundancy, blast radius, graceful degradation) that this is the stateful chapter of.
• Follow the full SRE career path to build these into a coherent operational practice.