Disaster Recovery: RTO, RPO & DR Strategies

The 3 a.m. region outage

A cloud region goes dark. Not a single instance, not one bad deploy, the whole region. Your API is down, your dashboard is down, and the on-call engineer reaches for the runbook that says "restore from backup." Then comes the discovery that ends careers: the nightly backups have been running green for two years, but nobody has ever restored one. The snapshots are there. They are also corrupt, incomplete, or in the same dead region. The "backup" was a job that produced files, not a recovery capability that produced a running system.

Disaster recovery (DR) is the discipline of being ready for that day before it arrives. It is not a product you buy; it is a set of decisions, how fast must we come back, how much data can we afford to lose, and how much are we willing to pay to guarantee it. This article gives you the vocabulary (RTO and RPO), the four standard DR patterns and their cost trade-offs, the difference between backups and replication, and the single habit that separates teams that recover from teams that find out: testing the restore.

RTO and RPO: the only two numbers that matter

Every DR conversation collapses into two numbers. Get them agreed with the business first; the technology choice falls out of them automatically.

RTO is how fast you get back. RPO is how much data you can lose.

RTO (Recovery Time Objective) is the maximum acceptable time to restore service after a disaster. "We must be back within 1 hour" is an RTO of 60 minutes. RPO (Recovery Point Objective) is the maximum acceptable amount of data loss, measured in time. "We can lose at most 5 minutes of writes" is an RPO of 5 minutes, meaning your recovery point can be no older than 5 minutes before the failure.

The picture: one primary, four ways out

Here is the mental model. You run in a primary region. A DR strategy is whatever you keep warm in a secondary region so that when the primary fails, you can fail over to it. The patterns differ only in how much you keep running on the other side.

Building a DR plan is a sequence, not a purchase. Walk it in order:

1. 1

   Set RTO and RPO with the business

   Per service, not for the whole company. Your billing pipeline and your blog probably deserve different numbers.

2. 2

   Pick the DR pattern that meets those numbers

   Choose the cheapest pattern whose recovery characteristics satisfy the agreed RTO/RPO, see the table below.

3. 3

   Replicate data and (optionally) infrastructure cross-region

   Get your source of truth out of the failure domain. Snapshots for loose RPO; continuous replication for tight RPO.

4. 4

   Automate detection and failover

   Health checks trigger DNS or load-balancer cutover. Manual failover at 3 a.m. blows your RTO.

5. 5

   Document the runbook and the failback

   Coming back is half the plan: how you return to primary once it recovers, without losing the writes that happened in the secondary.

6. 6

   Test it on a schedule

   An untested plan has an unknown RTO, which is the same as no plan.

The four DR patterns

There are four canonical patterns, ordered from cheapest/slowest to most expensive/instant. They are a spectrum: as you move down, you keep more of the secondary running, RTO and RPO shrink, and your monthly bill grows.

• Backup & restore, store backups in another region; spin everything up from scratch when disaster strikes. Cheapest, slowest.
• Pilot light, keep a minimal core always running in the secondary (e.g. a replicated database), with app servers switched off until needed.
• Warm standby, a scaled-down but fully functional copy of the stack is always running; you scale it up and shift traffic on failover.
• Multi-site active/active, both regions serve live traffic all the time; a region loss is just lost capacity, not lost service.

Backups vs. replication: not the same thing

These two get conflated constantly, and the confusion is what produces the 3 a.m. surprise. A backup is a point-in-time copy you can restore from later, it protects against data loss and corruption, including logical errors like a bad migration or a DROP TABLE. Replication is a continuously-updated live copy of your data in another location, it gives you a low-RPO failover target.

The critical difference: replication faithfully copies your mistakes. If someone deletes a table, replication deletes it in the replica milliseconds later. A backup from before the delete is the only thing that saves you. You need both, replication for fast failover, backups for the corruption and oops cases. Replication is not a backup, and a backup is not a failover target.

Build it: backup with a restore-verification check

The single most valuable thing you can automate is not the backup, it is proving the backup restores. Below, a nightly job takes a backup, copies it cross-region, then restores it into a throwaway database and runs a sanity query. If the restore or the check fails, it exits non-zero so your alerting fires. A backup whose restore is never exercised has an unknown value.

#!/usr/bin/env bash
set -euo pipefail

# --- config ---
DB_HOST="prod-db.internal"
DB_NAME="app"
TS="\$(date -u +%Y%m%dT%H%M%SZ)"
DUMP="/tmp/\${DB_NAME}-\${TS}.sql.gz"
BUCKET="s3://dr-backups-us-west-2"   # secondary region bucket
VERIFY_DB="verify_\${TS}"

# 1. take a consistent backup
pg_dump -h "\$DB_HOST" -d "\$DB_NAME" --format=plain \
  | gzip > "\$DUMP"
echo "backup written: \$DUMP (\$(du -h "\$DUMP" | cut -f1))"

# 2. ship it OUT of the primary region (your RPO copy)
aws s3 cp "\$DUMP" "\${BUCKET}/\${DB_NAME}/\${TS}.sql.gz"

# 3. RESTORE into a throwaway db and verify it is usable
createdb "\$VERIFY_DB"
trap 'dropdb --if-exists "\$VERIFY_DB"; rm -f "\$DUMP"' EXIT

gunzip -c "\$DUMP" | psql -q -d "\$VERIFY_DB"

# 4. sanity check: the data is actually there
ROWS=\$(psql -tA -d "\$VERIFY_DB" -c "SELECT count(*) FROM users;")
if [ "\$ROWS" -lt 1 ]; then
  echo "VERIFY FAILED: users table empty after restore" >&2
  exit 1
fi

echo "restore verified: \$ROWS rows in users, backup is usable"

Wire the exit code into your monitoring: a non-zero exit (failed backup, failed restore, or empty table) should page someone. The day this job goes red is the day you learn your DR is broken, on a Tuesday afternoon, not during a real outage.

Test your restores. Run game-day failovers.

A DR plan you have never executed is a hypothesis, not a plan. Two practices turn the hypothesis into a measured capability.

Restore drills

Regularly restore a backup into a fresh environment and confirm the application actually runs against it. This catches the things that silently rot: a backup that excludes a critical schema, a snapshot encrypted with a key you no longer have, an RPO that drifted because the job started failing weeks ago. The verification script above is a continuous restore drill; do a full manual one quarterly.

Game-day failover

On a scheduled day, deliberately fail over to your secondary region with the team watching, ideally on real traffic. Measure the actual RTO and RPO and compare them to your targets. You will discover the gaps that only show up under failover: DNS TTLs too long, a secret missing in the secondary, a runbook step that assumes the primary is reachable. Then practice the failback. Each drill makes the real event boring, which is exactly what you want.

Common mistakes that cost hours (or the company)

1. Untested backups. The backup job is green for years and nobody ever restores. You learn it's corrupt during the outage. Verify every restore.
2. No RPO target. You replicate "for safety" but never decided how much data loss is acceptable, so you can't tell if your setup is adequate. Set the number first.
3. Single region. Backups and replicas living in the same region as production die with it. Get your recovery copy into a different failure domain.
4. Confusing replication with backup. A replica copies your DROP TABLE instantly. Keep point-in-time backups for corruption and human error.
5. Manual failover. A runbook that requires a human to notice and act at 3 a.m. cannot meet a tight RTO. Automate detection and cutover.
6. Forgetting failback. You fail over fine, then have no plan to return to primary without losing the writes that happened in the secondary.
7. DNS TTLs too long. A 24-hour TTL means clients keep hitting the dead region long after you cut over. Lower TTLs on failover-critical records.

Takeaways

Where to go next

DR is one slice of designing systems that survive the bad day. Two companion reads go deeper on the surrounding ground:

• Multi-Region Architecture: When You Need It, the infrastructure side of running in more than one region, and when the cost is actually worth it.
• Reliability & Resilience: Designing for Failure, error budgets, redundancy, and graceful degradation, the everyday cousins of DR.

Ready to put it together end to end? Walk the Cloud Engineer career path to build resilient, multi-region systems from fundamentals up, including hands-on labs where you replicate data, automate failover, and measure your real RTO.