PrincipalSystem Design

Design multi-region failover for a system that currently runs in one region, targeting 15 minutes recovery.

What they are really testing: Whether you start from recovery objectives and data, or from infrastructure. The state layer decides everything, and candidates who begin with load balancers have not understood the problem.

A real interview question

Design multi-region failover for a system that currently runs in one region, targeting 15 minutes recovery.

What most people say

drag me

Deploy the same infrastructure in a second region and use DNS failover to switch traffic when the first one is down.

It describes the easy half. Duplicating stateless compute is straightforward, but it says nothing about the database, which is where the data loss, the split-brain risk and most of the recovery time actually live.

The follow-ups they ask next

  • How do you prevent split brain during promotion?

    Fencing: make certain the old primary cannot accept writes before promoting, using a quorum or an external arbiter. Two primaries accepting writes is worse than being down, since reconciliation may be impossible.

  • What is realistically the largest contributor to your 15 minutes?

    Usually detection plus DNS propagation and client caching, not the technical failover. That is why health check intervals, TTLs and client behaviour deserve as much attention as the database promotion.

What the interviewer is listening for

  • Starts from RTO and RPO
  • Focuses on the data layer
  • Raises fencing and rehearsal unprompted

What sinks the answer

  • Starts with load balancers
  • No data replication story
  • Assumes failover works without testing

If you genuinely do not know

Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.

First [pin RTO and RPO, since they determine everything]. Then [the data layer, since stateless compute is easy]. For 15 minutes, [active-passive with async replication] is right, and [active-active costs cross-region write latency and conflict resolution]. Then [health-checked DNS steering, accepting client cache delay], [automated promotion with fencing against split brain], and [quarterly drills].

Keep going with system design

All 87 devops engineer questions

Knowing the answer is not the same as recalling it under pressure

Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.

Start free