How do you limit the blast radius of a failure so one problem does not take down everything?

Question

Accepted Answer

Limiting blast radius is about containing failure so 1 problem hits a fraction of the system, not all of it, and it goes beyond plain redundancy. The strongest lever is partitioning: divide users or tenants into independent cells or shards, each a self-contained stack. With 10 cells, a cell failure is a 10 percent outage rather than a 100 percent one, and you can update 1 cell at a time. Then isolate across failure domains: spread across at least 3 availability zones and, where it matters, multiple regions, and use bulkheads internally so a misbehaving dependency is confined to its own compartment, say a connection pool capped at 20 for that one caller. The thing senior engineers really watch for is shared fate, the hidden global dependencies and global actions that can fail everything at once: 1 config push applied everywhere simultaneously, 1 shared database or cache, 1 control plane. Those defeat all your redundancy if they go bad, so I try to eliminate or shard them, and where a global change is unavoidable I stage it. That is the last piece: roll changes out progressively, canary at 1 percent, then waves, then region by region, with automated rollback on error-rate regression, because deploys and config changes are the biggest single cause of correlated, everything-at-once outages. Redundancy stops a component failure being fatal. Cells, isolation, no shared fate and staged rollouts stop any single failure or change becoming global.

Answer

Why does redundancy alone not limit blast radius?

If all the redundant copies share a dependency or receive the same bad config/deploy at once, they fail together, shared fate. You need isolation (cells, failure domains) and staged changes so one fault or rollout cannot hit everything.

Answer

How do staged/canary rollouts reduce blast radius?

A bad change reaches a small fraction first; if it misbehaves, automated rollback stops it before it goes wide. Since deploys/config are a top outage cause, progressive rollout contains their blast radius.

How do you limit the blast radius of a failure so one problem does not take down everything?

How do you limit the blast radius of a failure so one problem does not take down everything?

The follow-ups they ask next

What the interviewer is listening for

What sinks the answer

If you genuinely do not know

Keep going with reliability

Knowing the answer is not the same as recalling it under pressure