A deploy went bad in production. How do you handle rollback, and how do you make rollback easy?

Question

Accepted Answer

My first instinct in a production incident is to roll back, not forward fix. Restoring service beats finding root cause in the moment, and writing a hotfix live, under pressure, against a broken system is how you make things worse. So I roll back to the last known good version to stop the bleeding, aiming for a couple of minutes, then investigate calmly once users are fine. What makes that easy is build once, immutable artifacts. Because I promoted a versioned artifact and the previous builds are still in the registry, rollback is just redeploying the prior image tag. No rebuild, no surprises, and it is exactly what was running before. The one real complication is the database, because schema changes are not as easily reversed as code. That is why I make migrations backward compatible using expand and contract, so the old code still runs against the new schema, which means I can roll the app back without rolling back the database. And ideally no human has to notice at all. I wire automatic rollback into the deploy, so if health checks fail or canary analysis sees error rate go above about 1 percent over a 5 minute bake, the system reverts the release on its own. So the package is: roll back over forward fix, immutable artifacts to make it trivial, backward compatible migrations so the database does not block it, and automation so it happens in minutes and not in an hour.

Answer

Why are database changes the hard part of rollback?

Code is easy to revert (redeploy old artifact); schema changes are not easily un-migrated and data may already depend on them. Use backward-compatible (expand/contract) migrations so old code still works against the new schema, letting you roll back code independently.

Answer

When is forward-fixing actually the right call over rollback?

When rollback is impossible or worse, e.g. the bad release already made an irreversible data change, or the issue is in a dependency, not your deploy. Otherwise roll back first to restore service, then fix.

A deploy went bad in production. How do you handle rollback, and how do you make rollback easy?

A deploy went bad in production. How do you handle rollback, and how do you make rollback easy?

The follow-ups they ask next

What the interviewer is listening for

What sinks the answer

If you genuinely do not know

Keep going with deployment

Knowing the answer is not the same as recalling it under pressure