Severity levels: P0 through P4

• P0 — Critical — Complete service outage or data loss affecting all users. Revenue-generating systems are down. Examples: payment processing is completely broken, the entire website returns 500 errors, database corruption is spreading. Response: all-hands, executive notification, status page updated immediately.
• P1 — High — Major feature degradation affecting a significant percentage of users (>10%). Core functionality is impaired but not entirely unavailable. Examples: search results are returning stale data, login succeeds but session tokens expire after 30 seconds, API latency is 10x normal. Response: dedicated incident team, stakeholder notification within 15 minutes.
• P2 — Moderate — Partial degradation affecting a subset of users or a non-critical feature. Workarounds exist. Examples: image uploads fail for users in a specific region, email notifications are delayed by 2 hours, a dashboard widget shows incorrect data. Response: on-call engineer investigates, business-hours escalation.
• P3 — Low — Minor issue with minimal user impact. Functionality is degraded but operational. Examples: a tooltip displays the wrong text, CSV export is slow but functional, a non-essential API endpoint returns stale cache data. Response: ticket filed, addressed in normal sprint cycle.
• P4 — Informational — No current user impact but requires tracking. Potential future risk or cosmetic issue. Examples: a dependency has a known vulnerability with no exploit in the wild, disk usage is trending toward capacity in 3 months, a deprecated API is still receiving traffic. Response: tracked in backlog, reviewed during planning.

The five core incident roles

• Incident Commander (IC) — The single point of authority and coordination during an incident. The IC does NOT debug the problem — they coordinate the people who do. Responsibilities include: assigning all other roles, making severity classification decisions, authorizing mitigation actions (rollbacks, failovers), managing the timeline and checkpoints, deciding when to escalate, and determining when the incident is resolved. The IC should spend 80% of their time coordinating and communicating, and at most 20% on technical work. When in doubt, the IC delegates technical tasks and focuses on keeping the response organized. At Google, IC is a formally trained role with certification requirements.
• Communications Lead (Comms Lead) — Owns all external and internal communication during the incident. This includes updating the status page, posting regular updates to the incident Slack channel, sending email notifications to stakeholders, coordinating with customer support teams, and drafting any customer-facing communications. The Comms Lead shields the technical team from communication overhead — engineers should never be interrupted to write a status update while they are debugging. Update frequency depends on severity: P0 gets updates every 15 minutes, P1 every 30 minutes, P2 every hour.
• Operations Lead (Ops Lead) — Leads the technical debugging and mitigation effort. The Ops Lead is typically the most senior engineer available with domain expertise in the affected system. They direct the diagnostic work, propose mitigation strategies to the IC for approval, and execute or delegate the technical fix. The Ops Lead works closely with Subject Matter Experts and reports findings to the IC. In smaller incidents, the Ops Lead may also be the person hands-on-keyboard making changes, but in larger incidents they coordinate multiple engineers working in parallel on different investigation threads.
• Subject Matter Experts (SMEs) — Engineers with deep knowledge of specific subsystems who are pulled into the incident as needed. The IC or Ops Lead identifies which systems are involved and pages the appropriate SMEs. SMEs provide diagnostic insight, answer questions about system behavior, and execute targeted technical tasks. A typical P0 incident might involve SMEs from the database team, the networking team, and the application team working in parallel on different investigation threads. SMEs report to the Ops Lead, not directly to the IC, to maintain a clean command structure.
• Scribe — Documents everything that happens during the incident in real time. The scribe maintains a running timeline of events, decisions, actions taken, and their outcomes. This timeline becomes the foundation of the postmortem document and is invaluable for reconstructing what happened after the fact. Without a scribe, the postmortem relies on memory, which is unreliable under stress. The scribe captures: timestamps of key events, who did what, what commands were run, what the results were, when severity changed, and when communications were sent. Many teams automate parts of this role using incident management tools that log Slack messages and PagerDuty actions automatically.

Communication anti-patterns to avoid

• The Silent Incident — No updates for 45+ minutes. Stakeholders do not know if the team is working on it or if everyone went to lunch. Fix: set a recurring timer and post updates even when there is no new information.
• The Blame Channel — Engineers start pointing fingers in the incident channel: "this is the database team's fault" or "who approved this deploy?" Blame during an incident wastes time and damages trust. Fix: the IC immediately redirects blame conversations and reminds the team that root cause analysis happens in the postmortem, not during mitigation.
• The Noise Flood — Dozens of people join the incident channel and start asking questions, posting theories, or offering unsolicited help. The signal-to-noise ratio drops to zero. Fix: use a structured incident channel where only the IC and assigned roles post primary updates. Others can post in a separate thread or a companion discussion channel.
• The Stakeholder Interrupt — A VP joins the incident channel and starts asking "why is this happening?" and "when will it be fixed?" These are legitimate questions but asking them in the incident channel interrupts the team. Fix: the Communications Lead proactively sends executive updates on a regular cadence so executives have information without needing to interrupt the war room.
• The Internal Jargon Blast — The status page update says "connection pool exhaustion in pgbouncer causing 502s from the API gateway." Customers have no idea what this means. Fix: external communications use plain language — "Some customers may experience errors when attempting to make purchases. Our engineering team is actively working to resolve this."

The mitigation toolkit: ordered by speed and safety

• Rollback the last deployment — If the incident correlates with a recent deploy, rolling back is the fastest and safest mitigation. Modern CI/CD systems should support one-click rollback to the previous known-good version. Target: rollback should complete within 5 minutes. This is by far the most common mitigation strategy and resolves the majority of incidents caused by code changes. If your rollback takes more than 15 minutes, fix your deployment pipeline before fixing anything else.
• Toggle a feature flag — If the incident is caused by a specific feature, disabling that feature via a feature flag is faster than a full rollback and more targeted. Feature flags allow you to surgically disable problematic functionality while keeping the rest of the service healthy. This requires the feature to have been built behind a flag — which is why feature flags should be mandatory for all new features. Target: feature flag toggle takes effect within 1 minute.
• Shift traffic away from the affected region/instance — If the problem is localized to a specific region, availability zone, or set of instances, redirect traffic to healthy capacity. This can be done via DNS changes (slow, 5-30 minute TTL), load balancer configuration changes (fast, 1-2 minutes), or service mesh routing rules (fastest, seconds). This buys time to investigate the problem in the affected region without user impact.
• Scale up or out — If the incident is caused by capacity exhaustion (connection pool full, CPU saturated, memory exhausted), adding more capacity can restore service. This is a temporary mitigation, not a permanent fix — you need to understand why capacity was exhausted and either optimize the code or right-size the infrastructure. Auto-scaling should handle this automatically, but if auto-scaling is broken or too slow, manual scaling is the mitigation.
• Apply a targeted workaround or hotfix — If you can identify the specific code path or configuration causing the issue, a targeted fix may be faster than a full rollback. This is riskier than a rollback because you are pushing a change under pressure, and changes made under pressure have a higher error rate. Use this only when rollback is not possible (e.g., the previous version has a different critical bug, or data migration prevents rollback). Require a second engineer to review the hotfix before deployment.
• Fail over to a disaster recovery site — For catastrophic failures that cannot be mitigated within the primary environment, fail over to a DR site. This is the mitigation of last resort because failover is complex, often involves data synchronization risks, and takes the longest to execute. DR failover should be tested quarterly so the team knows how to execute it under pressure.

Core incident management tool categories

• On-call management and alerting — PagerDuty and Opsgenie are the market leaders for on-call scheduling, alert routing, escalation policies, and notification delivery. These tools ensure that when an alert fires, the right person is notified through the right channel (push notification, phone call, SMS) with automatic escalation if there is no acknowledgment. They are the entry point for the incident lifecycle — without reliable alert delivery, everything downstream fails.
• Incident coordination platforms — incident.io, FireHydrant, and Rootly provide the coordination layer: automated incident channel creation, role assignment, status page integration, stakeholder communication, and postmortem generation. These tools integrate with Slack and provide slash commands like /incident create P1 "description" that trigger the entire incident workflow automatically. They dramatically reduce the manual overhead of incident management.
• Status pages — Statuspage (Atlassian), Instatus, and Better Uptime provide external-facing incident communication. When an incident is declared, the status page is updated to show the current impact, and customers can subscribe to updates via email, SMS, or RSS. A well-maintained status page reduces inbound support tickets by 30-50% during incidents because customers get information proactively instead of contacting support.
• Incident communication (Slack/Teams) — Slack is the de facto war room for most engineering organizations. Dedicated incident channels (#inc-YYYYMMDD-description) keep incident discussion separate from normal work. Slack bots automate repetitive tasks: posting status update reminders, capturing timeline events, and paging additional responders. Microsoft Teams serves the same purpose for organizations in the Microsoft ecosystem.
• Postmortem and documentation tools — Notion, Confluence, or Google Docs for postmortem documents. incident.io and FireHydrant can auto-generate postmortem templates pre-filled with the incident timeline from Slack messages. The key requirement is that postmortem documents are searchable, so the team can find previous incidents when a similar issue occurs.
• Monitoring and observability (upstream) — Datadog, Grafana, New Relic, and Prometheus are not incident management tools per se, but they are critical infrastructure for incident detection and diagnosis. The monitoring stack feeds alerts to the on-call management platform, which triggers the incident management workflow. Without monitoring, there are no alerts, and without alerts, incidents are detected by customer complaints instead of automated systems.