---
name: "incident_response"
description: "Run a production incident end-to-end for a small team: declare severity, assign roles, mitigate, communicate with stakeholders, and write a blameless postmortem with owned action items. Use when production breaks, alerts fire, users report outages, an outage needs status-page updates, or after an incident when it's time to write the postmortem."
---

# Incident Response

## Purpose

Structure beats heroics when production breaks. This skill runs a small team (2–5 people, or even solo) through a production incident using Google SRE's incident management discipline — adapted from the Incident Command System — and ends with a blameless postmortem that converts the outage into systemic improvements. One guiding rule: **restore service first, learn second; mitigate before you root-cause.**

## Workflow

### 1. Declare and classify severity (minutes 0–5)

Declaring early is cheap; under-declaring is expensive. **When in doubt, declare.** Google SRE's declaration triggers: the fix needs a second team, the outage is customer-visible, or an hour of concentrated analysis hasn't solved it. Classify by **observed impact, not cause** — the cause is unknown yet.

| Sev | Meaning | Response | Comms |
|-----|---------|----------|-------|
| **SEV1** | Full outage, data loss, or security breach; revenue-critical | Page now. Full roster, incident commander assigned | Exec + status page + customers. Updates every 15–30 min |
| **SEV2** | Major feature down or severe degradation for many users | Page on-call. Commander assigned | Status page. Updates every 30–60 min |
| **SEV3** | Partial/minor degradation, workaround exists | Handle in business hours | Internal channel. Updates as needed |
| **SEV4** | Cosmetic, edge case, or no user impact | Normal backlog — no incident process | Ticket only |

Calibrate this table to your team once, before you need it. There is no universal standard; the value is that it's pre-agreed.

Write the declaration as one line where the team coordinates (Slack/IRC/Discord):
`🔴 SEV2 declared — checkout latency p99 > 8s for ~30% of users since 14:02 UTC. IC: @you.`

### 2. Assign roles (separate the hats)

Google SRE's central lesson: **do not let one person both fix and coordinate.** Assign explicitly, by name — even if one person wears two hats early on, announce it so everyone knows.

- **Incident Commander (IC)** — owns the incident, not the fix. Holds the high-level state, runs the room, delegates, removes roadblocks. The IC decides; the IC does NOT fix. If overloaded, delegates or spawns sub-incidents.
- **Ops / Tech Lead** — the **only** role that touches production. Diagnoses, mitigates, executes rollbacks. Reports findings to the IC; nobody else deploys anything during the incident.
- **Comms Lead** — the single voice outward: stakeholder emails, status-page updates, customer notices. Shields the Ops Lead from "any update?" pings.
- **Scribe** — keeps a timestamped live timeline in a shared doc. This is what the postmortem writes itself from later.

**Solo mode:** you are IC + Ops. Announce it ("I'm IC+Ops, someone pick up Comms") so others know what's covered. Hand off command explicitly — "IC is now @sam, acknowledge" — never let it be ambiguous.

### 3. Communicate on a cadence

Silence during an incident is worse than "still investigating" — stakeholders who hear nothing assume the worst. Comms Lead owns a living incident doc (concurrent-edit wiki/doc; not the system being fixed) and a cadence:

- SEV1: status update every 15–30 min, whether or not there's news
- SEV2: every 30–60 min
- SEV3+: as needed

Templates for the status page, customer email, and internal channel updates live in [references/comms-templates.md](references/comms-templates.md). Every update says: what's affected, what we're doing, when the next update lands. Never promise an ETA you can't defend; say "next update at 15:30" instead.

### 4. Mitigate before root-cause (minutes 5–60)

Goal: **stop the bleeding, not find the elegant cause.** Try generic mitigations first, before the root cause is understood: roll back the most recent release, drain/reroute traffic, enable the kill switch / feature flag, fail over to the backup. For security incidents: isolate and stop the attack before forensics. All production changes go through Ops, announced to the IC first, logged by the Scribe with timestamps. Preserve evidence (logs, config diffs) as you go — you'll need it for root-causing later.

### 5. Resolve and monitor

Confirm service is actually restored (metrics, not vibes), mark resolved on the status page, then monitor for recurrence for at least 30 minutes before standing down. Close the incident, keep the timeline doc.

### 6. Blameless postmortem (within 48 hours)

Google SRE: a postmortem is a **written record** — the incident, its impact, actions taken, root cause(s), and follow-up actions to prevent recurrence. Writing one is not punishment; it's the learning opportunity the incident was priced at.

**Postmortem triggers** (define before you need them): user-visible downtime/degradation beyond your threshold, any data loss, any on-call intervention (rollback, traffic reroute), resolution time above your threshold, or a monitoring failure (manual discovery means monitoring failed). Any stakeholder may also request one.

**Blamelessness is non-negotiable.** Assume everyone involved had good intentions and did the right thing with the information they had. The tradition comes from healthcare and avionics via John Allspaw's blameless-postmortem culture: you can't fix people, but you can fix the systems and processes that gave them incomplete or wrong information. A postmortem that names and shames guarantees the next incident gets hidden. Every action item must be **specific, owned by a named person, and deadline-bound** — never "be more careful."

The fill-in template is [references/postmortem-template.md](references/postmortem-template.md). Review flow: draft → a senior reviewer checks completeness (is the timeline captured? is the root cause deep enough? are action items owned and prioritized?) → share with the wider team → file in a shared repository of past incidents so future on-callers can learn.

**Practice between incidents.** Incident management atrophies fast. Google SRE teams keep the muscle alive with the Wheel of Misfortune: replay a past postmortem as a role-play drill (cast an IC, Ops, Comms; the original IC observes). Even quarterly, this keeps the process second nature for a small team.

## Output Contract

A completed incident run produces:
1. **Declaration line** — severity, impact summary, timestamp, named IC.
2. **Live timeline** — timestamped log of detections, decisions, mitigations, comms.
3. **Comms record** — the status-page/customer/internal updates actually sent.
4. **Postmortem doc** — filled template: summary, timeline, impact, root cause, contributing factors, what went well, and action items with named owners and deadlines.
5. **Repository entry** — the postmortem filed where the team can find it next time.

## Operating Rules

1. **Mitigate before root-cause.** Never block service restoration on finding the cause.
2. **One commander.** The IC holds all undelegated roles; handoffs are explicit and acknowledged.
3. **Only Ops touches production.** Everyone else coordinates through the IC.
4. **Classify by impact, not cause.** When in doubt, declare; it's cheap.
5. **Update on cadence.** SEV1 every 15–30 min even with no news — silence is the failure.
6. **Blameless, always.** Systems and processes are the subject; people are assumed competent and well-intentioned. Call out where services can improve, never who to punish.
7. **Action items are owned and dated** or they don't exist. Review them at the next team sync.
8. **Accuracy rule.** Every factual claim in comms and postmortems is verified (metrics, timestamps, logs). Unverifiable = cut it or label it as hypothesis. No confident unverified claims, ever.
9. **Small-team adaptations.** Roles may collapse onto one person, but the IC is still singular and named; comms and fixing stay on different people whenever humanly possible.
10. **Practice.** Run a Wheel-of-Misfortune drill at least quarterly so the process isn't being learned for the first time during a SEV1.