incident_rollback v1.0.0

---

name: incident_rollback description: Roll back a broken production deploy fast and safely: detect and declare, pick the right rollback path (flag, traffic flip, redeploy, or forward fix), handle data and config that cannot roll back, communicate to users, verify recovery, and run a blameless retro. Use when a deploy breaks production, error rates spike after a release, a canary or blue-green deploy needs aborting, or a team needs a tested rollback runbook before shipping.

---

Incident Rollback

Purpose

Restore service by undoing the last change, in minutes, without making the incident worse. Most production failures follow a change: new code, new config, or a dependency that changed. The fastest safe recovery is usually to reverse that change, then investigate. This skill covers the decision, the execution paths, the hard cases where code rolls back but data does not, and the verification and retro that close the loop.

Related skills: use deploy_safety to design the rollout before it ships, incident_response to run the wider incident, and database_migrations for expand-contract design. This skill owns the rollback itself.

Workflow

1. Detect and declare (first 5 minutes)

Do not wait for certainty. Declare when any of these are true:

• A user-facing metric breached its pre-declared threshold after a deploy: error rate, latency, saturation, or a business check such as checkout success.
• An automated canary or deployment alarm fired.
• Users report breakage that correlates with the deploy window, even if dashboards look clean.
• A data or config change shipped in the same window as the symptoms.

On declaration, record four facts in the incident doc: deploy identifier (version, commit, image tag, or config version), deploy start time, first symptom time, and current user impact. If you cannot name the deploy, say so. Rolling back the wrong change wastes the fastest recovery window.

Assign roles even on a small team: Incident Commander decides, Ops executes (only Ops touches production), Comms writes updates, Scribe keeps the timeline. One person can hold two roles, but name them out loud.

Rule for this phase: mitigate before you diagnose. Root cause work starts after service is restored, or in parallel by someone who is not blocking the rollback.

2. Decide: roll back, roll forward, or contain

Use the decision tree in references/decision-tree.md. Short version:

1. Is a feature flag or kill switch available for the broken behavior? Flip it off first. That is the fastest lever and it does not need a deploy.
2. Is traffic still split (canary or blue-green) with the old version running? Shift traffic back to the old version. That is a routing change, not a redeploy.
3. Is the change code-only, with no irreversible data, config, or external side effects? Redeploy or revert to the last known good artifact.
4. Did the deploy include an irreversible step (dropped column, destructive data transform, sent messages, charged cards, published external state)? Do not pretend a code rollback undoes it. Contain first (flag off, traffic off, pause workers), then follow the data recovery path in step 4.
5. Is a forward fix already written, tested, and smaller than the rollback risk? Only then fix forward, and only with Commander approval. A hopeful hotfix built during the incident is not a tested fix.

Run the helper to structure the call:

bash
python3 ~/workspace/skills/incident-rollback/bin/rollback_decide.py \
  --impact sev2 --flag-available --code-only --confidence high
python3 ~/workspace/skills/incident-rollback/bin/rollback_decide.py --help

If the deploy correlates with the outage and rollback is reversible, roll back first and investigate after. Waiting for proof while users are down trades user pain for diagnostic neatness.

3. Execute the rollback path

Pick one primary path. Do not run two paths at once unless the Commander orders it and the Scribe logs both.

A. Flag off (seconds) Turn the flag off for 100 percent of traffic, confirm the flag service propagated the change, and watch the symptom metric. Keep the flag in code until the retro. Removing the flag mid-incident destroys your fastest lever.

B. Traffic flip (seconds to about a minute)

• Blue-green: point the load balancer, DNS, or service routing back to the previous environment. Keep the new environment running for logs and inspection, but serving no traffic.
• Canary: set new-version weight to 0 percent. Confirm the routing change took effect, not just that the command succeeded.

C. Redeploy previous artifact (minutes)

• Kubernetes: check history, then undo. A rollback restores the Pod template only, so also check replica count, config, and secrets separately.

bash
kubectl rollout history deployment/<name>
kubectl rollout undo deployment/<name>
# or to a specific revision:
kubectl rollout undo deployment/<name> --to-revision=<n>
kubectl rollout status deployment/<name>

• Containers or VMs elsewhere: redeploy the exact prior image tag or artifact digest, not a rebuilt version of old code. A rebuild is a new artifact and can carry new surprises.
• Git revert path: revert the bad commit, push, and let the normal pipeline deploy the revert. Use this when no artifact rollback exists. It is slower, so contain with a flag or traffic flip first if you can.

D. Config rollback Reapply the prior config version from version control or the config store. Diff the restored config against the bad one and save the diff for the retro. Config rollbacks fail silently when a cache holds the old bad value, so restart or invalidate caches as the platform requires and verify the live value, not just the stored value.

In all paths: announce before you execute (channel, timestamp, path chosen, expected effect), execute, then announce completion with the observed metric change.

4. Handle what does not roll back

Code rolls back. Several other things do not, or only partly do:

| Thing | Rollback reality | What to do instead | |---|---|---| | Database expand phase (new column or table added) | Safe. Old code ignores it. | Roll back code only. Leave the new shape in place. | | Dual-write and backfill in progress | Partly reversible. Writes to both shapes can diverge if you stop mid-flight. | Stop writes or pause the backfill, roll back code, then reconcile counts before any retry. | | Contract phase (old column or table dropped) | Not reversible by code. Data in the dropped shape is gone. | Restore from the pre-contract backup or snapshot. Treat as data recovery, with its own timeline and comms. | | Data written by the bad version | Stays written. Bad rows do not vanish on redeploy. | Write a cleanup or repair job, scoped by deploy window and version marker. Test it on a copy first. | | Messages, emails, webhooks already sent | Cannot be unsent. | Send a correction if users acted on bad information. Add idempotency or a kill switch to the sender before re-enabling. | | Charges, orders, external API calls | Reversed only through that system's own reversal flow. | Use refunds, voids, or compensating transactions. Log every external identifier. | | Cache and CDN content | Old code can still serve new bad content until expiry or purge. | Purge or invalidate the affected keys and paths as part of the rollback, then verify from a fresh client. | | Mobile or desktop clients already updated | You cannot pull the update back from devices. | Use server-side flags or API compatibility to make the old server safe for the new client, or ship a fixed client through the store process. |

Design rule that prevents most of this table: use expand-contract for schema changes (add new, dual-write and backfill, switch reads, drop old last, in a separate later deploy), and keep the contract step out of the same deploy as the code that depends on it.

5. Communicate while you roll back

Comms runs in parallel, never after recovery. Use references/comms-templates.md.

• First update (within 15 minutes of declaration for SEV1 or SEV2): what is affected, that you are rolling back, when the next update lands. No root cause claim yet.
• Rollback update: rollback started, path used, expected check time.
• Recovery update: metrics returned to baseline, what you are monitoring, when you will confirm resolution.
• Resolved update: duration, impact summary, that a retro will follow. Link the incident doc.

Never promise a restore time you cannot defend. Promise the next update time instead. Keep internal and external updates consistent: same facts, different detail level.

6. Verify recovery

A rollback is not done when the command exits. Verify with references/verification-checklist.md:

1. Symptom metric back to baseline for a sustained window (at least 15 to 30 minutes for SEV1 or SEV2, or your pre-declared window).
2. Health checks and smoke tests pass: health endpoint, one real user path end to end, and the specific path that broke.
3. No new errors from the rollback itself: check logs for the old version running against any new schema or config left behind.
4. Data checks: row counts, reconciliation queries, or queue depths match expectations after any cleanup.
5. Traffic check: confirm 0 percent of traffic still reaches the bad version, including background workers, cron jobs, and async consumers, not just web traffic.

If verification fails, you have a second incident inside the first. Escalate to the Commander, contain again, and consider the next path in the decision tree (often data recovery or forward fix from the now-stable old version).

7. Stabilize, then run the blameless retro

Hold a freeze on re-deploying the bad change until the retro assigns a guard. Tag the bad artifact or commit as do-not-deploy so an automated pipeline cannot reship it overnight.

Run the retro within 24 to 48 hours while memory is fresh. Template in references/retro-template.md. Rules:

• Blameless: examine systems, process, and information available at the time. Assume people acted reasonably with what they knew.
• Timeline from the Scribe record and logs, with sources. Mark anything reconstructed from memory as such.
• Contributing factors, plural. A single root cause such as human error or bad deploy means the analysis stopped too early.
• Action items are specific, owned by a named person, dated, and classed as prevention, detection, or mitigation. Reject items like be more careful.
• Always answer three rollback questions: How long from declaration to rollback decision? How long from decision to service restored? What would make the next rollback faster or automatic?

Feed answers back into deploy design: add the missing flag, automate the canary abort, rehearse the rollback in a game day. A rollback path tested only during a real incident is not tested.

8. Rehearse before you need it

• Run a rollback drill at least quarterly per critical service: deploy a harmless bad version to staging or a small production canary, trigger the rollback path, time it.
• Keep the previous artifact retained and deployable until the new version has soaked. Deleting the old artifact right after deploy removes your recovery path.
• Keep the runbook to one page per service: deploy identifier location, flag name, traffic flip command, artifact rollback command, data contact, comms links. If the runbook needs a search during an incident, it is too long.

Output Contract

A completed rollback run produces:

1. Decision record: path chosen (flag, traffic flip, redeploy, contain plus data recovery, or forward fix), who decided, timestamp, and why alternatives were rejected.
2. Execution log: commands or console actions in order, with timestamps and observed metric response.
3. Comms record: copies or links for every internal and external update sent, with send times.
4. Verification record: completed verification checklist, including baseline comparison and the monitoring window used before declaring recovery.
5. Retro doc: filled retro template with timeline, impact, contributing factors, and owned, dated action items.
6. Do-not-redeploy marker: bad version tagged or blocked in the pipeline until retro actions clear it.

Operating Rules

1. Mitigate before diagnose. Roll back a correlated, reversible change first. Investigate from a restored service.
2. Only Ops touches production during the rollback. All changes go through the Commander and get logged by the Scribe.
3. Pre-declare thresholds before the deploy. Automatic rollback on a pre-declared metric breach beats a human debate under pressure.
4. Redeploy the exact prior artifact by tag or digest. Never rebuild old code and call it a rollback.
5. Never claim data is rolled back when only code was. State separately what happened to code, config, schema, and data.
6. Keep the old version alive until the new version has soaked and verification passed. Instant cleanup of the old environment removes the fastest path back.
7. Communicate on cadence even with no news. Silence reads as loss of control to users and to leadership.
8. Every factual claim in comms and the retro is verified against metrics, logs, or timestamps. Unverifiable claims get cut or marked as hypothesis.
9. If rollback and forward fix both look risky, contain first (flag off, traffic off, pause workers) and buy decision time without user impact growing.
10. Quality over speed of re-ship. The bad change does not go out again until the retro guard is in place and the rollback drill for the new version passes.