---
name: secret-rotation-runbook
description: Rotate production credentials (DB passwords, API keys, tokens, certs) without outages. Triggers when an agent needs to rotate secrets, plan a rotation schedule, build a secrets inventory, or respond to a credential leak.
---

# Secret rotation runbook

Rotate credentials end to end without breaking production. The order of operations
is the whole game: inventory first, rotate in blast-radius order, verify each
rotation with a canary before invalidating the old value, and keep an audit
trail of what changed and when.

## Purpose

- Build a secrets inventory before touching anything.
- Pick the right rotation pattern per secret type (dual-user, overlap window, invalidating).
- Run rotations one at a time, lowest blast radius first, with canary verification.
- Produce an audit trail: what rotated, when, by whom, and the verification result.

## Workflow

1. **Inventory.** Run `bin/secrets-inventory.py` to build or update
   `inventory.json`. Every row: name, type, where it lives (secret manager path),
   every consumer, rotation pattern, last rotated, grace window. If a secret has
   no owner or no known consumer list, that is a finding. Resolve it before rotating.
2. **Classify.** Tag each secret with its rotation pattern (see
   `references/rotation-patterns.md`):
   - dual-user (DB credentials): alternate two users, zero downtime.
   - overlap-window (API keys, tokens): issue new, deploy, keep old live through a grace window, then revoke.
   - invalidating (JWT signing keys, session secrets): rotation logs users out, so plan the timing and comms.
   - non-rotatable (encryption keys, hash salts): never blind-rotate; needs a key-migration plan first.
3. **Order the run.** Sort by blast radius ascending: staging-only first, then
   shared third-party API keys, then primary DB credentials last. One secret at a
   time; never batch rotations across interdependent systems.
4. **Rotate one secret** using its pattern:
   a. Generate the new value with a CSPRNG, never a hand-typed string:
      `python3 -c "import secrets; print(secrets.token_hex(32))"`.
   b. Store the new value in the secret manager alongside the old one (or under
      a new version label), never in git, never in a ticket.
   c. Canary: make a real authenticated request with the NEW value from a single
      node or canary pod. See `references/canary-checks.md` for per-type checks.
   d. Only after the canary passes: roll the new value out everywhere (rolling
      restart, secret-manager version flip).
   e. Keep the old value valid through its grace window (24h is a sane default),
      then invalidate it and confirm no traffic still uses it (auth-failure
      dashboards should stay flat after invalidation).
5. **Verify and audit.** After the grace window: confirm old credentials are dead
   (attempt a login with them, expect failure). Append one line per rotation to
   the audit log: timestamp, secret name, pattern, who ran it, canary result,
   invalidation confirmed. No secret VALUES in the audit log, ever.
6. **Rollback plan.** Before step 4 of each rotation, note the exact revert:
   flip the version label back to the previous version (or keep the old API key
   un-revoked through the grace window). If the canary fails, revert first and
   diagnose later.

## Output contract

- `inventory.json`: the full secrets inventory, values absent.
- `rotation-plan.md`: ordered list with pattern and grace window per secret.
- `rotation-audit.log`: one line per completed rotation; no secret values.
- A spoken or written summary: what rotated, verification evidence, what is
  still pending (e.g. non-rotatable keys awaiting migration plans).

## Operating rules

1. No rotation without an inventory row: unknown consumers mean unknown blast radius.
2. One secret at a time. Two rotations in flight is how correlated outages start.
3. Canary before invalidation, always. The old value is your rollback until the
   grace window closes.
4. Never commit a secret value to git. A value that touched history is compromised;
   rotate it on sight.
5. Rehearse in staging first: keys you have never rotated are keys you do not
   know how to rotate.
6. Alert on rotation health: auth-failure spikes after a rotation mean a missed
   spikes after a rotation mean a missed consumer. Pause the run and find it.
7. Encryption keys, salts, and JWT signing keys follow their special paths
   (references/rotation-patterns.md); they never go through the standard loop.