secret_rotation_runbook v1.0.0

name: secret-rotation-runbook description: Rotate production credentials (DB passwords, API keys, tokens, certs) without outages. Triggers when an agent needs to rotate secrets, plan a rotation schedule, build a secrets inventory, or respond to a credential leak.

Secret rotation runbook

Rotate credentials end to end without breaking production. The order of operations is the whole game: inventory first, rotate in blast-radius order, verify each rotation with a canary before invalidating the old value, and keep an audit trail of what changed and when.

Purpose

• Build a secrets inventory before touching anything.
• Pick the right rotation pattern per secret type (dual-user, overlap window, invalidating).
• Run rotations one at a time, lowest blast radius first, with canary verification.
• Produce an audit trail: what rotated, when, by whom, and the verification result.

Workflow

1. Inventory. Run bin/secrets-inventory.py to build or update inventory.json. Every row: name, type, where it lives (secret manager path), every consumer, rotation pattern, last rotated, grace window. If a secret has no owner or no known consumer list, that is a finding. Resolve it before rotating.
2. Classify. Tag each secret with its rotation pattern (see references/rotation-patterns.md):

• dual-user (DB credentials): alternate two users, zero downtime.
• overlap-window (API keys, tokens): issue new, deploy, keep old live through a grace window, then revoke.
• invalidating (JWT signing keys, session secrets): rotation logs users out, so plan the timing and comms.
• non-rotatable (encryption keys, hash salts): never blind-rotate; needs a key-migration plan first.

1. Order the run. Sort by blast radius ascending: staging-only first, then shared third-party API keys, then primary DB credentials last. One secret at a time; never batch rotations across interdependent systems.
2. Rotate one secret using its pattern: a. Generate the new value with a CSPRNG, never a hand-typed string: python3 -c "import secrets; print(secrets.token_hex(32))". b. Store the new value in the secret manager alongside the old one (or under a new version label), never in git, never in a ticket. c. Canary: make a real authenticated request with the NEW value from a single node or canary pod. See references/canary-checks.md for per-type checks. d. Only after the canary passes: roll the new value out everywhere (rolling restart, secret-manager version flip). e. Keep the old value valid through its grace window (24h is a sane default), then invalidate it and confirm no traffic still uses it (auth-failure dashboards should stay flat after invalidation).
3. Verify and audit. After the grace window: confirm old credentials are dead (attempt a login with them, expect failure). Append one line per rotation to the audit log: timestamp, secret name, pattern, who ran it, canary result, invalidation confirmed. No secret VALUES in the audit log, ever.
4. Rollback plan. Before step 4 of each rotation, note the exact revert: flip the version label back to the previous version (or keep the old API key un-revoked through the grace window). If the canary fails, revert first and diagnose later.

Output contract

• inventory.json: the full secrets inventory, values absent.
• rotation-plan.md: ordered list with pattern and grace window per secret.
• rotation-audit.log: one line per completed rotation; no secret values.
• A spoken or written summary: what rotated, verification evidence, what is still pending (e.g. non-rotatable keys awaiting migration plans).

Operating rules

1. No rotation without an inventory row: unknown consumers mean unknown blast radius.
2. One secret at a time. Two rotations in flight is how correlated outages start.
3. Canary before invalidation, always. The old value is your rollback until the grace window closes.
4. Never commit a secret value to git. A value that touched history is compromised; rotate it on sight.
5. Rehearse in staging first: keys you have never rotated are keys you do not know how to rotate.
6. Alert on rotation health: auth-failure spikes after a rotation mean a missed spikes after a rotation mean a missed consumer. Pause the run and find it.
7. Encryption keys, salts, and JWT signing keys follow their special paths (references/rotation-patterns.md); they never go through the standard loop.