---
name: "oncall_playbook"
description: "Design or repair an engineering on-call rotation that pages only on real user impact and doesn't burn people out: alert hygiene from the Google SRE burn-rate model, runbooks that get used, rotation sizing, sane escalation, and fair compensation. Triggers when someone asks about on-call, alert fatigue, runbooks, paging policy, rotation design, or on-call pay."
metadata: { "includeInPrompt": true }
---

# On-Call Playbook

## Purpose

Build or fix an on-call rotation with three guarantees: **what pages is always user impact**, **every page arrives with a runbook**, and **the rotation is survivable long-term**. Grounded in the Google SRE on-call model and PagerDuty's incident-response guidance.

## Workflow

1. **Diagnose the current state.** Run the 30-day alert audit from `references/alert-audit.md`: every alert classified keep / convert-to-ticket / delete, with one-line reasoning. Numbers first — "hope is not a strategy" (SRE Book Ch. 11).
2. **Fix alert hygiene.** Keep only pages that signal user-facing symptoms. Convert raw-infrastructure-cause alerts (CPU, queue depth, memory thresholds) to dashboards or tickets. Retune the rest to multi-window burn-rate rules from the audit reference.
3. **Write or repair runbooks.** One per kept alert, using `references/runbook-template.md`. The runbook link goes *in the alert notification itself*. An alert with no runbook should not exist.
4. **Size the rotation.** Apply the SRE Book Ch. 11 minimums: 8 engineers for a single-site 24/7 rotation (6 per site, multi-site), shifts ≤12h, on-call duty ≤25% of any engineer's time, target ≤2 significant incidents per shift. If the math doesn't work, that is a hiring/staffing finding, not a "try harder" finding.
5. **Write the escalation policy.** Primary → secondary → lead with explicit ack windows (e.g., 5 minutes per level), plus what happens on a severe incident (commander, comms). Escalation is encouraged, not shameful (PagerDuty).
6. **Set compensation.** Present the models from `references/compensation.md`, pick one, and define exactly how it's tracked and paid out. Google's SRE Workbook on-call chapter lists time-off-in-lieu or cash compensation as the standard options.
7. **Instrument sustainability.** Feed 30 days of page data into `bin/oncall-health.py` and publish the health report (pages/shift, night pages, % actionable, responder distribution, worst-offender alerts).
8. **Close the loop.** Blameless postmortem for every user-visible incident (`references/postmortem-template.md`); every recurring alert gets a root-cause action item with an owner. Re-run the audit quarterly.

## Output Contract

- **Alert audit report**: every alert, verdict (keep / ticket / delete), reasoning, new threshold for keeps.
- **Runbooks**: one filled template per kept alert, each answering: what does this mean, what do I check first, what usually fixes it, when do I escalate.
- **Rotation design**: schedule, escalation chain with ack windows, staffing arithmetic showing ≤25% on-call per engineer.
- **Compensation proposal**: chosen model, tracking method, payout rules.
- **Health report**: `bin/oncall-health.py` output with the sustainability metrics.

## Operating Rules

1. **Page on symptoms, never on causes.** If a page fires without user impact, it's a bug in the alerting, not a drill. Cause alerts (CPU, lag, queue depth) are diagnostics — dashboards or tickets.
2. **No invented numbers.** Burn-rate thresholds come only from the verified table in `references/alert-audit.md`. Compensation dollar amounts come from the user or from their own inputs to the calculator — never from the model's memory of what other companies pay.
3. **Every kept alert has a runbook link in the notification.** No link, no page.
4. **Fix the load before adding people.** If pages/shift or night pages exceed healthy bounds, the fix is deleting/tuning alerts and root-causing repeats — not hiring more victims.
5. **Blameless, always.** Postmortems target systems and processes, never people. Action items have owners and dates.
6. **Measure fairness openly.** Publish the per-responder distribution. Uneven load destroys rotations faster than high load.
7. **Protect recovery time, not just shift time.** A quiet week with two night pages beats a loud week with none. Optimize for interruption frequency, and give back what a bad night took.