oncall_playbook v1.0.0

name: "oncall_playbook" description: "Design or repair an engineering on-call rotation that pages only on real user impact and doesn't burn people out: alert hygiene from the Google SRE burn-rate model, runbooks that get used, rotation sizing, sane escalation, and fair compensation. Triggers when someone asks about on-call, alert fatigue, runbooks, paging policy, rotation design, or on-call pay." metadata: { "includeInPrompt": true }

On-Call Playbook

Purpose

Build or fix an on-call rotation with three guarantees: what pages is always user impact, every page arrives with a runbook, and the rotation is survivable long-term. Grounded in the Google SRE on-call model and PagerDuty's incident-response guidance.

Workflow

1. Diagnose the current state. Run the 30-day alert audit from references/alert-audit.md: every alert classified keep / convert-to-ticket / delete, with one-line reasoning. Numbers first — "hope is not a strategy" (SRE Book Ch. 11).
2. Fix alert hygiene. Keep only pages that signal user-facing symptoms. Convert raw-infrastructure-cause alerts (CPU, queue depth, memory thresholds) to dashboards or tickets. Retune the rest to multi-window burn-rate rules from the audit reference.
3. Write or repair runbooks. One per kept alert, using references/runbook-template.md. The runbook link goes in the alert notification itself. An alert with no runbook should not exist.
4. Size the rotation. Apply the SRE Book Ch. 11 minimums: 8 engineers for a single-site 24/7 rotation (6 per site, multi-site), shifts ≤12h, on-call duty ≤25% of any engineer's time, target ≤2 significant incidents per shift. If the math doesn't work, that is a hiring/staffing finding, not a "try harder" finding.
5. Write the escalation policy. Primary → secondary → lead with explicit ack windows (e.g., 5 minutes per level), plus what happens on a severe incident (commander, comms). Escalation is encouraged, not shameful (PagerDuty).
6. Set compensation. Present the models from references/compensation.md, pick one, and define exactly how it's tracked and paid out. Google's SRE Workbook on-call chapter lists time-off-in-lieu or cash compensation as the standard options.
7. Instrument sustainability. Feed 30 days of page data into bin/oncall-health.py and publish the health report (pages/shift, night pages, % actionable, responder distribution, worst-offender alerts).
8. Close the loop. Blameless postmortem for every user-visible incident (references/postmortem-template.md); every recurring alert gets a root-cause action item with an owner. Re-run the audit quarterly.

Output Contract

• Alert audit report: every alert, verdict (keep / ticket / delete), reasoning, new threshold for keeps.
• Runbooks: one filled template per kept alert, each answering: what does this mean, what do I check first, what usually fixes it, when do I escalate.
• Rotation design: schedule, escalation chain with ack windows, staffing arithmetic showing ≤25% on-call per engineer.
• Compensation proposal: chosen model, tracking method, payout rules.
• Health report: bin/oncall-health.py output with the sustainability metrics.

Operating Rules

1. Page on symptoms, never on causes. If a page fires without user impact, it's a bug in the alerting, not a drill. Cause alerts (CPU, lag, queue depth) are diagnostics — dashboards or tickets.
2. No invented numbers. Burn-rate thresholds come only from the verified table in references/alert-audit.md. Compensation dollar amounts come from the user or from their own inputs to the calculator — never from the model's memory of what other companies pay.
3. Every kept alert has a runbook link in the notification. No link, no page.
4. Fix the load before adding people. If pages/shift or night pages exceed healthy bounds, the fix is deleting/tuning alerts and root-causing repeats — not hiring more victims.
5. Blameless, always. Postmortems target systems and processes, never people. Action items have owners and dates.
6. Measure fairness openly. Publish the per-responder distribution. Uneven load destroys rotations faster than high load.
7. Protect recovery time, not just shift time. A quiet week with two night pages beats a loud week with none. Optimize for interruption frequency, and give back what a bad night took.