---
name: "testing_strategy"
description: "Design a testing strategy for a codebase: unit/integration/E2E balance, coverage gates that actually mean something, and flaky-test triage with quarantine policy. Use when setting up a test plan, reviewing test health, or fixing a slow, flaky CI suite."
---

# Testing Strategy

## Purpose

Produce a testing strategy a team or agent can execute: where to invest test effort, which coverage numbers gate merges, and how flakes get triaged — each decision grounded in cost-versus-confidence tradeoffs and measured numbers, never vibes.

## Workflow

1. **Audit the suite.** Count tests per layer (unit/integration/E2E). Measure full-suite wall time, flake rate over recent runs (use `bin/flake_triage.py` on CI history), coverage per module, and which critical paths have no tests. The audit usually reveals an inverted pyramid — that explains slow CI and high flakiness.

2. **Allocate effort — pick the model that fits.** The test pyramid (Mike Cohn, *Succeeding with Agile*, 2009): many unit tests, some integration, few E2E — rule of thumb ~70/20/10. The testing trophy (Kent C. Dodds, 2019): static analysis as the base layer, weight on integration tests, unit tests for complex logic only, E2E on critical paths. Choose by where bugs escape: unit-heavy suite with integration escapes → shift effort up; slow/flaky suite → shift down. Every user-facing app keeps at least one E2E test on its critical happy path.

3. **Set coverage gates on numbers that prove something.** See `references/coverage-playbook.md`. Never gate on line coverage alone — a test that calls code and asserts nothing still "covers" it. Gate merges on diff (changed-line) branch coverage; gate critical modules on mutation score.

4. **Install the flake policy.** See `references/flake-playbook.md`. Quarantine candidates come out of `bin/flake_triage.py`: they keep running but stop blocking merges, get an owner and a TTL, and get deleted on expiry if unfixed.

5. **Emit the strategy doc and schedule a quarterly review.** Distributions drift as code changes; re-audit quarterly.

## Output Contract

A testing-strategy document contains all five, nothing more:

1. **Layer allocation target + current gap** (e.g., "target ~70/20/10, currently ~30/30/40").
2. **Merge gates in order** — which suites block, in what order (fast unit first, expensive E2E last).
3. **Coverage thresholds** — per tier: new code (diff coverage) and critical modules (mutation score). No line-coverage-alone gates.
4. **Flake policy** — entry criteria, owner assignment, TTL, list cap.
5. **Review cadence** — when the allocation gets re-audited.

Every number names its evidence (CI timings, flake_triage.py output, incident history). Ungrounded numbers are cut or labeled as examples.

## Operating Rules

1. **Never invent metrics.** Every number in the strategy comes from a measurement or is labeled as an example threshold — never asserted as a fact.
2. **Cite doctrine, don't paraphrase it into facts.** Pyramid: Cohn 2009. Trophy: Dodds 2019. Flake data: Google Testing Blog, John Micco, 2016 (1.5% of runs report flaky results; ~16% of tests have some flakiness).
3. **Quarantine is not deletion.** A quarantined test keeps running, non-blocking, with an owner and a TTL. Silence is not a policy.
4. **No E2E-heavy edge coverage.** E2E tests cover critical journeys only; edge cases belong lower.
5. **SKILL.md stays tight.** Doctrine detail lives in `references/`; the triage collector lives in `bin/`.