testing_strategy v1.0.0

name: "testing_strategy" description: "Design a testing strategy for a codebase: unit/integration/E2E balance, coverage gates that actually mean something, and flaky-test triage with quarantine policy. Use when setting up a test plan, reviewing test health, or fixing a slow, flaky CI suite."

Testing Strategy

Purpose

Produce a testing strategy a team or agent can execute: where to invest test effort, which coverage numbers gate merges, and how flakes get triaged — each decision grounded in cost-versus-confidence tradeoffs and measured numbers, never vibes.

Workflow

1. Audit the suite. Count tests per layer (unit/integration/E2E). Measure full-suite wall time, flake rate over recent runs (use bin/flake_triage.py on CI history), coverage per module, and which critical paths have no tests. The audit usually reveals an inverted pyramid — that explains slow CI and high flakiness.

1. Allocate effort — pick the model that fits. The test pyramid (Mike Cohn, Succeeding with Agile, 2009): many unit tests, some integration, few E2E — rule of thumb ~70/20/10. The testing trophy (Kent C. Dodds, 2019): static analysis as the base layer, weight on integration tests, unit tests for complex logic only, E2E on critical paths. Choose by where bugs escape: unit-heavy suite with integration escapes → shift effort up; slow/flaky suite → shift down. Every user-facing app keeps at least one E2E test on its critical happy path.

1. Set coverage gates on numbers that prove something. See references/coverage-playbook.md. Never gate on line coverage alone — a test that calls code and asserts nothing still "covers" it. Gate merges on diff (changed-line) branch coverage; gate critical modules on mutation score.

1. Install the flake policy. See references/flake-playbook.md. Quarantine candidates come out of bin/flake_triage.py: they keep running but stop blocking merges, get an owner and a TTL, and get deleted on expiry if unfixed.

1. Emit the strategy doc and schedule a quarterly review. Distributions drift as code changes; re-audit quarterly.

Output Contract

A testing-strategy document contains all five, nothing more:

1. Layer allocation target + current gap (e.g., "target ~70/20/10, currently ~30/30/40").
2. Merge gates in order — which suites block, in what order (fast unit first, expensive E2E last).
3. Coverage thresholds — per tier: new code (diff coverage) and critical modules (mutation score). No line-coverage-alone gates.
4. Flake policy — entry criteria, owner assignment, TTL, list cap.
5. Review cadence — when the allocation gets re-audited.

Every number names its evidence (CI timings, flake_triage.py output, incident history). Ungrounded numbers are cut or labeled as examples.

Operating Rules

1. Never invent metrics. Every number in the strategy comes from a measurement or is labeled as an example threshold — never asserted as a fact.
2. Cite doctrine, don't paraphrase it into facts. Pyramid: Cohn 2009. Trophy: Dodds 2019. Flake data: Google Testing Blog, John Micco, 2016 (1.5% of runs report flaky results; ~16% of tests have some flakiness).
3. Quarantine is not deletion. A quarantined test keeps running, non-blocking, with an owner and a TTL. Silence is not a policy.
4. No E2E-heavy edge coverage. E2E tests cover critical journeys only; edge cases belong lower.
5. SKILL.md stays tight. Doctrine detail lives in references/; the triage collector lives in bin/.