flaky_test_triage v1.0.0

name: flaky_test_triage description: Triage flaky tests end to end: detect and measure flakiness, quarantine without hiding signal, root-cause by cause category, and hold a flake budget. Use when CI is red for no reason, a test fails intermittently, or someone says "just re-run it".

Flaky Test Triage

Purpose

A flaky test is one that passes and fails on the same code. It is worse than a broken test, because a broken test tells you what to do while a flaky one teaches the team to ignore red builds. Martin Fowler's warning still holds: non-deterministic tests are "a virulent infection that can completely ruin your entire test suite." This skill is the field procedure: prove it is flakiness, get it out of the blocking path, fix the actual cause, and keep the suite honest over time.

Context for scale: Google's 2016 data showed about 1.5% of test runs reporting a flaky result, almost 16% of tests carrying some flakiness, and 84% of pass-to-fail transitions involving a flaky test rather than a real regression (John Micco, Google Testing Blog, May 2016).

Workflow

1. Confirm it is flakiness, not a regression

A real regression fails consistently after a specific commit. Flakiness fails sporadically across commits, often correlating with CI load, time of day, or execution order rather than code changes.

• Run the suspect test in isolation many times. pytest path/test_file.py::test_name --count 50 -x (pytest-repeat) is the standard hammer.
• If 50 isolated runs are green but the test flakes in the suite, the cause is shared state or order dependence. That narrows the search immediately.
• A one-off green re-run proves nothing. Flakiness is a rate, so measure it: failures per N runs.

2. Tier the flake by rate

Meta's useful framing: every real-world test is flaky to some degree; the question is how flaky. Treat it as a measurement, not a yes/no label. Common tiers practitioners use:

• High: fails in more than ~25% of runs. Quarantine immediately.
• Moderate: fails in ~5-25% of runs. Investigate and fix soon.
• Low/suspected: fails in ~1-5% of runs. Monitor and collect more data before quarantining.

Thresholds are team-configured, not gospel. Pick numbers, write them down, revisit quarterly.

3. Quarantine with guardrails

Quarantine is a state machine, not a flag: healthy -> quarantined -> (fixed -> healthy) or (stale -> deleted). The rule from Fowler: quarantine reduces the damage, "but you still have to fix them soon."

What quarantine does in CI: the test still runs, but its failure does not fail the build. It runs in a separate reporting lane so you keep collecting pass/fail data and can notice when a code change fixes it. Implementation patterns per ecosystem live in references/quarantine-recipes.md.

Guardrails (without these, quarantine becomes a graveyard):

• A named owner (a person, not "the team") and a tracking ticket.
• A hard expiry, 30 days at most. Past expiry the test gets fixed, deleted, or escalated, never silently renewed.
• Broken tests are not quarantine candidates. A test that fails consistently is a real regression: fix it directly, do not hide it.
• Quarantine is visible. Mark it in code (tag, marker, or comment with the ticket link) and keep a quarantine record: test name, file, date, owner, suspected cause, expiry, ticket.

4. Root-cause by category

Most flakiness falls into a few patterns. Classify before fixing; the fix direction follows the category. The full taxonomy with symptoms and fix recipes is in references/cause-taxonomy.md. The short version:

• Timing/async (hardcoded sleeps, racing assertions): wait for conditions, never sleep fixed durations.
• Concurrency (shared ports, files, rows): isolate resources per test.
• Order dependence (leaks between tests): fix setup/teardown so tests are independent.
• External dependencies (real network calls): mock or contract-test the boundary.
• Environment (CI resource variance, timezone, locale): pin what you can, tolerate what you cannot.
• Randomness/time (unseeded RNG, date logic): seed explicitly, inject the clock.

An intermittent failure can be a real production race condition, not test noise. Google's own warning on auto-quarantine: it "could easily mask a real race condition or some other bug in the code being tested." Investigate before dismissing.

5. Verify the fix with runs, not hope

A fix is not verified by one green run. Run the formerly flaky test at least 20 times in isolation and 20 times in the full suite. Both must be clean. If the flake was order-dependent, the suite runs matter more than the isolated ones.

Then release it from quarantine: remove the marker, delete the quarantine record row, close the ticket linking the fixing PR.

6. Hold the flake budget

Set a cap on how much flakiness the suite is allowed to carry, e.g. under 2% of tests quarantined or flagged flaky at any time. Breaching the budget is a team priority, not a backlog item. Review the number on a cadence (weekly for active suites), and track the trend: whether the overall flakiness rate is rising or falling tells you if the effort is working.

Output Contract

• Quarantine record per flaky test: test name, file, date quarantined, owner, suspected cause, expiry date, ticket link.
• Flake rate per tiered test: failures over N measured runs, isolation vs suite results.
• Fix evidence: 20+ clean isolated runs and 20+ clean suite runs before un-quarantining.
• Flake budget: current rate vs cap, reviewed on cadence, trend over time.

Operating Rules

1. Never delete signal to fix signal. Quarantined tests keep running in a non-blocking lane; that data is what tells you a code change fixed them.
2. A test that fails reliably is far better than a flaky one. Fix broken tests directly, quarantine only the flaky.
3. No owner and no expiry means no quarantine. Every quarantine entry ships with both.
4. Investigate before dismissing. An intermittent failure may be a genuine bug in production code.
5. Timebox the first fix attempt (an hour or two). If it resists, quarantine it and move on rather than blocking the suite.
6. Retries are a band-aid, not a fix. They buy back a green build while you work; they do not resolve anything, and they teach developers to ignore flakiness.
7. Keep the budget small, public, and enforced. A breached budget is a team priority, not background noise.

Sources

Research notes and source URLs: ~/workspace/skill-production/scratch/flaky-test-triage-sources.md. Key references: John Micco, "Flaky Tests at Google and How We Mitigate Them" (Google Testing Blog, May 2016); Martin Fowler, "Eradicating Non-Determinism in Tests" (2011); Luo et al., "An Empirical Analysis of Flaky Tests" (FSE 2014); Meta Engineering, "Probabilistic Flakiness" (2020).