mutation_testing v1.0.0

name: "mutation_testing" description: "Mutation test a test suite: pick the right tool for the stack, run mutants on one scoped module first, read the mutation score honestly, kill surviving mutants with stronger assertions, and set a CI gate that ratchets. Use when coverage looks high but tests feel weak, when a suite passes code that is plainly wrong, or when setting up Stryker, PIT, mutmut, Infection or cargo-mutants."

Mutation Testing

Purpose

Find out whether a test suite would catch real bugs, not just whether it runs the code. A mutation tool seeds small faults into the source, one at a time, and reruns the suite against each faulty version (a mutant). A mutant a test fails on is killed. A mutant every test passes is a survivor, and each survivor names a behaviour no test actually checks. Line coverage cannot tell you this: a test that calls a function and asserts nothing covers the line and kills no mutants.

Workflow

1. Start from a green, fast suite. Mutation testing reruns the suite once per mutant, so a red or slow suite makes the run useless or endless. Fix failing tests first. Time the plain suite: if it takes minutes, scope down (step 3) or speed it up before going further. PIT, for one, expects a green suite before it starts.

1. Pick the tool for the stack. Use references/tool-picker.md. Short version: StrykerJS for JavaScript and TypeScript, PIT for Java and Kotlin on the JVM, mutmut for Python (Cosmic Ray when you need resumable sessions), cargo-mutants for Rust, Infection for PHP, Stryker.NET for C#. Do not switch the project's test runner to please the mutation tool.

1. Scope to one module first. Choose the module where a silent bug costs most: pricing, permissions, eligibility, billing rules, validation. Exclude generated code, data classes, config wiring and thin glue. A first run should finish in minutes and produce a survivor list a person can read in one sitting. Whole repo runs come later, on a schedule.

1. Run the tool and save the machine readable report. Keep the JSON or XML report, not just the console summary, because the gate and the triage both read it. Commands per stack are in references/tool-picker.md. Parse the report with bin/mutation_score.py REPORT (Stryker family JSON and PIT XML supported). It prints counts, two scores, and the survivors grouped by file.

1. Read the score honestly. Mutation score is killed mutants divided by the mutants that could have been killed. Equivalent mutants (mutated code that behaves identically for every input) can never be killed and stay out of the denominator. The parser reports two numbers and you quote both:

• Detected score: (killed + timeout) / (killed + timeout + survived). Strength on code the tests reach.
• Total score: the same numerator with no-coverage mutants added to the denominator. This is the number that stops a team hiding behind a small well tested corner. Never quote a score without its scope, its tool, and which denominator it uses. A score of 80 on one pricing module and 80 on a whole repo are different claims. A score from a run with many compile or runtime error mutants is a broken run, not a result.

1. Triage every survivor into a bucket, then kill it with an assertion. Use references/survivor-triage.md. The common patterns: boundary mutants die to a test at the exact boundary value, arithmetic and return mutants die to exact value assertions with inputs that separate the operators (never only 0 or 1), branch mutants die to one test per branch outcome, and call removal mutants die to assertions on the side effect. Prove an equivalent mutant in writing or treat it as a survivor you do not understand yet. Worked example: in examples/, a weak two test suite kills 5 of 12 hand applied mutants in pricing.py (41.7 percent). The strengthened suite in test_pricing_after.py, which adds boundary, exact value and rejection tests, kills all 12. Both suites pass against the unmutated code, which is the whole point: passing was never the evidence.

1. Set a CI gate that ratchets, on the scoped module only. Three placements, in the order most teams should adopt them:

• Pull requests: mutate only changed code where the tool supports it (StrykerJS --incremental reuses unchanged results, Infection and Stryker.NET scope by config, PIT scopes by targetClasses). Keep this run inside a few minutes or developers will route around it.
• Scheduled: a nightly or weekly full run on the scoped modules, reported as an artifact, triaged the next working day. Full runs on every push are rarely worth the compute.
• The floor: set the break threshold at or just below the current verified score on the scoped module, so the gate stops backsliding without demanding instant perfection. Raise the floor in small steps as survivors die, and never lower it without a written reason in the same change. Stryker spells this as thresholds.break (a score below it exits 1, defaults are high 80, low 60, break 0). Infection splits it into minMsi for all code and minCoveredMsi for covered code, and the covered floor should sit higher. Where the tool has no gate (mutmut), gate with bin/mutation_score.py REPORT --break N, which exits 1 below the floor.

1. Report and stop at the agreed scope. Deliver the report contract below. Expanding to the next module is a new decision with its own baseline run, not a continuation of this one.

Output Contract

A mutation testing report contains all six:

1. Scope and tool: module or files mutated, tool and version, suite runtime, date of the run.
2. Counts: mutants generated, killed, survived, no coverage, timeout, excluded or error, each as a raw count.
3. Both scores: detected score and total score, each with its denominator stated.
4. Survivor list with buckets: every survivor and no-coverage mutant, its file and line, its triage bucket from references/survivor-triage.md, and the test added to kill it or the written reason it stays.
5. Equivalent mutant log: each claimed equivalent mutant with its short proof, or the code simplification that removed it.
6. Gate: where the run sits in CI (pull request, scheduled, or not yet gated), the break floor, what score it was set from, and the next planned raise.

Operating Rules

1. Never invent a score, a count, or a tool behaviour. Every number comes from a report file produced in the run or is labelled as the worked example in examples/. Tool flags and config keys come from the tool's own docs, checked in references/tool-picker.md.
2. Coverage is not the target and neither is 100 percent. Chasing a perfect score produces tests that pin implementation details and exclusions with no proof. A high score on code that matters, with every remaining survivor explained, beats a gamed 100.
3. Green suite first, always. A mutation run on a failing or flaky suite produces noise that looks like survivors. Fix the suite, then mutate.
4. Do not contort tests to kill equivalent mutants. Prove equivalence, simplify the code, or exclude the mutant with the proof recorded. Those are the only three honest exits.
5. Timeouts count as detected, errors count as a broken run. A timeout usually means a test caught an infinite loop the mutant created. A pile of compile or runtime errors means the tool or scope is wrong.
6. Keep the gate scoped and ratcheting. A whole repo blocking gate on day one gets disabled within a month. A scoped floor that only moves up survives.
7. Mutated code never ships. Tools such as PIT hold mutants in memory and mutmut 3 works on a copy, but verify the working tree is clean after every run before committing anything.