test_gap_analysis v1.0.0

name: "test_gap_analysis" description: "Find the test gaps that matter in an existing codebase: map the test ecosystem first, rank untested and weakly tested paths by risk and change frequency, spot tests that pass without asserting anything real, and run a bounded pass that adds the highest-value tests first. Use when auditing test coverage, deciding what to test next, or checking whether a high coverage number is real."

Test Gap Analysis

Purpose

Produce a ranked list of test gaps worth fixing, then fix the top of that list in one bounded pass. Coverage percentage alone does not rank anything. It counts lines that ran, not lines that were checked, and it treats a money bug in pricing the same as a typo in a display label.

Workflow

1. Map the test ecosystem first. Inventory the repo before judging it. List the test runners and how to run each suite, where tests live, which modules have no test file at all, the coverage tool and its config, any mutation testing setup, and the CI jobs that gate merges. Run the suite once and record the baseline: test count, pass or fail state, wall time, and current coverage by file with branch coverage on. A gap hunt on a suite you have not run is guesswork.

1. Build the per-file inventory. For each production file record line coverage, branch coverage, churn (commits in a stated window, usually 90 days, from git log), complexity, defects fixed in the same window, and an impact score from 1 to 5. Exclude generated code, vendored code, and lockfiles. The commands and the impact guide are in references/risk-scoring.md.

1. Rank by risk, not by missing percent. Run bin/gap_score.py inventory.csv to get priority scores. Priority is ISTQB-style risk (likelihood estimated from churn, complexity, and defect history, times impact) weighted by how uncovered the file is. Read the top and the bottom of the table out loud in the report. The bottom matters: it names the untested files you are choosing to leave alone, and why.

1. Check assertion quality on the covered code. High coverage with weak tests is a hidden gap. For Python, run bin/weak_assert.py on the test tree: it flags NO_ASSERT tests (run code, check nothing), TAUTOLOGY tests (asserts that cannot fail), and WEAK_ONLY tests (nothing stronger than a not-None or truthiness check). For other languages, and for sampling judgement calls, use the manual three-question check in references/weak-assertions.md. Report the weak count next to the coverage number wherever you quote coverage.

1. Spot check with mutants on the highest risk module. Pick two to four small changes a real bug could look like (flip a comparison, change a constant, drop a multiplier) and run the suite against each changed copy, one at a time, restoring the file after each run. Count killed versus survived. If a full mutation tool fits the stack (PIT for JVM, Stryker for JS, TS, C#, Scala, mutmut or Cosmic Ray for Python), run it scoped to that one module instead. Surviving mutants in a high risk module jump the queue ahead of uncovered low risk files.

1. Run one bounded pass, then stop. Set the budget before writing tests: for example, the top three ranked files, or one working session. Write tests with exact expected values drawn from the spec or the code contract, covering the missing branches and error paths the inventory named, and rewrite weak tests in those files instead of adding new weak ones beside them. When the budget is spent, stop, even if the repo total still looks low. Re-run coverage, the weak assertion check, and the mutant spot check, and record before and after numbers side by side.

1. Report decisions, not just numbers. Name the files you deferred and the reason for each. A reader should be able to disagree with a deferral and point at the score behind it.

Output Contract

A gap hunt delivers all five:

1. Ecosystem map: runners, suite baseline (count, time, state), coverage tool, and modules with zero tests.
2. Ranked gap table: the bin/gap_score.py output, with the churn window and any complexity proxy stated.
3. Weak assertion report: counts by class (NO_ASSERT, TAUTOLOGY, WEAK_ONLY, OK) and the worst offenders by name.
4. Bounded pass record: which ranked gaps got tests, which weak tests were rewritten, and the budget that bounded the pass.
5. Before and after numbers: coverage by fixed file, weak count, and mutant kill count, plus the deferred list with reasons.

A worked hunt with real before and after numbers is in references/worked-example.md.

Operating Rules

1. Coverage measures execution, not correctness. Never present a coverage number as proof that code works, and never quote one without the weak assertion count beside it. A test with no assertion still counts toward coverage, which is why a suite can sit at high coverage and kill zero mutants.
2. Branch gaps outrank line gaps. A line can run while one of its branches never does, and every if has a hidden else side. Hunt missing branches and partial branches before missing lines.
3. Never invent the inputs. Churn, complexity, and defect counts come from commands you ran. Impact scores are labelled as judgement, and guessed impact scores are flagged as guesses. Thresholds and scoring bands in the references are stated starting points, not findings.
4. Do not chase a repo-wide percentage. Testing a 0 percent covered display helper while a partly covered payment file has surviving mutants is backwards. The ranked table decides, and deferring low risk files is a correct outcome, documented in the report.
5. Expected values need a source. Take each expected value from a spec, a doc, a contract, or a worked hand calculation you show. If nobody can say what the right answer is, report an unspecified behaviour gap instead of inventing one.
6. Restore every mutant. Mutation spot checks change production code temporarily. Restore the file and re-run the clean suite before reporting, and never commit a mutant.