property_based_testing v1.0.0

name: "property_based_testing" description: "Property-based testing for real codebases: finding invariants worth testing, choosing generators, shrinking failing cases, and stateful testing for state machines. Use when a function, parser, serializer, or stateful system needs tests that explore inputs instead of a handful of hand-picked examples, when a property test fails and the counterexample needs reading, or when deciding whether example tests are enough."

Property-Based Testing

Purpose

Turn a rule about code into a test that tries hard to break it. Instead of asserting one input and one expected output, state a property that must hold for every valid input, let the framework generate many inputs, and let it shrink any failure to the smallest case that still fails. The method comes from QuickCheck, written by Koen Claessen and John Hughes and presented at ICFP 2000, and it now has mature libraries in most languages. This skill is standalone: it covers picking the property, building the generator, reading a shrunk failure, and testing stateful systems against a model. It also covers the honest stopping point, where a property adds nothing and a plain example test is the right tool.

Workflow

1. Decide whether a property exists at all. A property needs a claim you can state in one sentence: a round-trip returns the input, a total is conserved, a second run changes nothing, a filtered result is a subset of the unfiltered one. If no such sentence exists, write example tests and stop. Do not force a property onto code whose behavior is a list of special cases, a user interface layout, or a single fixed report. See the stopping rules at the end of this section.

1. Pick the pattern before writing code. Work through references/property-patterns.md and name the pattern in the test: round-trip, invariant preservation, idempotence, commutativity or associativity, oracle comparison, metamorphic, hard to compute and easy to verify, or a no-crash postcondition as a starting floor. One pattern per property. Then apply the worth-it test from that file: would a plausible wrong implementation still pass this property? If yes, strengthen it or drop it.

1. Build the generator for the valid domain. Constrain the strategy itself to valid inputs instead of generating anything and filtering most of it away. Compose small generators into domain objects with the library combinators, put boundary and nasty values in on purpose, and keep the generator fast and pure. Full guidance, plus how to check the generated distribution by sampling values before you trust a pass, is in references/generators-and-stateful.md.

1. Write the test in the codebase framework. Use the library the project already has: Hypothesis for Python, fast-check for JavaScript and TypeScript, proptest or quickcheck for Rust, ScalaCheck for Scala, jqwik for Java and Kotlin, FsCheck for .NET. Shape for Hypothesis: decorate with @given(...) strategies and @settings(max_examples=100) to start. Shape for fast-check: fc.assert(fc.property(arbitraries, predicate)) with 100 runs to start. Runnable starting points: run bin/pbt_lab.py scaffold --lang python or --lang js. Worked Python tests for five patterns live in examples/hypothesis_examples.py.

1. Run it and read failures the right way. First run the lab in this skill to confirm you understand the mechanics: python3 bin/pbt_lab.py check runs three stated properties over generated inputs with a fixed seed, and python3 bin/pbt_lab.py shrink shows a large failing list reduced step by step to the minimal case [10]. In a real failure, read the shrunk counterexample before the stack trace. It is usually the bug report. Replay the failure exactly using the seed or falsifying example the library reports, fix the code, rerun the property, then keep the shrunk case as a named example test.

1. Go stateful when the bug needs a history. For caches, stores, allocators, protocol clients, and anything where order matters, single-call properties miss the point. Build a simple model, define one rule or command per operation, run generated sequences against both the model and the real system, compare after each step, and check invariants at every step. Include a restart operation when persistence is part of the claim. The step-by-step method and the framework shapes for Hypothesis state machines and fast-check commands are in references/generators-and-stateful.md.

1. Set run counts as a stated choice. Start at 100 examples per property on pull requests, which is the default run count in both Hypothesis and fast-check. Raise counts into the thousands on nightly runs for properties guarding money, stored data, or safety. Lower a count only with the trade named in the profile or commit. Keep per-example deadlines off unless a slow example is itself a failure worth catching, because a tight deadline on a loaded CI machine produces flaky failures.

When example tests are enough: the behavior is a fixed set of cases with no general rule, the input space is tiny and already covered one by one, the cost of building a honest generator exceeds the risk the property would retire, or the property you found fails the worth-it test in step 2. Keep the examples, name the reason in the test file if it is not obvious, and move on.

Output Contract

A finished property-based testing job delivers all of these:

1. Named properties. Each property states its pattern and its claim in one sentence, in the test name or a comment a reader sees first.
2. Generators that cover the domain. Strategies constrained to valid inputs, with boundary values present, and a sampled distribution the author actually looked at.
3. A run record. Framework, run count per property, seed handling, and where the count changes between pull request and nightly runs.
4. Shrunk failures resolved. Every failure found during the work is fixed, or kept as a named example test with an open issue reference. No failure is left described only by a large unshrunk input.
5. A stopping note where relevant. If part of the target was judged example-only, the reason is stated in one line, using the stopping rules in the Workflow.

Operating Rules

1. Never invent library behavior. Defaults, settings names, and API shapes in a delivered test must come from the library docs for the version in the project. The two defaults this skill asserts, 100 examples in Hypothesis and 100 runs in fast-check, were verified against library documentation and worked examples on 2026-10-14.
2. A passing property over a weak generator proves little. Sample the generated values and fix the generator when interesting inputs never appear. Say what you sampled in the run record.
3. Do not restate the implementation as the assertion. A property that copies the code under test can only agree with it. Use a pattern from the catalog instead: a relation, an invariant, an inverse, or a simpler trusted reference.
4. Keep shrinking working. Build values compositionally with library strategies and combinators. Opaque one-step generation from raw randomness produces failures nobody can read.
5. Example tests and property tests are partners. Examples document known cases and pin regressions. Properties explore. A suite with only properties loses the readable record of specific cases that mattered.
6. No new library without need. If the project has no property library and the target is small, write the example tests, name the library you would add, and let the project owner decide. Do not add a dependency as a side effect of a testing task.