visual_regression_testing v1.0.0

name: visual_regression_testing description: "Visual regression testing for web interfaces: choose capture points and viewports, make captures deterministic, set diff budgets that catch small real changes, triage flaky diffs from fonts and animation, review and approve baseline updates, and wire the check into CI in stages. Ships a working Python pixel-diff tool with exit codes, masks and region counting, Playwright and BackstopJS setups, and a worked baseline update. Use when a layout breaks unnoticed, when screenshot tests flake, or when setting up visual testing from scratch."

Visual Regression Testing

Purpose

Catch visual breaks that functional tests walk past: a button that moved off screen, a colour that changed by accident, text that now overflows its card. The method is simple on paper. Capture a screenshot of a known good state (the baseline), capture the same view later, diff the two, and have a person judge the diff. The work that makes it useful sits around that loop: choosing what to capture, making captures repeatable, keeping baselines honest, and putting the check in CI at a strength the team can live with.

This skill ships working code, not just advice: bin/vrt_diff.py compares two PNGs and reports diff pixels, diff ratio, connected diff regions, and masked pixels, with exit codes designed for CI. Playwright and BackstopJS setups and a fully worked baseline update live in examples/.

Workflow

1. Pick the stack you already have. Playwright if the project runs Playwright: toHaveScreenshot() captures and compares in one assertion, with baselines stored beside the test file. BackstopJS if you want config-driven page scenarios without a test suite. A hosted service (Percy or Applitools) if cross-browser rendering and a hosted review queue are worth paying for. The shipped diff tool fits static captures and non-Playwright stacks, and teaches the budget ideas every engine uses.

1. Choose capture points by risk. Use references/capture-and-viewports.md. Start with money paths, shared components, and the states where layout breaks hide (empty, error, long text, many items). Two viewports by default, desktop 1440 by 900 and phone 390 by 844, device pixel ratio pinned to 1. Grow the list from visual bugs that escaped, not from coverage pride. A first suite of 10 to 30 captures that reviewers actually read beats a large suite they skim.

1. Make the capture deterministic before writing any budget. This step decides whether the suite lives or gets muted. Pin the environment (same container, OS, and browser build for baseline and comparison, because rendering varies by host and screenshots differ between platforms). Pin viewport and DPR in config. Wait for fonts (document.fonts.ready), for images to decode, and for a ready marker the page sets itself. Freeze animation in two layers: the tool option plus the screenshot stylesheet in examples/playwright/screenshot.css. Pin time and seed the data. Clear hover and hide the caret. The Playwright setup in examples/playwright/ wires all of this.

1. Set two budgets, not one. The per-pixel threshold decides whether one pixel counts as different at all (Playwright threshold default 0.2, pixelmatch library default 0.1). The image budget decides how many counted pixels fail the test: use maxDiffPixelRatio so the budget scales with image size, and add a region cap where you can, because a small real change can be a tiny ratio and one clear region. Start near a 0.1 percent ratio, then tune from your own run history. Never tune a budget to make a specific failure pass without naming the change that caused it.

1. Capture the baseline on stable, merged code and commit it. In Playwright the first run writes the golden file and fails by design: example-test-1-chromium-darwin.png style names carry browser and platform for a reason. In BackstopJS, backstop reference captures and backstop test compares. Commit baselines to version control, in the same change as the code they describe once the suite is running. A baseline captured on a half-finished branch poisons every later run.

1. Triage every failure with the flake rules before touching a threshold. Use references/flake-triage.md, in order: environment, viewport, fonts, timing, dynamic content, anti-aliasing. Most false diffs are capture faults. Fix the capture. Mask only regions that are dynamic by design (a live clock, an ad slot), record the reason in the test, and keep masked area visible in the report: the shipped tool counts masked pixels on every run. Never approve a flaky capture as the new baseline.

1. Review diffs like code, and update baselines on purpose. Use references/review-and-ci.md. Read baseline, current, and diff images. Write down what changed and which ticket intended it. Regenerate only approved captures (Playwright --update-snapshots scoped to the spec, BackstopJS backstop approve, or promote the reviewed PNG with the shipped tool), log the approval with the template in examples/worked-example/approval-log.md, commit baselines with the causing code, and re-run green before merging. The worked example in examples/worked-example/ walks one real change, one masked clock change, and one baseline promotion end to end, with the tool output quoted.

1. Wire CI in stages so the gate earns trust. Stage 1: run on pull requests as a non-blocking job and upload diffs as artifacts even on failure, so review becomes a habit with evidence attached. Stage 2: make a short core set (money paths, shared components) required once it runs clean. Stage 3: require the full job, with the logged baseline-update path as the escape hatch. A blocking whole-repo gate on day one gets disabled. A staged gate survives.

Output Contract

A visual regression setup or run report contains all six:

1. Capture inventory: each capture point with page or component, state, viewport, and why it earned a place (risk or an escaped bug).
2. Determinism record: the pinned environment (container or OS and browser build), viewport and DPR settings, font and image waits, animation freeze, time and data pinning, and every mask with its reason.
3. Budgets: per-pixel threshold, image budget (ratio or pixel cap), region cap if used, and what each was tuned from.
4. Run result: per capture, the verdict plus diff pixels, diff ratio, and region count or boxes from the tool or report, not just pass or fail.
5. Review record: for every failed capture, what changed, intended or not, and the action taken (bug filed, capture fixed, or baseline approved).
6. Baseline and CI state: approval log entries for any promoted baselines, the commit they landed in, and the CI stage (report only, core blocking, or full blocking).

Operating Rules

1. Every number comes from a run. Diff counts, ratios, and region numbers are quoted from the tool output or the test report of that run. The only other numbers allowed are the worked example values, which were produced by running the shipped tool on the shipped fixtures.
2. Threshold and budget are different knobs. Say which one you are changing and why. Raising the per-pixel threshold to hide a layout shift, or raising the ratio to hide a button change, is a dishonest fix.
3. Fix captures before budgets. A diff that moves between runs on the same code is a timing or environment fault until proven otherwise. Budgets come after the determinism checklist passes.
4. Masks stay narrow and visible. Mask the smallest rectangle that covers the dynamic element, state the reason, and watch the masked pixel count in reports. A mask that covers a component under test voids that capture.
5. Baselines are reviewed artifacts. No baseline is promoted without a person opening the diff, naming the intended change, and logging the approval. Batch approvals of unread diffs are forbidden.
6. Same environment both sides, always. Baselines generated on one OS or container and compared on another will diff on fonts alone. Regenerate baselines in the CI environment after any deliberate environment change and log it as a reset.
7. Size change means stale, not fail. When the current capture has different dimensions, the baseline cannot be compared. Regenerate it through the approval path. The shipped tool returns exit code 2 for this case so CI can route it separately.
8. Keep the review load human. If reviewers cannot read every diff in a normal pass, cut capture points or split the suite. An unreviewed suite that auto-approves is worse than a small suite that gets read.