---
name: "ab_testing"
description: "Design, power, and read A/B tests without fooling yourself. Use when someone wants to run an experiment: hypothesis design, sample-size math, guardrail metrics, pre-registered segments, peeking discipline, SRM checks, and honest result readouts."
---

# A/B Testing Done Right

## Purpose

Turn a product question into an experiment that produces a decision you can
trust. This skill covers the full discipline: writing a real hypothesis, doing
the power math before launch, protecting the read with guardrails and honest
segmentation, and reading results without fooling yourself.

## Workflow

1. **Write the hypothesis in this format:** "Because [observation or insight],
   we believe [change] will cause [result] for [audience], measured by [metric]."
   A test without a stated hypothesis cannot produce learning, only noise.

2. **Pick one primary metric and a short list of guardrails.** The primary
   metric decides winner vs loser. Guardrails are metrics that must not get
   worse (revenue per visitor, error rates, page load time). Cap the total at a
   handful; every extra metric is another chance at a false positive.

3. **Set the minimum detectable effect (MDE) from business meaning, not from
   wishful thinking.** MDE is the smallest lift that would justify shipping.
   A 10% baseline and a 1% absolute MDE means you need roughly 15,000 visitors
   per variant at 80% power. If the math says the test needs 200,000 users you
   don't have, raise the MDE or don't run the test.

4. **Run the sample-size calculator.** Use the bundled CLI
   (`bin/ab_sample_size.py`): two-proportion z-test, defaults α = 0.05
   two-sided, power = 0.80. Feed it baseline rate, relative MDE, and daily
   traffic for a duration estimate.

5. **Pre-register the segments before launch.** New vs returning, mobile vs
   desktop, free vs paid, whatever matters. Segments declared after seeing the
   data are just p-hacking with a nicer name.

6. **Pre-flight checks:** run an A/A test (both variants identical) for about
   24 hours. If it shows a significant difference, fix the randomization or the
   instrumentation before proceeding. Randomize at the user level, not the
   session level, so the same person keeps seeing the same variant.

7. **Run it without peeking.** Either set the sample size, walk away, and read
   once at the end; or use a proper sequential method (mSPRT always-valid
   p-values, or group-sequential with alpha-spending). Stopping because
   "p < 0.05 today" inflates your real false-positive rate to 26% or more with
   continuous monitoring.

8. **Read the results in order:** primary metric first, then guardrails, then
   pre-registered segments, then nothing else. A win on the primary that tanks
   a guardrail is a loss. A win that only appears in one unplanned segment is
   a hypothesis for the next test, not a ship decision.

## Output Contract

A test plan must contain: hypothesis, primary metric, guardrail metrics,
baseline rate, MDE (relative and absolute), α, power, sample size per variant,
planned duration, pre-registered segments, the peeking policy (fixed horizon or
sequential method), and the ship/no-ship criteria written before launch.

A result readout must contain: actual sample per variant, an SRM check result,
primary metric outcome with confidence interval, guardrail outcomes, segment
results limited to pre-registered segments, and the ship/no-ship decision with
the reasoning.

## Operating Rules

- **Peeking math (Evan Miller's table, true 5% level):** 1 peek needs a nominal
  2.9% to stay honest, 2 peeks 2.2%, 3 peeks 1.8%, 5 peeks 1.4%, 10 peeks 1.0%.
  If you don't have a sequential method running, don't look.
- **Sample ratio mismatch:** after launch, chi-square the actual split against
  the intended split. A 50/50 test running 60/40 has a randomization or
  instrumentation bug. Common culprit: client-side redirect tests where the
  redirect drops users asymmetrically.
- **Multiple comparisons stack with peeking.** Both corrections apply; most
  platforms apply sequential correction but not a formal Bonferroni across
  guardrails. Know which guarantees your platform actually gives you.
- **Duration floor and ceiling:** minimum 7 days to cover weekly seasonality,
  maximum 4 to 6 weeks before novelty effects fade and the setup goes stale.
- **Novelty vs primacy:** returning visitors may underperform on a superior
  change until they adjust. Check new vs returning separately before blaming
  the variant.
- **Program-level false positives are a fact of life.** At 5% significance,
  100 null experiments produce about 5 false positives by chance. Chasing
  program-wide correction would demand p < 0.0005 and crush power; instead
  track win rates over time and require guardrail-clean wins.
- **When not to A/B test:** the change is forced (security, legal, outage),
  the sample math says it will take months, or the outcome is obviously
  determined by one confounded segment. Use judgment instead.
- **Bayesian option:** if the team needs to look mid-test, a Bayesian readout
  (probability variant B is best, expected loss of shipping) stays interpretable
  at any point. A frequentist p-value does not.