ab_testing v1.0.0

name: "ab_testing" description: "Design, power, and read A/B tests without fooling yourself. Use when someone wants to run an experiment: hypothesis design, sample-size math, guardrail metrics, pre-registered segments, peeking discipline, SRM checks, and honest result readouts."

A/B Testing Done Right

Purpose

Turn a product question into an experiment that produces a decision you can trust. This skill covers the full discipline: writing a real hypothesis, doing the power math before launch, protecting the read with guardrails and honest segmentation, and reading results without fooling yourself.

Workflow

1. Write the hypothesis in this format: "Because [observation or insight], we believe [change] will cause [result] for [audience], measured by [metric]." A test without a stated hypothesis cannot produce learning, only noise.

1. Pick one primary metric and a short list of guardrails. The primary metric decides winner vs loser. Guardrails are metrics that must not get worse (revenue per visitor, error rates, page load time). Cap the total at a handful; every extra metric is another chance at a false positive.

1. Set the minimum detectable effect (MDE) from business meaning, not from wishful thinking. MDE is the smallest lift that would justify shipping. A 10% baseline and a 1% absolute MDE means you need roughly 15,000 visitors per variant at 80% power. If the math says the test needs 200,000 users you don't have, raise the MDE or don't run the test.

1. Run the sample-size calculator. Use the bundled CLI (bin/ab_sample_size.py): two-proportion z-test, defaults α = 0.05 two-sided, power = 0.80. Feed it baseline rate, relative MDE, and daily traffic for a duration estimate.

1. Pre-register the segments before launch. New vs returning, mobile vs desktop, free vs paid, whatever matters. Segments declared after seeing the data are just p-hacking with a nicer name.

1. Pre-flight checks: run an A/A test (both variants identical) for about 24 hours. If it shows a significant difference, fix the randomization or the instrumentation before proceeding. Randomize at the user level, not the session level, so the same person keeps seeing the same variant.

1. Run it without peeking. Either set the sample size, walk away, and read once at the end; or use a proper sequential method (mSPRT always-valid p-values, or group-sequential with alpha-spending). Stopping because "p < 0.05 today" inflates your real false-positive rate to 26% or more with continuous monitoring.

1. Read the results in order: primary metric first, then guardrails, then pre-registered segments, then nothing else. A win on the primary that tanks a guardrail is a loss. A win that only appears in one unplanned segment is a hypothesis for the next test, not a ship decision.

Output Contract

A test plan must contain: hypothesis, primary metric, guardrail metrics, baseline rate, MDE (relative and absolute), α, power, sample size per variant, planned duration, pre-registered segments, the peeking policy (fixed horizon or sequential method), and the ship/no-ship criteria written before launch.

A result readout must contain: actual sample per variant, an SRM check result, primary metric outcome with confidence interval, guardrail outcomes, segment results limited to pre-registered segments, and the ship/no-ship decision with the reasoning.

Operating Rules

• Peeking math (Evan Miller's table, true 5% level): 1 peek needs a nominal 2.9% to stay honest, 2 peeks 2.2%, 3 peeks 1.8%, 5 peeks 1.4%, 10 peeks 1.0%. If you don't have a sequential method running, don't look.
• Sample ratio mismatch: after launch, chi-square the actual split against the intended split. A 50/50 test running 60/40 has a randomization or instrumentation bug. Common culprit: client-side redirect tests where the redirect drops users asymmetrically.
• Multiple comparisons stack with peeking. Both corrections apply; most platforms apply sequential correction but not a formal Bonferroni across guardrails. Know which guarantees your platform actually gives you.
• Duration floor and ceiling: minimum 7 days to cover weekly seasonality, maximum 4 to 6 weeks before novelty effects fade and the setup goes stale.
• Novelty vs primacy: returning visitors may underperform on a superior change until they adjust. Check new vs returning separately before blaming the variant.
• Program-level false positives are a fact of life. At 5% significance, 100 null experiments produce about 5 false positives by chance. Chasing program-wide correction would demand p < 0.0005 and crush power; instead track win rates over time and require guardrail-clean wins.
• When not to A/B test: the change is forced (security, legal, outage), the sample math says it will take months, or the outcome is obviously determined by one confounded segment. Use judgment instead.
• Bayesian option: if the team needs to look mid-test, a Bayesian readout (probability variant B is best, expected loss of shipping) stays interpretable at any point. A frequentist p-value does not.