experiment_design v1.0.0

name: "experiment_design" description: "Design cheap product experiments that measure real behavior before anything is built: pretotype recipes (fake door, Wizard of Oz, concierge, Mechanical Turk), XYZ hypotheses, sample-size honesty, and kill-or-continue decision rules. Use when testing whether a product idea is worth building, choosing how to test demand or willingness to pay, or deciding what to do after a test result."

Experiment Design

Purpose

Stop ideas from dying expensively. Most new products fail in the market, not in the lab, even when competently executed (Savoia's Law of Market Failure). This skill turns a product idea into a cheap, time-boxed test of real customer behavior, picks the lightest experiment type that can answer the question, and gives you an honest decision rule before the data comes in. It sits downstream of assumption mapping (which finds WHAT to test) and upstream of A/B testing (which validates rigorously later); use the two companion skills there.

Workflow

1. Write the XYZ hypothesis

Force the idea into Savoia's form: At least X% of Y will do Z.

• Y: a specific audience (not "people"). "Suburban dog owners who buy food at independent pet stores."
• Z: an observable behavior, with skin in the game (not an opinion). Z with cash (paid, pre-order, deposit) beats Z with email, which beats Z with words.
• X%: the minimum that makes the idea worth building. Be honest, not heroic. If 15% conversion at $4 is what your model needs, write 15%.

Bad: "Dog owners will like a dog-safe beer." Untestable. Good: "At least 15% of dog owners who shop at the test store will buy a $4 sixpack of dog-safe beer with their dog food." (This is Savoia's own worked example from the pretotyping planning canvas.)

If the idea is too big to test in one shot, hypozoom: narrow it to the smallest testable component first.

2. Pick the lightest experiment type

Match the technique to the riskiest assumption, and escalate only when the cheaper test says yes. Full recipes live in references/pretotype-recipes.md.

| Riskiest assumption | Start here | Escalate to | |---|---|---| | Anyone cares at all | Pinocchio (show a non-working mock) or Fake Door (button for a feature that does not exist) | YouTube (video demo + pre-order) | | They will pay | Fake Door with real price shown, then Facade (sell a lesser version) | One-Night Stand (limited, real version) | | The experience works end to end | Concierge (do it manually for 1-3 customers) | Wizard of Oz (automated front, human back) | | They will use it repeatedly | Infiltrator (sell in an existing marketplace) | Relabel (resell a competitor's product under your positioning to test messaging) |

Rule of thumb: demand before usability. Never run a concierge or Wizard of Oz before a fake door says anyone wants the thing.

3. Pre-register the decision rule

Write this BEFORE the experiment runs:

• Time box: days, not weeks. Cap it: "7 days or $150, whichever comes first."
• Continue threshold: the X% from your XYZ hypothesis, measured the same way it is written.
• Kill-or-continue rule (full rules in references/decision-rules.md):
• Result at or above X% -> persevere. Move to the next riskiest assumption.
• Result at half of X% or worse -> kill the idea or pivot the offer. Do not re-run the same test hoping for better weather.
• Result between half and X% -> iterate on exactly one variable (price, audience, channel) and retest once. One retry, not five.
• Inconclusive (tiny sample, broken setup) -> hypozoom and run a smaller, cleaner test. Never read a broken test as a positive.

4. Report honestly

Report the behavior and the interval, never the point estimate alone. See references/decision-rules.md for the sample-size math. "3 of 30 bought it (10%; 95% CI roughly 0-21%)" is honest. "10% conversion rate!" on n=30 is a lie wearing a number.

Output Contract

Every experiment design produced with this skill includes:

1. XYZ hypothesis in the exact "At least X% of Y will do Z" form, with the skin-in-the-game behavior named.
2. Technique choice: which recipe, why it is the lightest one that answers the question, and what it costs in days and dollars.
3. Storyboard: the sequence of events in the experiment, how data is recorded, and how you follow up with people who showed interest.
4. Pre-registered decision rule: time box, spend cap, continue/kill/iterate thresholds, and the rule for inconclusive results.
5. Result readout: behavior measured, sample size, confidence interval, and the decision taken (persevere / pivot / kill), with the threshold comparison shown.

Operating Rules

1. Measure behavior, not opinions. Surveys and interviews explain; they do not validate. If Z is "says they like it," rewrite the hypothesis.
2. The hypothesis exists to be falsified. A test that cannot fail is theater, not an experiment.
3. Pre-register thresholds before collecting data. Moving the goalpost after seeing the numbers invalidates the test.
4. Sample-size honesty is mandatory: report confidence intervals, name what the sample cannot distinguish, and never round an inconclusive result up.
5. One variable per iteration. If you change price, audience, and copy at once, you learned nothing.
6. No confident unverified claims in any experiment writeup: numbers, dates, and market facts come from a real source or are cut.