metric_baselining v1.0.0

name: metric_baselining description: Sets an honest, seasonality-aware baseline for a metric before a campaign or experiment. Computes typical-range bands, day-of-week and seasonal indexes, confidence intervals, and produces a pre-registration record with decision rules. Triggers when someone needs a baseline before launching a test, reading a campaign result, or deciding whether a change in a metric is real.

Metric Baselining

Purpose

Produce a defensible "what would this metric have done anyway" before you spend money or run a test. A baseline is not one number. It is a typical-range band, a seasonal expectation, a statement of uncertainty, and a pre-registered decision rule, all written down before the campaign starts so nobody moves the goalposts after.

Workflow

1. Pick the metric and define it exactly. One operational definition: what counts, over what window, for which segment. If the definition changed recently (new attribution, new conversion event), the old history is not comparable until you audit it.
2. Gather enough history. Minimum: 4 full weeks of daily data for day-of-week effects. Seasonal category: a full year. Thin history gives a jumpy baseline, so say so in the record instead of pretending.
3. Compute the baseline record. Run bin/baseline.py on a CSV of date,value rows. It reports:

• Overall mean, daily standard deviation, and observation count.
• 95% confidence interval for the mean (mean plus or minus 1.96 x std / sqrt(n)). This is the uncertainty of your center, not of a single day.
• Typical-range band: the 25th to 75th percentile of daily values. A future day landing inside this band is "business as usual", not a win or a loss.
• Day-of-week indexes: each weekday's mean divided by the overall mean. Use these to seasonally adjust the expectation for a future window.
• A seasonally adjusted expected range for a named future window (e.g. "Nov 1 to Nov 30"): expected mean and typical-range band scaled by the relevant weekday mix or prior-year seasonal factor.

1. Set the decision rule and pre-register it. Before launch, write down: the primary metric, the baseline record (value plus date range), the minimum change that would count as real, how long you will run, and the exact ship / kill / extend criteria. Lock it. No peeking and no early stops: stopping when the interval "looks good" biases the conclusion.
2. After the run, compare against the record, not your memory. Actual versus the pre-registered expected range. If it lands inside the band, it is noise. If it lands outside, check the confounds list (seasonality, definition changes, external events) before claiming the win.

Output Contract

Every baseline the skill produces ends as a Baseline Record:

markdown
## Baseline Record: {metric name}
- Definition: {what counts, window, segment}
- History: {date range, N daily observations}
- Typical range: {P25} to {P75} per day
- 95% CI of the mean: {lo} to {hi}
- Day-of-week effect: {e.g. Sat/Sun run 1.4x weekday}
- Expected range for {future window}: {lo} to {hi}
- Volume floor: below {N} observations per day, do not judge
- Pre-registered decision: ship if {condition}, kill if {condition}, extend if {condition}

Operating Rules

1. Never baseline a metric whose definition is unknown or recently changed without flagging it. "Converted" meaning three different things over three months is three baselines, not one.
2. Compare like to like: same weekday, same season, same segment. A December holiday week is not comparable to a quiet January week, and weekend traffic is not comparable to weekday traffic.
3. Bands before verdicts. A single number invites argument; a band plus a confidence interval keeps everyone honest about uncertainty.
4. Volume floors are mandatory for rate metrics. Conversion rate from 40 sessions tells you nothing; set the floor and refuse to judge below it.
5. Pre-register before the launch, not after the first interesting result. A baseline written after you saw the outcome is a rationalization, not a baseline.
6. Uncertainty grows the further out you project. A credible read on "what would next week look like" is worth more than a precise-sounding forecast of next quarter.
7. If the history is too thin for a real baseline, say so plainly and widen the bands. A weak baseline is worse than an honest "we don't know yet."