---
name: metric_baselining
description: Sets an honest, seasonality-aware baseline for a metric before a campaign or experiment. Computes typical-range bands, day-of-week and seasonal indexes, confidence intervals, and produces a pre-registration record with decision rules. Triggers when someone needs a baseline before launching a test, reading a campaign result, or deciding whether a change in a metric is real.
---

# Metric Baselining

## Purpose
Produce a defensible "what would this metric have done anyway" before you spend money or run a test. A baseline is not one number. It is a typical-range band, a seasonal expectation, a statement of uncertainty, and a pre-registered decision rule, all written down before the campaign starts so nobody moves the goalposts after.

## Workflow
1. **Pick the metric and define it exactly.** One operational definition: what counts, over what window, for which segment. If the definition changed recently (new attribution, new conversion event), the old history is not comparable until you audit it.
2. **Gather enough history.** Minimum: 4 full weeks of daily data for day-of-week effects. Seasonal category: a full year. Thin history gives a jumpy baseline, so say so in the record instead of pretending.
3. **Compute the baseline record.** Run `bin/baseline.py` on a CSV of `date,value` rows. It reports:
   - Overall mean, daily standard deviation, and observation count.
   - 95% confidence interval for the mean (mean plus or minus 1.96 x std / sqrt(n)). This is the uncertainty of your center, not of a single day.
   - Typical-range band: the 25th to 75th percentile of daily values. A future day landing inside this band is "business as usual", not a win or a loss.
   - Day-of-week indexes: each weekday's mean divided by the overall mean. Use these to seasonally adjust the expectation for a future window.
   - A seasonally adjusted expected range for a named future window (e.g. "Nov 1 to Nov 30"): expected mean and typical-range band scaled by the relevant weekday mix or prior-year seasonal factor.
4. **Set the decision rule and pre-register it.** Before launch, write down: the primary metric, the baseline record (value plus date range), the minimum change that would count as real, how long you will run, and the exact ship / kill / extend criteria. Lock it. No peeking and no early stops: stopping when the interval "looks good" biases the conclusion.
5. **After the run, compare against the record, not your memory.** Actual versus the pre-registered expected range. If it lands inside the band, it is noise. If it lands outside, check the confounds list (seasonality, definition changes, external events) before claiming the win.

## Output Contract
Every baseline the skill produces ends as a **Baseline Record**:

```markdown
## Baseline Record: {metric name}
- Definition: {what counts, window, segment}
- History: {date range, N daily observations}
- Typical range: {P25} to {P75} per day
- 95% CI of the mean: {lo} to {hi}
- Day-of-week effect: {e.g. Sat/Sun run 1.4x weekday}
- Expected range for {future window}: {lo} to {hi}
- Volume floor: below {N} observations per day, do not judge
- Pre-registered decision: ship if {condition}, kill if {condition}, extend if {condition}
```

## Operating Rules
1. Never baseline a metric whose definition is unknown or recently changed without flagging it. "Converted" meaning three different things over three months is three baselines, not one.
2. Compare like to like: same weekday, same season, same segment. A December holiday week is not comparable to a quiet January week, and weekend traffic is not comparable to weekday traffic.
3. Bands before verdicts. A single number invites argument; a band plus a confidence interval keeps everyone honest about uncertainty.
4. Volume floors are mandatory for rate metrics. Conversion rate from 40 sessions tells you nothing; set the floor and refuse to judge below it.
5. Pre-register before the launch, not after the first interesting result. A baseline written after you saw the outcome is a rationalization, not a baseline.
6. Uncertainty grows the further out you project. A credible read on "what would next week look like" is worth more than a precise-sounding forecast of next quarter.
7. If the history is too thin for a real baseline, say so plainly and widen the bands. A weak baseline is worse than an honest "we don't know yet."