metric_definitions v1.0.0

name: "metric_definitions" description: "Define the metric before you build the dashboard: unambiguous metric definitions (population, event, window, filters, edge cases), event-naming conventions, common traps (double-counting, timezone windows, null-vs-zero, sampling bias), a one-page fill-in template, and sanity-checking numbers that look wrong. Triggers on: define a metric, how should we measure X, KPI definition, metric definition, metric looks wrong, dashboard numbers disagree." metadata: { "includeInPrompt": true }

Purpose

Turn "how many active users?" into a written, reviewable definition before anyone queries or charts. Most dashboard disagreements are not query bugs — they are two people counting different things under the same name. Scope is the definition discipline (unit, population, window, edge cases), not charting, BI tooling, or pipeline code.

Workflow

1. Name the counted unit. "Orders" (events) and "ordering users" (distinct people) are different metrics. Demo on 5 sample order rows, one user ordering 3×: COUNT() → 5, COUNT(DISTINCT user_id) → 3. If the unit isn't in the name, it isn't defined.
2. Fill the five-field definition (template below): population, event, window, filters, edge cases. Every field must be answerable; "n/a" is only allowed for edge cases with a reason.
3. Name events object_verb in past tense (order_placed, not order or placeOrder). One event = one occurrence; dedupe by event id at write time, never by query-time guessing.
4. Resolve the four traps explicitly in the definition — each has a verified demo:

• Double-counting: see step 1 — 5 vs 3. Ratios need the same grain on both sides.
• Timezone windows: 5 orders, "Sep 18": UTC-day count → 4, America/Chicago-day count → 5 (one order at 2026-09-19T03:10Z = 22:10 CDT Sep 18). Store timestamps in UTC, convert at query time, state the zone in the definition.
• NULL vs zero: 5 sessions (120s, 0s, NULL, 60s, NULL) — AVG ignoring NULLs → 60.0s, treating NULL as 0 → 36.0s. NULL = unknown (bot, adblock, lost beacon); 0 = measured zero (bounced). Decide per field, write it down.
• Sampling bias:* a 1% sampled log still needs the sample rate in the definition; extrapolations must state the multiplier. Prefer full-population counts for small denominators (< ~30 distinct units — treat as directionally indicative, not precise).

1. Sanity-check any number that looks wrong before debugging the query: (a) recompute at a coarser grain — does daily sum to the weekly? (b) check the denominator — is the population what you think? (c) compare two windows — did a deploy or holiday move it? (d) look for duplicate event ids; (e) re-run in the definition's stated timezone. In most cases one of these explains the surprise.

Output Contract

After applying this skill, an agent should produce:

• A filled metric-definition template (one page):
• Metric name: <unit>_<what>[_<window>] (unit in the name)
• Plain-language meaning: one sentence a non-analyst can repeat
• Population: who/what is eligible to be counted (and who is excluded)
• Event: the exact event or state change, with its name per the convention
• Window: period + timezone (e.g. "rolling 7d, America/Chicago, cutoff 00:00 local")
• Filters: segments, exclusions, test-account handling
• Edge cases: duplicates, NULLs, late-arriving events, backfills — decision for each
• Owner + last reviewed: who re-verifies this and when
• A one-line trap checklist showing the chosen resolution for each of the four traps.
• The verification grain: what unit was counted, over what population and window, so a second person can reproduce the number.

Operating Rules

• Write the definition before the query. No dashboard, no chart, no "quick count" until the five fields exist in writing.
• The unit goes in the name. If two readings exist, the metric has two names.
• Timestamps are UTC at rest, local at query. The definition always names the timezone; "daily" without a zone is not a definition.
• NULL is unknown, 0 is measured. Never COALESCE one into the other without saying so in the definition.
• Prefer small, reviewable definitions over exhaustive ones — a one-page definition people actually read beats a wiki nobody opens.
• Soften absolutes. "Prefer full-population counts below ~30" not "never extrapolate"; "usually dedupe by event id" not "always" — the right call depends on the pipeline you actually have.