anomaly_detection_basics v1.0.0

name: "anomaly_detection_basics" description: "Detect anomalies in agent-watched metric streams (costs, latencies, traffic): z-score, IQR fences, and rolling/median baselines, plus when to alert vs just log. Triggers on: detect anomaly, alert on metric spike, watch this stream, rolling baseline, flag outliers, reduce false alarms. Distinct from metric-definitions, which is about defining what to measure — this is the detection layer: how to compute whether the newest point is anomalous." metadata: { "includeInPrompt": true }

Purpose

Given an agent watching a metric stream (latency, cost, traffic, error rate), decide whether the newest point is anomalous and whether it deserves an alert or just a log entry. It covers three detection methods (z-score, IQR, rolling baselines), how to pick one, and a triage ladder (alert / log / suppress) that keeps false alarms low.

Workflow

1. Baseline first, score second. Collect a trailing window before scoring anything. Rule of thumb: window ≥ ~30 points or one full cycle of the metric (hourly metric → 24-point window; minute metric → 60-point window). Do not run detection on a partially filled window — that gap is bootstrap time, not clean data.
2. Pick the method by the data's shape.

• z-score — roughly symmetric data with no heavy tail. Default threshold k=3.0.
• IQR fences — skewed or heavy-tailed data (p99 latency, per-customer cost, request sizes). Default 1.5×IQR.
• Rolling baseline — drifting data (growing traffic, rising spend). Rolling mean for smooth drift; rolling median when spikes would contaminate the window.
• Counter streams (monotonically increasing counters): difference first to get rates, then detect on the deltas — never on the raw counter.

1. Score the candidate against the window, not inside it. Compute the baseline from the window excluding the newest point, then score the point. Including the candidate masks spikes (a big spike inflates σ/μ and hides itself).
2. Check seasonality before crying wolf. Compare against the same window last cycle (same hour yesterday, same weekday last week). If the deviation matches the seasonal pattern, it is expected — log, don't alert.
3. Triage: alert vs log vs suppress.

• ALERT (notify/page): business-impacting AND confirmed. Confirmation = two consecutive breaches, OR breach persists across your check interval, OR a second signal agrees (latency up AND error rate up).
• LOG (record, watch): first breach with no business impact yet, single-point spikes, anything you cannot confirm. Revisit on the next check; repeated logs escalate to alert.
• SUPPRESS (no action, reason noted): known events — deploys, cron spikes, marketing pushes, scheduled batch jobs. Record the reason so the suppression is auditable.

Short stdlib snippets (verified with python3):

python
import statistics

def z_anomalies(series, k=3.0):
    """Score each point against baseline; flag |z| > k. Pass the window
    WITHOUT the candidate, then score the candidate separately."""
    m, s = statistics.mean(series), statistics.stdev(series)
    return [x for x in series if abs((x - m) / s) > k], m, s

def iqr_outliers(series, mult=1.5):
    q1, _, q3 = statistics.quantiles(sorted(series), n=4)
    iqr = q3 - q1
    lo, hi = q1 - mult * iqr, q3 + mult * iqr
    return [x for x in series if x < lo or x > hi], lo, hi

def rolling_alert(window, new_point, k=3.0):
    """True if new_point breaches the trailing window's z-threshold."""
    m, s = statistics.mean(window), statistics.stdev(window)
    return abs((new_point - m) / s) > k, m, s

Usage pattern: keep a trailing window (e.g. collections.deque(maxlen=60)); on each new point call rolling_alert(list(window), point) — append the point only after scoring. For spike-contaminated data, compare the point against the window's median with a median-absolute-deviation band instead of mean/stdev.

Output Contract

Every detection pass reports:

• Baseline: method, window size, and parameters (μ/σ, IQR fences, or median band).
• Score: the candidate's deviation and the threshold it did or didn't cross.
• Verdict: ALERT / LOG / SUPPRESS — with the reason (confirmation evidence, seasonality match, or known-event cause).
• Action taken: who/what was notified for ALERT; where the entry landed for LOG; the suppression reason for SUPPRESS.

Operating Rules

• Never alert on a single unconfirmed point unless the value is impossible or catastrophic (negative latency, zero revenue mid-day, cost rate orders of magnitude above budget). When in doubt, LOG and watch the next point.
• Exclude the candidate from the baseline window. Scoring a point against statistics that include it is the most common silent failure.
• Recompute baselines continuously. A baseline from last week is a guess about this week on drifting metrics — use a trailing window, not a frozen one.
• Match the method to the shape. Symmetric → z-score; skewed/heavy-tailed → IQR; drifting → rolling; spiky-but-watchable → median baseline. A method mismatch (z-score on skewed latency) produces noise, not signal.
• Alerts must cite their evidence: baseline method + window, the threshold, and what confirmed it. An alert without numbers is a guess.
• Prefer log-and-watch over alerting when unsure. False alarms cost attention; a repeated log that still hasn't resolved is what escalation is for.
• Suppress only with a recorded reason. Scheduled events get a named suppression ("deploy 14:02", "weekly batch 02:00"); unexplained repeat deviations never get suppressed.