---
name: "anomaly_detection_basics"
description: "Detect anomalies in agent-watched metric streams (costs, latencies, traffic): z-score, IQR fences, and rolling/median baselines, plus when to alert vs just log. Triggers on: detect anomaly, alert on metric spike, watch this stream, rolling baseline, flag outliers, reduce false alarms. Distinct from metric-definitions, which is about defining what to measure — this is the detection layer: how to compute whether the newest point is anomalous."
metadata: { "includeInPrompt": true }
---

## Purpose

Given an agent watching a metric stream (latency, cost, traffic, error rate), decide whether the newest point is anomalous and whether it deserves an alert or just a log entry. It covers three detection methods (z-score, IQR, rolling baselines), how to pick one, and a triage ladder (alert / log / suppress) that keeps false alarms low.

## Workflow

1. **Baseline first, score second.** Collect a trailing window before scoring anything. Rule of thumb: window ≥ ~30 points or one full cycle of the metric (hourly metric → 24-point window; minute metric → 60-point window). Do not run detection on a partially filled window — that gap is bootstrap time, not clean data.
2. **Pick the method by the data's shape.**
   - **z-score** — roughly symmetric data with no heavy tail. Default threshold k=3.0.
   - **IQR fences** — skewed or heavy-tailed data (p99 latency, per-customer cost, request sizes). Default 1.5×IQR.
   - **Rolling baseline** — drifting data (growing traffic, rising spend). Rolling mean for smooth drift; rolling **median** when spikes would contaminate the window.
   - **Counter streams** (monotonically increasing counters): difference first to get rates, then detect on the deltas — never on the raw counter.
3. **Score the candidate against the window, not inside it.** Compute the baseline from the window *excluding* the newest point, then score the point. Including the candidate masks spikes (a big spike inflates σ/μ and hides itself).
4. **Check seasonality before crying wolf.** Compare against the same window last cycle (same hour yesterday, same weekday last week). If the deviation matches the seasonal pattern, it is expected — log, don't alert.
5. **Triage: alert vs log vs suppress.**
   - **ALERT (notify/page):** business-impacting AND confirmed. Confirmation = two consecutive breaches, OR breach persists across your check interval, OR a second signal agrees (latency up AND error rate up).
   - **LOG (record, watch):** first breach with no business impact yet, single-point spikes, anything you cannot confirm. Revisit on the next check; repeated logs escalate to alert.
   - **SUPPRESS (no action, reason noted):** known events — deploys, cron spikes, marketing pushes, scheduled batch jobs. Record the reason so the suppression is auditable.

Short stdlib snippets (verified with python3):

```python
import statistics

def z_anomalies(series, k=3.0):
    """Score each point against baseline; flag |z| > k. Pass the window
    WITHOUT the candidate, then score the candidate separately."""
    m, s = statistics.mean(series), statistics.stdev(series)
    return [x for x in series if abs((x - m) / s) > k], m, s

def iqr_outliers(series, mult=1.5):
    q1, _, q3 = statistics.quantiles(sorted(series), n=4)
    iqr = q3 - q1
    lo, hi = q1 - mult * iqr, q3 + mult * iqr
    return [x for x in series if x < lo or x > hi], lo, hi

def rolling_alert(window, new_point, k=3.0):
    """True if new_point breaches the trailing window's z-threshold."""
    m, s = statistics.mean(window), statistics.stdev(window)
    return abs((new_point - m) / s) > k, m, s
```

Usage pattern: keep a trailing window (e.g. `collections.deque(maxlen=60)`); on each new point call `rolling_alert(list(window), point)` — append the point only after scoring. For spike-contaminated data, compare the point against the window's median with a median-absolute-deviation band instead of mean/stdev.

## Output Contract

Every detection pass reports:
- **Baseline:** method, window size, and parameters (μ/σ, IQR fences, or median band).
- **Score:** the candidate's deviation and the threshold it did or didn't cross.
- **Verdict:** ALERT / LOG / SUPPRESS — with the reason (confirmation evidence, seasonality match, or known-event cause).
- **Action taken:** who/what was notified for ALERT; where the entry landed for LOG; the suppression reason for SUPPRESS.

## Operating Rules

- **Never alert on a single unconfirmed point** unless the value is impossible or catastrophic (negative latency, zero revenue mid-day, cost rate orders of magnitude above budget). When in doubt, LOG and watch the next point.
- **Exclude the candidate from the baseline window.** Scoring a point against statistics that include it is the most common silent failure.
- **Recompute baselines continuously.** A baseline from last week is a guess about this week on drifting metrics — use a trailing window, not a frozen one.
- **Match the method to the shape.** Symmetric → z-score; skewed/heavy-tailed → IQR; drifting → rolling; spiky-but-watchable → median baseline. A method mismatch (z-score on skewed latency) produces noise, not signal.
- **Alerts must cite their evidence:** baseline method + window, the threshold, and what confirmed it. An alert without numbers is a guess.
- **Prefer log-and-watch over alerting when unsure.** False alarms cost attention; a repeated log that still hasn't resolved is what escalation is for.
- **Suppress only with a recorded reason.** Scheduled events get a named suppression ("deploy 14:02", "weekly batch 02:00"); unexplained repeat deviations never get suppressed.