evals_for_agents v1.0.0

name: "evals_for_agents" description: "Builds and maintains eval suites for AI agent work: golden datasets, task-specific rubrics, deterministic and LLM-judge graders, regression gates, and spotting when a metric lies. Triggers when the user asks to evaluate an agent, write evals, build a golden dataset, create a scoring rubric, add regression tests, or debug a failing quality gate."

Evals for Agents

Purpose

Stand up evals that answer one question: did this change make the agent better or worse? Covers golden datasets, rubrics, graders, baselines, CI gates, and the lies metrics tell.

Workflow

When asked to BUILD evals

1. Define the task space. List what the agent must do and the failure modes to guard. Per Anthropic's "Demystifying Evals for AI Agents": a task is one test with defined inputs and success criteria, a trial is one attempt (run several, outputs vary), a grader is logic scoring one aspect of performance, the transcript is the full record of a trial, and the outcome is the final state of the environment, not the agent's claim. You are evaluating the scaffold plus the model, not the model alone.
2. Collect a golden dataset. Pull 20 to 100 real inputs from actual usage, tickets, or support logs. Label each with an expected output or a clear pass condition. Include common cases, edge cases, adversarial cases, known past failures, and negative cases where the right answer is refusal, escalation, or "not enough information." See references/suite-anatomy.md for the record schema and the three-suite split (golden regression, hard progress, red-team).
3. Choose the cheapest valid grader per case. Deterministic checks first: exact/structural match for labels and JSON fields, state assertions against the environment (database rows, files, API state) instead of the agent's message text, and a golden trajectory envelope (required/forbidden tool calls as partial-order constraints, never one exact path). LLM-as-judge only where ground truth does not exist, with a written rubric per task. Recipes in references/grader-recipes.md; rubric patterns in references/rubric-patterns.md.
4. Calibrate every judge. Human-label a sample of cases, measure judge-human agreement, and fix the rubric or swap the judge below about 85% agreement. Report the agreement number. A judge model should differ from the generator's model family where possible, and every verdict must quote the evidence it rests on. An uncalibrated judge is noise with confidence.
5. Run and baseline. Run the suite, record per-case and aggregate scores, plus cost telemetry (tokens, latency, p50/p95). That number is the line no change crosses downward without a decision.
6. Gate deploys. Wire the suite into CI so any prompt, model, or tool change runs the evals and blocks on regression. Seed one deliberately broken case periodically and confirm the gate actually fails on it; an untested gate is decoration.
7. Close the loop. Every production failure, thumbs-down, or HITL correction becomes a new golden-dataset case. A static eval set decays as traffic shifts.

When asked to REVIEW an eval setup

Ask, in order: is the dataset representative of real usage and failure modes? Are deterministic graders used wherever a check can be mechanical? Is every judge calibrated with a reported agreement number? Are thresholds and CI gates actually wired? Is everything versioned so a score is reproducible? Has anyone read a transcript this week? Any "no" is the next piece of work.

When asked to ADD a case

Turn the production failure into a golden-dataset record with input, expected behavior, and a grader. Run it once to confirm it catches the failure, then add it to the regression set.

Output Contract

• For a build: versioned golden dataset (JSONL per references/suite-anatomy.md), grader definitions, calibration report with agreement numbers, baseline run with per-case scores, and a CI gate definition that was verified to fail on a seeded regression.
• For a review: a checklist of findings keyed to the questions above, each with the concrete fix.
• For a new case: the dataset record plus the before/after scores proving it catches the failure.

Operating Rules

1. No invented numbers, dates, quotes, or expected outputs. Every factual claim in a rubric or dataset record is verified or cut.
2. Never grade with a single 1-10 "overall quality" call. Decompose into narrow binary or anchored criteria.
3. Judge prompts always include an "unknown" escape hatch and require a quoted evidence span for each verdict.
4. Gate on confidence intervals and per-slice regressions, not only aggregate means. Report pass@k (succeeds in at least one of k trials) and pass^k (succeeds in all k trials) where retrying is part of the product; never report a bare pass rate without saying which.
5. Keep eval scaffold and production scaffold identical. State must not leak between trials.
6. A suite at 100% is a saturated suite: it has no signal left. Add harder cases or rotate canaries.
7. Never claim evals are done from a plan alone. An eval suite you have not run is a plan, not a result.