benchmark_methodology v1.0.0

name: "benchmark_methodology" description: "Design and run trustworthy benchmarks for LLM and agent systems. Task selection, scoring rubrics, statistical rigor, and anti-gaming defenses. Use when building an eval suite, choosing between models or agents, writing a benchmark paper, or deciding whether a leaderboard gap means anything."

Benchmark Methodology

Purpose

A single benchmark number is not evidence. This skill takes you from a vague "let's compare models" to a benchmark whose scores a skeptic can trust: tasks that match the real job, scoring that cannot be gamed, statistics that say whether the gap is real, and defenses against contamination. It covers LLM benchmarks and agentic benchmarks (tool use, multi-step tasks, coding agents).

Workflow

1. Fix the decision the benchmark serves

Start by writing one sentence: "This benchmark exists to decide ___." A benchmark that does not name its decision drifts. Common decisions: pick a model for a support bot, compare two agents on real tasks, publish a claim, block a regression. The decision shapes everything: task difficulty, acceptable cost per run, and how rigorous the stats must be.

2. Define scenarios before metrics

List 3 to 6 concrete usage scenarios (who uses the system, for what purpose, in what context). One summarization benchmark does not serve legal and social-media scenarios the same way. Per scenario, write down 2 to 4 desiderata beyond accuracy: latency, cost, robustness to messy input, calibration (does it know when it is wrong), fairness, refusal behavior. A benchmark that measures only accuracy will pick a model that is accurate and wrong in every other way. See HELM's design principles in the references.

3. Select tasks

• Pull from real work, not from what is easy to grade. Sample from production logs, support tickets, or domain expert exercises.
• Stratify by difficulty (easy / medium / hard) and by sub-domain, the way GAIA uses three levels and tau-bench splits airline vs retail. A flat task list lets one sub-domain quietly dominate the score.
• Keep a public dev split for iteration and a private test split for final numbers, roughly 20/80. Publish the dev split, hold the test answers. GAIA ships 466 public questions with a separate private 300-question test set for exactly this reason.
• Size the task set to the precision you need. Rough rule: to distinguish a 5-point gap at 95% confidence on a binary metric you need a few hundred tasks; for a 1-point gap you need thousands. If you cannot afford that many, report the gap as unresolved rather than crowning a winner.
• Document what the benchmark does NOT cover. HELM's principle: name the major scenarios and metrics that are missing. An honest blind spot beats a silently skewed benchmark.

4. Choose scoring from the trust ladder

Pick the most objective scoring each task allows, from strongest to weakest:

1. Closed-form answers with exact, numeric, or F1 match. Most tool-agnostic, hardest to game.
2. Checker scripts over the end state (tests pass, file contents correct, database state correct), run in a fresh disposable sandbox per trial, like SWE-bench.
3. Structured rubrics scored by an LLM judge, with the judge model from a different model family than the candidates, and both the judge prompt and scores logged. Rubric criteria must be checkable against the task artifact, not the model's process. (See Agentic Rubrics: rubrics built by exploring the actual context align with ground-truth tests at high ROC-AUC and catch failures tests miss.)

Never let the model being evaluated also be the only judge of itself. If you must use a judge, use two judges from different families and show both judgments side by side instead of averaging.

5. Add anti-gaming defenses

• Plant a canary string in the benchmark (the BIG-bench canary pattern) and ask that it be excluded from training data.
• Run contamination checks before publishing results: n-gram overlap (n = 8 to 13 words; report the n) between the benchmark and known pretraining corpora or the model's web-visible training cutoff, exact-match hashing on normalized text, and near-duplicate detection.
• Prefer fresh or post-cutoff tasks for high-stakes comparisons. A public benchmark older than the model's training cutoff is a memorization test, not a capability test.
• Refresh the private slice periodically. Public benchmarks saturate the internet; the score's meaning decays as contamination rises.
• Do not send private benchmark items to closed APIs that train on user data. A systematic review found roughly 42% of 225 evaluated papers leaked benchmark data to GPT-3.5/4 class APIs (Balloccu et al., 2024).
• Full checklist in references/anti-contamination.md.

6. Report with statistics, not vibes

• 95% bootstrap confidence intervals (at least 1000 resamples) on every headline metric. Report as mean +/- CI.
• For comparing two systems on the same task set, use a paired test: McNemar's test for binary outcomes (it looks only at items where the models disagree; a one-item edge out of 300 gives p ~ 1.0, i.e., nothing). For continuous metrics, paired bootstrap or permutation tests. p < 0.05 after Bonferroni correction when you run many comparisons.
• Run each candidate at least 3 times to separate within-model noise from between-model differences. The gap must exceed the model's own variance to mean anything.
• Resample at the scenario or project level, not per item, when tasks are correlated (same source, same template).
• Report effect sizes alongside p-values (Cohen's d or Cliff's delta) so readers see whether a significant gap is large enough to care about.
• A worked calculator is in bin/bench_stats.py; method details and formulas are in references/statistical-rigor.md.

7. Publish the benchmark properly

A benchmark result is reproducible only if someone else can rerun it. Ship: the task set with version number, the exact prompts and sampling parameters, the scoring code, the environment (library versions, seeds), per-task scores for every candidate (not just aggregates), and the contamination report. If a leaderboard gives only three-decimal aggregates, it is giving false precision.

Output Contract

When the user asks for a benchmark design, produce a filled copy of assets/benchmark-plan-template.md: the decision sentence, scenarios, task selection plan with splits, scoring choice per scenario from the trust ladder, anti-gaming defenses, the statistical plan (run counts, tests, CIs), and the publication checklist. When asked to read someone else's benchmark or leaderboard, produce a verdict memo: what was measured, whether the stats support the claim (gap vs noise floor), contamination risk, and what a trustworthy number would take.

Operating Rules

1. Never invent numbers. A benchmark needs a cutoff date, corpus size, or cost figure only if you can verify it. Otherwise mark it "verify before publishing."
2. Do not crown a winner on a gap inside the confidence interval. Say "unresolved at this sample size" and suggest the runs needed.
3. Do not present a contaminated comparison as capability evidence. If the benchmark predates the model's training cutoff and no contamination check was run, the number is unverified.
4. Judge models must be disclosed: family, version, prompt. An undisclosed judge makes the score unverifiable.
5. Prefer fewer, harder, fresher tasks over thousands of stale ones. Quality of tasks beats quantity.