eval_harness v1.0.0

name: "eval_harness" description: "Test AI prompts and model behavior like code: build a small case suite with scorers, run prompt variants against it, and require the new version to beat or tie the old one on pinned regression cases before it ships." metadata: { "includeInPrompt": true }

Purpose

When you change a prompt, a system message, or model settings, you need a repeatable answer to one question: did this help, and did it break anything that already worked? This skill gives you a tiny stdlib-only Python harness that scores prompt variants against a JSONL dataset, logs scores per run, and compares old vs new with a hard regression gate. No dependencies, no SaaS, no guessing.

Workflow

1. Write the contract first: 10 to 30 cases. Each line of cases.jsonl is one case: an id, an input, the expect value, a scorer, and optionally "regression": true. Cover three groups: the happy path, edge cases, and pinned past failures. Pin every real incident as a regression case the day it happens. Example:

json
{"id": "billing-clear", "input": "I was charged twice for my subscription", "expect": "billing", "scorer": "exact"}
{"id": "tech-clear", "input": "the app crashes when I open settings", "expect": "tech", "scorer": "exact"}
{"id": "refund-edge", "input": "I want a refund for the extra fee", "expect": "billing", "scorer": "exact_strict", "regression": true}

2. Pick the weakest scorer that still catches the failure. Deterministic first: exact (case/space-insensitive), exact_strict (raw match, catches formatting bugs), contains, starts_with, regex. Reach for the judge scorer only for things no deterministic check can see (tone, structure, coverage of a rubric). The judge is a callable you supply in runs.py as JUDGE(output, rubric) -> (passed, detail). Judges are flaky: if you use a model as judge, run it at temperature 0 with a fixed seed, and accept that judge drift is a real failure mode you have to re-check.

3. Save the harness. Save the code below as eval.py, then run it. It has three commands:

• python3 eval.py run --cases cases.jsonl --runs runs.py --run v1 --out v1.jsonl --log scores.log scores one variant. Per-case results go to --out; one summary line is appended to --log so trends are visible across runs.
• python3 eval.py compare --cases cases.jsonl --runs runs.py --compare v1 v2 scores both variants and prints a per-case table with FIXED/BROKE deltas. It exits nonzero and names the broken cases if any pinned regression case fails on the new variant.

Your runs.py defines RUNS = {name: callable} where each callable takes the case input and returns the model's output string. In production each callable calls your real prompt or model; in the demo below they are deterministic stand-ins so the harness runs offline.

python
#!/usr/bin/env python3
"""Tiny eval harness. stdlib only. Save as eval.py, then run it.

Usage:
  python3 eval.py run --cases cases.jsonl --runs runs.py --run v1 --out scores.jsonl
  python3 eval.py compare --cases cases.jsonl --runs runs.py --compare v1 v2
"""
import argparse, importlib.util, json, re, sys

def scorer_exact(output, expect):
    return norm(output) == norm(expect)

def scorer_contains(output, expect):
    return expect in output

def scorer_regex(output, expect):
    return re.search(expect, output, re.DOTALL) is not None

def scorer_starts_with(output, expect):
    return output.strip().startswith(expect)

def scorer_exact_strict(output, expect):
    # no normalization: catches formatting bugs (capitalization, stray spaces)
    return output.strip() == expect.strip()

def norm(s):
    return " ".join(str(s).strip().lower().split())

SCORERS = {
    "exact": scorer_exact,
    "exact_strict": scorer_exact_strict,
    "contains": scorer_contains,
    "regex": scorer_regex,
    "starts_with": scorer_starts_with,
}

def load_cases(path):
    cases = []
    with open(path) as f:
        for i, line in enumerate(f, 1):
            line = line.strip()
            if not line:
                continue
            try:
                case = json.loads(line)
            except json.JSONDecodeError as e:
                sys.exit(f"cases line {i}: bad JSON: {e}")
            for key in ("id", "input", "expect"):
                if key not in case:
                    sys.exit(f"cases line {i}: missing key '{key}'")
            case.setdefault("scorer", "exact")
            cases.append(case)
    return cases

def load_module(path):
    spec = importlib.util.spec_from_file_location("runs", path)
    mod = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(mod)
    return mod

def load_runs(path):
    mod = load_module(path)
    if not hasattr(mod, "RUNS") or not isinstance(mod.RUNS, dict):
        sys.exit("runs.py must define RUNS = {name: callable(input) -> str}")
    return mod.RUNS

def run(args):
    cases = load_cases(args.cases)
    runs = load_runs(args.runs)
    if args.run not in runs:
        sys.exit(f"unknown run '{args.run}'. available: {sorted(runs)}")
    fn = runs[args.run]
    mod = load_module(args.runs)
    results = []
    for case in cases:
        name = case["scorer"]
        try:
            output = fn(case["input"])
        except Exception as e:
            output = f"<RUN ERROR: {type(e).__name__}: {e}>"
        if name == "judge":
            judge_fn = getattr(mod, "JUDGE", None)
            if judge_fn is None:
                sys.exit("scorer 'judge' used but runs.py defines no JUDGE callable")
            passed, detail = judge_fn(output, case.get("rubric", ""))
        else:
            if name not in SCORERS:
                sys.exit(f"case {case['id']}: unknown scorer '{name}'")
            passed = bool(SCORERS[name](output, case["expect"]))
            detail = f"expect={case['expect']!r}"
        results.append({
            "run": args.run, "id": case["id"], "scorer": name,
            "passed": passed, "detail": detail, "output": output,
        })
    passed = sum(r["passed"] for r in results)
    summary = {
        "run": args.run, "total": len(results), "passed": passed,
        "failed": len(results) - passed,
        "pass_rate": round(passed / len(results), 3) if results else 0.0,
        "failed_ids": [r["id"] for r in results if not r["passed"]],
    }
    with open(args.out, "w") as f:
        for r in results:
            f.write(json.dumps(r) + "\n")
    with open(args.log, "a") as f:
        f.write(json.dumps(summary) + "\n")
    print(f"run={args.run} passed={passed}/{len(results)} "
          f"rate={summary['pass_rate']} failed={summary['failed_ids']}")
    print(f"per-case: {args.out}  summary log: {args.log}")

def compare(args):
    runs = load_runs(args.runs)
    for name in args.compare:
        if name not in runs:
            sys.exit(f"unknown run '{name}'. available: {sorted(runs)}")
    cases = load_cases(args.cases)
    table = {}
    for name in args.compare:
        fn = runs[name]
        row = {}
        for case in cases:
            scorer = SCORERS.get(case["scorer"], None)
            try:
                output = fn(case["input"])
            except Exception as e:
                output = f"<RUN ERROR: {type(e).__name__}: {e}>"
            if case["scorer"] == "judge":
                sys.exit("compare mode does not support judge scorers yet; use run mode")
            row[case["id"]] = bool(scorer(output, case["expect"]))
        table[name] = row
    old, new = args.compare[0], args.compare[1]
    reg = [c["id"] for c in cases if c.get("regression")]
    print(f"{'case':<22}{'regr':<6}{old:<8}{new:<8}delta")
    wins = losses = 0
    for c in cases:
        cid = c["id"]
        a, b = table[old][cid], table[new][cid]
        delta = "" if a == b else ("FIXED" if b else "BROKE")
        if a != b and b:
            wins += 1
        if a != b and not b:
            losses += 1
        print(f"{cid:<22}{str(bool(c.get('regression'))):<6}{str(a):<8}{str(b):<8}{delta}")
    reg_broken = [cid for cid in reg if not table[new][cid]]
    print(f"\n{old} -> {new}: fixed={wins} broke={losses}")
    if reg_broken:
        print(f"REGRESSION FAIL on pinned set: {reg_broken} -- do not ship")
        sys.exit(2)
    print("regression set green.")

def main():
    ap = argparse.ArgumentParser()
    sub = ap.add_subparsers(dest="cmd")
    r = sub.add_parser("run")
    r.add_argument("--cases", required=True)
    r.add_argument("--runs", required=True)
    r.add_argument("--run", required=True)
    r.add_argument("--out", default="scores.jsonl")
    r.add_argument("--log", default="scores.log")
    c = sub.add_parser("compare")
    c.add_argument("--cases", required=True)
    c.add_argument("--runs", required=True)
    c.add_argument("--compare", nargs=2, required=True, metavar=("OLD", "NEW"))
    args = ap.parse_args()
    if args.cmd == "run":
        run(args)
    elif args.cmd == "compare":
        compare(args)
    else:
        ap.print_help()
        sys.exit(1)

if __name__ == "__main__":
    main()

4. Baseline, change, compare. Run the current prompt first and keep its score. Make your prompt change, add it as a new entry in RUNS, and run compare. Real output from a 5-case demo (two prompt variants of a toy triage classifier, verified by running the harness above):

run=v1 passed=3/5 rate=0.6 failed=['refund-edge', 'label-format']
run=v2 passed=5/5 rate=1.0 failed=[]
case                  regr  v1      v2      delta
billing-clear         False True    True
tech-clear            False True    True
other-clear           False True    True
refund-edge           True  False   True    FIXED
label-format          True  False   True    FIXED

v1 -> v2: fixed=2 broke=0
regression set green.

A failing per-case row in the output JSONL looks like this (also from the real run), and tells you exactly what to look at:

json
{"run": "v1", "id": "refund-edge", "scorer": "exact_strict",
 "passed": false, "detail": "expect='billing'", "output": " Billing"}

5. Read the failures before you change the prompt again. Cluster them: many failures in one capability means the prompt is missing a concept, fix the prompt; scattered single misses mean the dataset is thin, add cases. Add a case whenever a user or a test finds a real miss, and pin it as regression. A green suite means the change did not regress what you measured. It does not mean the prompt is good. Evals are a floor, not a ceiling.

Output Contract

• cases.jsonl: one JSON object per line with id, input, expect, scorer; regression cases carry "regression": true.
• runs.py: RUNS = {name: callable}; each callable takes the case input and returns an output string. Optional JUDGE(output, rubric) -> (passed, detail) for judge-scored cases (expected shape; a real one calls a judge model API at temperature 0 with a fixed seed).
• <run>.jsonl: one result object per line with run, id, scorer, passed, detail, output.
• scores.log: one summary line per run (run, total, passed, failed, pass_rate, failed_ids) for tracking trends.
• Compare output: per-case old-vs-new table with FIXED/BROKE deltas; exit code 2 and a REGRESSION FAIL line when the new variant breaks a pinned case.

Operating Rules

1. Run the suite before and after every prompt change. No change ships on vibes.
2. A prompt ships only with a green pinned regression set. New variant must beat or tie the old one there.
3. Pin real incidents as regression cases the day they happen. The dataset grows from production, not from imagination.
4. Deterministic scorers first, judge only for what they cannot see. If you use a judge model, temperature 0, fixed seed, and re-check it periodically for drift.
5. If you care about contamination, keep the eval dataset out of any training or fine-tuning context. Test data that leaks into training stops measuring anything.
6. Evals are samples, not proofs. A green run means no measured regressions, not a guarantee of quality.