agent_arena_tournaments v1.0.0

name: "agent_arena_tournaments" description: "Run a single-elimination tournament across N competing subagents to pick the best answer for a hard question. Drafts distinct strategy cards, brackets the contestants, and judges pairwise matches with debiased judging. Use when one agent's first answer is not trustworthy enough and you need the strongest answer from several independent attempts, e.g. architecture decisions, tricky designs, high-stakes plans."

Agent Arena Tournaments

Purpose

For hard questions, a single agent locks onto the first reasonable answer and defends it. Research on multi-agent debate (Du et al., 2023) shows competing agents with diverse approaches converge on more accurate answers than a lone agent reasoning in isolation. This skill operationalizes that: spawn N contestants with different strategy cards, pair them in a single-elimination bracket, judge each match blind and debiased, and hand back the champion plus the evidence trail.

Workflow

1. Decide the field. Pick N = 4 or 8 (powers of two keep the bracket clean; 8 is the default for high-stakes work). Cost is roughly N answers plus N-1 judged matches. For anything where one answer costs almost nothing, skip the tournament and just ask twice.
2. Write strategy cards. Each contestant gets one card: a distinct approach, a distinct set of constraints to respect, and a distinct failure mode to avoid. See references/strategy-cards.md. Cards must force genuine divergence. Eight cards that all say "be thorough" are eight copies of the same contestant and the tournament is theater.
3. Spawn contestants. Give every contestant the identical task brief plus only its own card. Run them in parallel. Tell each one its answer will face blind judging, so it should optimize for correctness, not persuasion.
4. Seed and run the bracket. Use bin/bracket.py to generate pairings. Seed cards so the two strongest-looking approaches meet late, not in round one. One match = one judge verdict; the loser is eliminated, no appeals. An 8-player bracket needs 7 matches across 3 rounds.
5. Judge every match blind. Follow the judge protocol in references/judge-rubric.md: anonymize the entries, run both orderings (A,B) and (B,A), require order-consistency, score against a written rubric, and prefer a judge from a different model family than the contestants. Log every verdict, including disagreements between orderings.
6. Optional: second chance. If the stakes justify it, give each contestant one extra life (double-elimination lite: a loss moves the entry to a loser's bracket, a second loss ends it). This protects against a single bad judge call killing the best answer. It roughly doubles the matches, so make it a deliberate choice.
7. Crown and report. Return the champion, the runner-up, and the match log. If two finalists each nailed half the problem, synthesize a merged answer and say so, citing which parts came from whom.

Output Contract

The tournament report contains:

• The field: contestant IDs, their strategy cards, the task brief they shared.
• The bracket: round-by-round pairings and seeding rationale.
• Per match: the two anonymized entries, the judge verdict for each ordering, the winning rationale, and any order disagreements.
• The champion: its full answer, plus a one-paragraph account of why it beat the runner-up.
• The judge's rubric scores and the method used (model family, orderings, anonymization).

Operating Rules

1. Diversity is the whole point. If the cards do not produce visibly different answers, the field is not diverse enough. Rewrite the cards before running more matches.
2. Blind everything. Contestants never know who they face; the judge never knows which model wrote which entry or which round it is judging.
3. Both orderings or it does not count. A verdict that flips when the entries are swapped is a biased verdict, not a winner. Re-judge or call it a tie and break it with the tiebreak rule below.
4. Ties break on the rubric, not on gut. If both orderings disagree and no winner emerges, score both entries against the rubric criterion by criterion; the one that loses fewer criteria wins. If still tied, the entry with the stronger reasoning trace (per the judge's notes) advances.
5. Never let a contestant judge its own match. Use a judge from a different model family when possible. A judge grading its own family's work inflates that work (self-enhancement bias is measured, not hypothetical).
6. Judge math and code symbolically first. An LLM judge is weak at exact quantitative checking. Run any verifiable claims (tests, calculations, queries) through an actual checker before the judge weighs them.
7. Cap the field. More than 8 contestants and judge noise starts to swamp the signal; the bracket math still works, but the quality of each match's judging degrades as attention spreads. Split the question instead.
8. Report the trail, not just the trophy. The user gets the match log, not a bare "winner". A tournament that hides its judging is a black box with extra steps.