multi_agent_orchestration v1.0.0

name: "multi_agent_orchestration" description: "Run a lead agent with parallel worker subagents: decide when multi-agent beats one agent, decompose work into independent units, dispatch in one batch, reconcile results, control token cost, and isolate failures. Triggers when a task has breadth-first parallel work, independent research threads, or output that exceeds one context window."

Multi-Agent Orchestration

Purpose

Run agent swarms that finish with a verified result instead of a pile of conflicting drafts. The orchestrator owns decomposition, the worker contract, cost, the merge, and failure handling. Workers own execution only and never see the full conversation.

This is the playbook for orchestrator-worker runs used daily in production: task fan-out, result reconciliation, cost control, failure isolation.

Workflow

1. Go/no-go test. Multi-agent wins on breadth-first, read-heavy work where independent units can each own a clean context window. Anthropic's multi-agent research system (lead Opus 4 + parallel Sonnet 4 subagents) beat a single Opus 4 agent by 90.2% on their internal research eval, and parallel tool calls cut wall time up to 90% for complex queries. It fails on tightly coupled work: Cognition's finding is that parallel workers make incompatible hidden decisions (their example: two subagents building one game produced a bird and a background with clashing styles). Split by context needs, not by job title. If units share mutable state, must run in sequence, or the coordination overhead exceeds the work, stay with one agent.

1. Pick the pattern (full catalog in references/patterns.md): orchestrator-worker for breadth-first exploration, fixed pipeline for known sequences, fan-out/fan-in with one synthesizer for multiple perspectives on the same input, queue swarm for high-throughput identical units.

1. Decompose. Each unit must be independent (no shared mutable state, no unit needs another's output mid-flight), self-contained (the brief carries all context: paths, constraints, definition of done; workers do not see your conversation), and verifiable (a clear, checkable deliverable). Cap concurrency at the real independence width. More agents than independent units just adds merge cost.

1. Write the worker brief. Use assets/worker-brief-template.md. Every brief carries: objective, exact scope (in and out), inputs and paths, output format (structured, not prose), done criteria, and evidence requirements (file paths, tool output, URLs). Structured output matters: one source found that orchestrators choke when workers return full prose blobs. The orchestrator resolves conflicts, writes no feature code.

1. Dispatch in one batch. Launch all units together so they run concurrently. Never spawn one, await it, then spawn the next. Match the agent to the job: lead on your strongest model, workers on a fast cheap tier (the canonical tiering is lead on Opus/Sonnet-class, workers on Haiku-class for routine execution).

1. Budget the run. See references/cost-model.md. Multi-agent runs use roughly 15x the tokens of a single chat interaction, and re-sending context is 86% to 100% of per-agent spend. Set a per-run token or dollar cap, a per-unit timeout, and a max retry count before dispatching. Track spend per unit; a unit that burns 3x its estimate gets killed, not watched.

1. Reconcile. Choose one strategy per merge and apply it uniformly:

• Schema merge when units produced non-overlapping fields: combine, check for gaps and contradictions.
• Voting/majority when units classified or judged the same input: report the tally, do not average opinions silently.
• LLM synthesis when results need narrative reconciliation: the synthesizer works only from worker outputs plus the briefs, never from memory of the conversation. Record what each unit claimed, what the merge kept, and what it dropped. Never propagate a worker's output into the final answer without the reconciliation step touching it.

1. Isolate failures. See references/failure-modes.md. Per-unit timeouts with kill, not extension. Retry a failed unit once with a narrowed scope; if it fails again, quarantine it and proceed with partial results rather than re-running the whole batch. A worker that returns malformed output gets one re-ask with the exact schema violation quoted; malformed twice means quarantined. Use bin/work_claim.py for queue-based swarms: atomic claims, heartbeat files, and automatic requeue of units whose heartbeat goes stale, so a dead worker never blocks the queue.

1. Verify the merged output. Check it against the original objective and each unit's done criteria. Flag which parts are verified by evidence, which are reconciled judgment calls, and what was dropped. Ship partial with a gap list before shipping complete-looking and wrong.

Output Contract

Every orchestrated run ends with a structured result:

• Result: the merged deliverable in the format the original task required.
• Unit ledger: one line per unit: unit id, status (done/retried/quarantined), evidence pointer (file path, tool output, URL), token cost.
• Reconciliation notes: what conflicts existed, which strategy resolved them, what was dropped and why.
• Cost log: total tokens/dollars vs the budget set at dispatch.
• Gaps: anything unclaimed, timed out, or quarantined, listed plainly.

Operating Rules

1. Parallelize reads, serialize writes. Independent investigation in parallel; one writer produces the final artifact. Conflicting reads waste tokens; conflicting writes produce incompatible work.
2. Workers get the full brief in the dispatch call. They never see the conversation, and the conversation never sees their raw tool traffic.
3. Structured output only from workers: JSON or a fixed schema, with evidence attached. No prose blobs.
4. The orchestrator reconciles everything. A worker's claim never reaches the final answer untouched.
5. Set budget, timeout, and retry limits before dispatching. Kill over-budget units; do not watch them.
6. One retry per failed unit with narrowed scope, then quarantine and partial results. Never re-run the whole batch for one unit.
7. Report partial with a gap list instead of complete-looking and wrong.
8. Quote verified numbers only. Anthropic's figures above are measured on their research eval, not a universal guarantee; frame them as what they are.