agent_architecture_audit v1.0.0

name: "agent_architecture_audit" description: "Audit an AI agent system's architecture with a scored rubric: autonomy boundaries, tool scoping, failure propagation paths, memory hygiene, and eval coverage. Triggers when asked to review, audit, harden, or assess an agent, agent platform, or agentic deployment before it ships or when something already broke."

Agent Architecture Audit

Purpose

Give any agent system a structured, evidence-backed health check before it earns more autonomy. The audit covers five dimensions that decide whether a deployment is sound or a headline waiting to happen: autonomy boundaries, tool scoping, failure propagation, memory hygiene, and eval coverage. The output is a scored rubric (0-5 per dimension, 30 points total) plus a concrete fix list ordered by blast radius.

Workflow

1. Inventory the system. Map: the agent(s), the model(s) underneath, every tool it can call (with read/write/delete flags), every data store it reads and writes (including memory), every human touchpoint, and every external party that receives its output. No audit starts without this map.

1. Score autonomy boundaries. Verify per-ACTION autonomy tiers, not a blanket "autonomous" label. An agent that drafts autonomously but sends only with approval is three different tiers. Check: are irreversible actions (sending, spending, deleting, publishing, messaging) gated? Are gates enforced in code, or only suggested in prompts? Prompt-only gating scores 0.

1. Score tool scoping. For each tool: least-privilege scopes, allowlist vs blocklist, argument schema validation before execution, and whether execution passes through a mediated gateway (a PDP or adapter that can deny) instead of a direct path. Check for credential handling: short-lived scoped credentials issued outside agent memory, never long-lived tokens in context.

1. Score failure propagation. Trace every failure path: what happens when a tool fails, the policy engine is unreachable, a credential expires, or memory is unreachable? Look for: fail-closed defaults, bounded retries with backoff (no infinite retry loops), circuit breakers between agents, kill switches, rollback plans, and blast-radius limits (one tenant, one region, one percent before going wide). Failure handling must be in the platform layer, not in the agent's reasoning.

1. Score memory hygiene. Memory is a write surface, so treat it like one. Check: scoping by task/session/tenant, provenance recorded per entry, TTLs and decay, redaction of secrets, quarantine of externally-sourced entries, and a rule that nothing from tool output or retrieved documents enters long-term memory without validation. Cross-tenant memory reuse without explicit policy is an automatic 0.

1. Score eval coverage. Ask what gets tested and what never does. Require: regression evals that run on every prompt/model change, policy-enforcement tests (attempts that MUST be denied), fail-closed tests (what happens when the guardrail service is down), red-team probes for goal hijack and injection through tool outputs, and shadow mode before any new write privilege. Also check that escalation rate, rollback frequency, and cost per task are tracked, not just task success.

1. Write the report. Score each dimension 0-5 using the rubric in references/audit_rubric.md. Every score must cite evidence: a config snippet, a log line, a test result, an observed behavior. A claim with no evidence is an assumption and must be labeled as one. Total the scores and grade: 25-30 solid, 18-24 needs work before expansion, 10-17 contains real risk, 0-9 do not deploy. Order fixes by blast radius (what kills the most first), each fix with the evidence that made it matter.

Output Contract

• A five-dimension scorecard: each dimension with 0-5 score, the evidence behind it, and the single highest-leverage fix.
• A total score with the grade band.
• A fix list ordered by blast radius, each item: what to change, why it matters (with evidence), and what "done" looks like in one verifiable sentence.
• No hand-waving: every factual claim cites a concrete artifact (config, log, test output, observed behavior) or is explicitly labeled an assumption.

Operating Rules

1. Verify every claim against a live artifact. Never infer architecture from marketing copy, README claims, or the agent's own prose about itself. If you cannot see the config, say so and score on the worst reasonable reading.
2. "Reasoning is not authority": treat what the model says it will do as intent, never as permission. Score what the platform ENFORCES.
3. Gates in prompts are not gates. Any safety property that depends on the model following instructions scores at most 1 in its dimension.
4. Every factual claim follows the accuracy rule: verified via the system itself or a primary source; unverifiable gets cut or labeled an assumption.
5. When the system under audit cannot be inspected (closed vendor, no access), deliver a black-box audit: probe it like an adversary within the allowed scope, and label every score with how it was obtained.
6. Never use em dashes in the report. Write plainly: short sentences, direct claims, no stacked adjectives, no motivational filler.