agent_decision_audit v1.0.0

name: agent_decision_audit description: Auditing the decisions an agent made during a run, not just its final output: keep a choices ledger as the work happens, rank choices by confidence and blast radius, review the least certain ones first, and land each correction in the spec, instructions, tests, or a tool guard. Use when reviewing a finished agent run, when a run went wrong and the diff looks innocent, when setting up decision logging for a new agent, or when the same wrong choice keeps recurring across runs. Works from any run log, with a ledger format and a stdlib extraction and ranking script.

Agent Decision Audit

Purpose

A finished diff shows what changed. It does not show the forks: the options the agent passed over, the assumption it never checked, the spec line it never had. Most bad runs are not bad typing. They are one or two bad choices with good typing after them.

This skill audits the choices. Keep a ledger of decisions while the run happens, score each one for confidence and blast radius, review the riskiest few in depth, and land every correction somewhere durable. The method borrows from decision records in software architecture: an Architectural Decision Record captures a single decision and its rationale, and the collection of records is the decision log (adr.github.io). It applies the same habit inside one agent run, at run speed.

Workflow

1. Capture decisions as the work happens, not after. A decision is a fork where a different plausible option existed: which file to change, which source to trust, which interpretation of the spec to follow, whether to act or ask, whether to delete or deprecate. Routine steps with no real alternative are not decisions, do not log them. For each decision, append one JSON line to the run ledger with the fields in the Output Contract. Writing it at the moment of choice matters: confidence written later is hindsight, not confidence.

1. If the run already happened, extract candidates and mark them. Run scripts/decision_audit.py extract RUNLOG.jsonl on any JSONL run log. It recognises tool execution lines (including OpenTelemetry style gen_ai.operation.name values such as execute_tool, and gen_ai.tool.name) and explicit choice statements, and writes draft records with guessed blast radius and null confidence. Mark these provenance: extracted. Fill in options, evidence, and basis from the log where the log supports it. Where it does not, leave the field empty and let the verdict be unverifiable. Never invent a rationale the log does not contain.

1. Validate the ledger before reviewing it. Run scripts/decision_audit.py validate LEDGER.jsonl. Fix duplicate ids, missing choices, and out of range scores first. An audit built on a broken ledger produces confident nonsense. Then check completeness the other way: list every side effect in the run (writes, sends, deletions, spends, state changes) and confirm each one traces to a ledger record or to an explicit standing default. A side effect with no decision and no default is a finding on its own.

1. Rank, then review in rank order. Run scripts/decision_audit.py rank LEDGER.jsonl. Priority is this skill's heuristic: (1 - confidence) blast_radius 2, plus 1.0 for irreversible, 0.5 for costly to reverse, 0.5 for a missing spec basis, and 0.5 for empty evidence. Unscored records sort first. Review in depth from the top until the remaining priorities are low and the records are well evidenced, then quick read the rest. Reviewing in run order wastes the hour on the easy early choices.

1. Judge each reviewed decision against what was known at the time. Use one verdict per decision:

• sound: reasonable choice, evidence and basis held up.
• sound-but-lucky: choice worked, but a premise was unchecked and the outcome depended on coverage or chance the agent did not arrange.
• flawed-premise: the choice rested on a fact or assumption that was wrong or never verified.
• flawed-execution: the choice was right, the carrying out was wrong.
• unverifiable: the record does not contain enough to judge. Treat a cluster of these as a logging failure, not as a pass. For flawed verdicts, name the root cause in plain words: missing information, misread spec, ignored evidence, overconfidence, or tool error. Do not judge a decision by its outcome alone. A bad choice that got lucky is still a bad choice, and a good choice with a bad outcome is not a correction target.

1. Score honesty before scoring correctness. Check the confidence numbers against the evidence in the same record. Anchors for this skill: 0.9 means the spec or a primary source states it directly; 0.7 means strong evidence with one unchecked link; 0.5 means a coin flip between real options; below 0.5 means the agent knew it was guessing. A record claiming 0.9 with an empty evidence list is a calibration finding, and calibration findings matter because the ranking trusts those numbers.

1. Land every correction somewhere durable. Each flawed or lucky decision produces a correction with a named home, exactly one of:

• a spec edit, when the instruction was missing or ambiguous;
• an instruction rule, when the agent should behave differently next time;
• a test or eval case, when a check would have caught it;
• a tool guard or new default, when the tool allowed a risky act too easily;
• an accepted risk note, when the reviewer decides the choice was fine to repeat. A correction with no home is a comment, not a correction. Record where each one landed in the audit summary.

1. Close with patterns, not just incidents. Count verdicts, count empty spec basis fields, count empty evidence fields. The same root cause in three runs is a system problem: thin spec, missing tool check, or a standing instruction that invites the error. Report the pattern and fix that level, not only the single run. For ongoing programs this maps onto the NIST AI Risk Management Framework habit of continuous review: its core functions are Govern, Map, Measure, and Manage, and Govern calls for ongoing monitoring and periodic review with roles and responsibilities defined (NIST AI RMF 1.0, 2023). For high risk work, check the record keeping duties that apply to the deployment, for example under Regulation (EU) 2024/1689, before deciding how long ledgers are kept.

Output Contract

• Ledger: one JSON object per line, fields id (sequential, never reused in the run), chosen, options, rejected_because, evidence, spec_basis, confidence (0 to 1, or null if unscored), blast_radius (integer 1 to 5), reversibility (reversible, costly, or irreversible), plus provenance (live or extracted) when known. Blast radius scale: 1 is a local reversible note or read; 2 is one file or one record, easy to undo; 3 is several files, a data write, or behaviour other parts inherit; 4 is an external effect such as a send, publish, spend, or merge, or a change that is hard to reverse; 5 is irreversible, or touches security, money movement, legal exposure, or other people's data.
• Ranked review queue: the table printed by the rank command, kept with the audit.
• Audit summary: decisions recorded, reviewed in depth, quick read; verdict counts by label; each correction with where it landed; pattern notes across runs when this is not the first audit.
• Worked example: references/worked-audit.md audits the five record ledger in examples/ledger.jsonl end to end, verdicts and landing places included.

Operating Rules

• Audit choices, not effort. Ten careful records beat a transcript of every step.
• Never backfill confidence or rationale as if it were written live. Extracted records stay marked extracted, and gaps stay visible.
• Never delete or rewrite a ledger record at review time. Add the verdict and outcome alongside it, the same way a superseded decision record stays in a decision log with its status changed. The wrong record is the evidence.
• Review the reviewer too: if the ranking script and the reviewer disagree about what matters most, say why in the summary instead of silently reordering.
• Keep secrets and personal data out of the ledger. Cite the evidence by reference (file path, tool result id, URL), do not copy credential values or private content into a record that will be kept and shared.
• Stop and escalate instead of auditing around a live problem. If the audit finds an irreversible act still causing harm, report that first and audit later.
• Quality over coverage: a shallow pass over every record is worth less than a real pass over the top of the queue. Say in the summary where the review stopped.