--- name: agent_retrospectives description: "Run a backward-looking review of a finished period of agent work: compare what was claimed against what shipped, classify each failure, spot repeat findings, and turn the results into a small number of concrete rule or workflow changes. Triggers at the end of a work period, after a batch or project closes, or when the same failure keeps recurring." metadata: { "includeInPrompt": true } --- # Agent Retrospectives ## Purpose This skill reviews a finished period of agent work and answers three questions from evidence: what was claimed, what actually shipped, and what will change as a result. It produces one retro document, one claim-vs-shipped table, and a short list of rule or workflow changes that get checked in the next period. Scope lines against close cousins: - weekly-review is a GTD-lite forward planning review of backlogs and commitments; this skill looks backward at a period that has already ended. - receipts-standard is logging verifiable claim receipts; this skill uses those receipts as evidence and judges the claims they support. - agent-observability is run instrumentation; this skill reads the logs and traces that instrumentation produced and turns them into findings. This skill owns the backward-looking review of a finished period of agent work: what was claimed vs what shipped, failure classification, and turning findings into a small number of concrete rule or workflow changes. ## Workflow ### 1. Define the period before pulling evidence Write down, at the top of the retro document: - Start date and end date for the period, with timezone. - Scope: which projects, batches, or work streams are in, and which are out. - The previous retro document for this work stream, if one exists. Step 6 needs it. - The claim sources for the period: status reports, completion messages, task lists, or delivery notes written during the period. Freeze the period once defined. Work that finishes after the end date belongs to the next period, even if it fixes a gap found in this one. ### 2. Pull the evidence Collect three evidence sets before judging any claim: - Run logs: run records, tool outputs, timestamps, error messages, and retry notes for the period. - Claim list: every completion or delivery claim made for the period, copied with its date and source. One row per claim. Do not merge two claims into one row and do not split one claim into two. - Shipped artifacts: the files, posts, records, commits, or other outputs each claim points to. Record a path, link, or identifier for each artifact so a reader can open it. If a claim has no artifact and no log entry, mark its evidence as missing in step 3. Do not reconstruct evidence from memory. Build the claim list as a JSON file in the helper format from step 4: a list of objects with `id`, `claim`, `status`, and `failure_class`. Fill `status` and `failure_class` in during step 3. ### 3. Build the claim-vs-shipped table For each claim, check the artifact and the log, then assign one status: - `shipped`: the artifact exists, matches the claim, and the log supports the claimed timing. - `partial`: part of the claim shipped, or the artifact exists but falls short of the claim in a stated way. - `missing`: no artifact, or the artifact does not match the claim. - `blocked`: the work did not ship and the log shows a specific external cause that stopped it. Record the table in the retro document with these columns: claim id, claim text, status, evidence (path, link, or log reference), failure class, and a one-line note. Rules for this step: - Evidence decides the status. A confident report with no artifact is `missing` or `partial`, never `shipped`. - Write the note as a fact: what was found, where, and what was absent. Keep opinion out of the table. - A claim that was withdrawn or corrected during the period still gets a row. Note the correction in the note column. ### 4. Tally the table with the helper Save the script at the end of this skill as `retro_tally.py`, then run: python3 retro_tally.py claims.json The helper prints JSON counts by status and by failure class. Copy those counts into the retro document without editing them. If a count looks wrong, fix the claim list and rerun the helper. Do not adjust the printed numbers by hand. Worked example, run during drafting of this skill. Fixture `claims.json` held 8 claims: `c1` to `c3` shipped, `c4` partial with failure class planning, `c5` missing with failure class verification, `c6` missing with failure class tool, `c7` blocked with failure class external_block, and `c8` partial with failure class reporting. Running the helper printed: { "total": 8, "by_status": { "blocked": 1, "missing": 2, "partial": 2, "shipped": 3 }, "by_failure_class": { "external_block": 1, "none": 3, "planning": 1, "reporting": 1, "tool": 1, "verification": 1 } } An invalid status such as `done` makes the helper exit 1 with an error naming the claim id, so a bad row fails loudly instead of being silently dropped. ### 5. Classify each failure Every claim with status `partial`, `missing`, or `blocked` gets exactly one primary failure class: - `planning`: the work was scoped wrong, sequenced wrong, missing a step, or based on a wrong assumption that was visible at planning time. - `tool`: a tool, script, command, or API failed, returned wrong output, or lacked a capability the plan required. - `verification`: the work shipped or was reported without a check that would have caught the gap, or a check ran and its result was ignored or misread. - `reporting`: the work state was fine or partly fine, but the report about it was wrong, overstated, incomplete, or sent to the wrong place. - `external_block`: a cause outside the work itself stopped it, for example an outage, a rate limit, a revoked permission, or a missing approval. Name the cause in the note column. Shipped claims use failure class `none`. Use `unclassified` only when the evidence is too thin to choose a class, and state what evidence is missing. When two classes fit, pick the earlier cause in the chain: a planning gap that led to a tool failure is `planning`. ### 6. Detect repeat findings Compare this period against the previous retro document for the same work stream: - List each finding from the previous retro by its failure class and a short pattern label, for example `verification: totals reported without rerun`. - Mark a current finding as a repeat when it shares the failure class and the pattern label, or when it concerns the same claim area and the same root cause in plain words. - Record the repeat count for each pattern: first seen this period, or seen in 2, 3, or more consecutive periods. - A pattern seen in 2 or more consecutive periods gets named at the top of the findings section. It is a priority candidate for a rule change in step 7, ahead of any first-time finding. - If there is no previous retro, write `no prior retro; repeat detection starts next period` and move on. ### 7. Turn findings into a small number of changes Convert findings into rule or workflow changes. Cap the list at 3 changes per retro as guidance. The cap keeps each change checkable in the next period. If a fourth change looks urgent, write it down and record why it was added or why it waits. Each change must state: - The change itself, written as an instruction that can be followed or ignored, with no middle state. - Where it applies: the workflow step, checklist, template, or document it modifies, named exactly. - The finding it answers, by pattern label and failure class. - How the next retro will check it: what evidence will show the change was followed, and what result counts as it working. Prefer changes to checklists, templates, and verification steps over general advice. A change that reads as `be more careful` fails this step. Rewrite it until it names an action, a place, and a check. Do not create a change for an `external_block` finding unless the change reduces the damage from the next block, for example an earlier escalation step or a fallback path. The block itself is outside the workflow. ### 8. Write the retro document Use this structure, in this order: 1. Header: period start and end dates, scope, claim sources, previous retro reference. 2. Counts: the helper output from step 4, copied exactly. 3. Claim-vs-shipped table: the full table from step 3. 4. Findings: grouped by failure class, repeat patterns first, each with its evidence and repeat count. 5. Changes: the list from step 7, capped at 3 as guidance, each with its check method. 6. Open items: claims still blocked or partial at period end, with the named cause and the next action for each. 7. Follow-up checks: copied from the previous retro, with the result recorded for each in step 9. Keep the document short enough to read in one sitting. Put raw logs and full claim files with the document as attachments or linked paths, not pasted into the findings. ### 9. Run follow-up checks at the next period At the start of the next retro for the same work stream, before step 3: - Pull the change list from the previous retro. - For each change, check the evidence named in its check method. Record one result per change: followed and working, followed and not working, or not followed. - A change that was not followed becomes a finding in the new period, class `reporting` if the change was recorded but never applied, or class `planning` if the change was too vague to apply. - A change followed for one full period with the target failure absent can be marked settled and dropped from the active list. Keep the settled note in the document. - Carry unsettled changes forward into the new retro document follow-up section. Do not drop them silently. ### 10. Helper script Save the following as `retro_tally.py`. It uses only the Python standard library. ```python #!/usr/bin/env python3 """Tally claim outcomes for an agent retrospective. Input: a JSON file containing a list of claim objects: id str, required claim str, required status str, required: shipped, partial, missing, or blocked failure_class str, optional: planning, tool, verification, reporting, external_block, or none. Defaults to none for shipped claims and unclassified for other claims. Usage: python3 retro_tally.py claims.json Output: JSON counts by status and by failure class, plus the total. Exit code: 0 on success, 1 on invalid input. """ import json import sys from collections import Counter from pathlib import Path STATUSES = {"shipped", "partial", "missing", "blocked"} FAILURE_CLASSES = {"planning", "tool", "verification", "reporting", "external_block", "none", "unclassified"} def main(): if len(sys.argv) != 2: print("usage: python3 retro_tally.py claims.json", file=sys.stderr) return 1 try: data = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8")) except Exception as exc: print(f"error: cannot read JSON: {exc}", file=sys.stderr) return 1 if not isinstance(data, list): print("error: top-level JSON must be a list", file=sys.stderr) return 1 by_status = Counter() by_failure = Counter() for item in data: if not isinstance(item, dict) or not item.get("id") or not item.get("claim"): print("error: each claim needs id and claim", file=sys.stderr) return 1 status = str(item.get("status", "")).strip().lower() if status not in STATUSES: print(f"error: claim {item['id']} has invalid status {status!r}", file=sys.stderr) return 1 failure = str(item.get("failure_class") or ("none" if status == "shipped" else "unclassified")).strip().lower() if failure not in FAILURE_CLASSES: print(f"error: claim {item['id']} has invalid failure_class {failure!r}", file=sys.stderr) return 1 by_status[status] += 1 by_failure[failure] += 1 result = { "total": len(data), "by_status": dict(sorted(by_status.items())), "by_failure_class": dict(sorted(by_failure.items())), } print(json.dumps(result, indent=2)) return 0 if __name__ == "__main__": sys.exit(main()) ``` ## Output Contract A completed retrospective produces all of the following: - One retro document with the seven sections from step 8, in order. - The claim list JSON file used by the helper, saved with the retro document. - The helper output copied into the counts section, matching a fresh run of `retro_tally.py` on the saved claim list. - A change list of 3 changes as guidance, each with an applying location, the finding it answers, and a check method for the next period. Any change past the cap carries a written reason. - A follow-up section that records a result for every change carried from the previous retro, or the line `no prior retro; repeat detection starts next period` when there is no previous retro. ## Operating Rules 1. Evidence first. No claim is judged before the run logs, claim list, and shipped artifacts for the period are collected. 2. One row per claim. Do not merge, split, drop, or soften a claim to make the counts look better. 3. The artifact decides the status. Reports and memory do not override a missing or mismatched artifact. 4. One primary failure class per failed claim. Record secondary causes in the note column only. 5. Copy helper counts exactly. Fix the claim list and rerun the helper instead of editing printed numbers. 6. Cap changes at 3 per retro as guidance. Quality of the check method matters more than the number of changes. 7. Every change names an action, a place it applies, and a check. Vague advice is rewritten or dropped before the document is finished. 8. Repeat findings outrank new findings. A pattern seen in 2 or more consecutive periods is addressed before any first-time pattern. 9. Follow-up checks run at the start of the next retro, not at the end of the current one. Results are recorded per change, never as a single summary line. 10. Freeze the period. Late completions are recorded in the next period and noted as late, not backdated into this one. 11. If evidence for a claim cannot be found, say so in the table and classify from what exists. Do not invent logs, artifacts, or causes. 12. Keep the retro about the work: claims, artifacts, classes, and changes. Leave praise and blame out of the findings.