failure_recovery_playbooks v1.0.0

name: failure_recovery_playbooks description: Recovers an agent run that crashed, stalled in a loop, or died with partial state: checkpoint discipline, failure classification, and resume strategies. Use when an agent run dies mid-task, a loop stops making progress, or partial work needs to be resumed instead of restarted. metadata: { "includeInPrompt": true }

Failure Recovery Playbooks

Purpose

Long agent runs die. Processes crash, loops stall, sessions vanish, and tools stop answering. Restarting from zero throws away everything the run already finished. This skill is the run-recovery playbook: keep cheap checkpoints while you work, classify the failure when it happens, and resume from the last good point instead of starting over. It operates at the agent-run level: crashes, stuck loops, and partial state. It does not cover swapping models or providers, routing requests, or degrading prompts.

Workflow

1. Checkpoint as you go

A checkpoint is a small, fast-to-read record of where the run stands. Write one:

• After every completed step, not just at the end of the run.
• Before every irreversible step (a send, a delete, a publish, a payment), so you can check what actually happened before repeating it.
• Into one predictable location for the whole run, for example a checkpoint.json in the run's working directory. One file, always overwritten, always the same shape.

Each checkpoint entry carries: step_id, status (done, in_progress, failed), started_at and finished_at timestamps, inputs (what the step consumed), outputs (what it produced, or the identifiers the result returned: post ids, order numbers, file paths), and next_step (what should run after this one). Keep it small. It is a bookmark, not a log.

2. Classify the failure

Read the evidence before you touch anything. The four classes look different:

• Crashed. The run process is gone or the session is unreachable, but the checkpoint file exists and the last entry is marked in_progress. Heartbeat or progress timestamps stop abruptly. Nothing is running to observe.
• Stuck loop. The run is alive but the last several actions are near-identical tool calls with near-identical results, or output deltas are zero across multiple cycles (same file written, same status returned, same page seen). The giveaway is repetition without progress, not silence.
• Partial state. The run died partway through a multi-step sequence and some steps committed while others did not. The checkpoint shows a mix of done and in_progress/failed, and outside state (files, records, sent items) needs checking to see which steps actually landed.
• Tool blocked. One tool call keeps failing or never returns: permission denied, rate-limited, service down. The failure is attached to a single dependency, not to the run itself. Everything before the blocked call is fine.

Two traps when classifying. First, a missing failure report is not proof nothing happened. An action that returned an error may still have committed on the other side; verify before repeating anything irreversible. Second, distinguish stuck from slow. One long-running tool call can look like a hang for minutes. If the tool supports status checks or progress output, poll it with a sensible interval before declaring the run stuck.

3. Recover, per class

• Crashed: read the checkpoint, re-run the last in_progress step first (only after confirming its side effects did not land), then continue from next_step. Do not re-run steps already marked done.
• Stuck loop: do not stack another identical retry. Change the input, not the attempt count. Alter the query, narrow the scope, switch the strategy, or split the step into smaller pieces. If the loop persists across strategy changes, stop and report it as blocked instead of burning more cycles.
• Partial state: audit committed steps against their recorded outputs before resuming. For any step whose output identifiers are missing or unverified, check the real destination (the inbox, the file listing, the records system) to see whether it landed. Resume only the steps that are genuinely missing.
• Tool blocked: back off with jitter at the agent-run level: wait, then re-check status or retry the single blocked call with growing intervals plus randomness so parallel runs do not all hammer at once. After a small bounded number of attempts, either work around the dependency or mark the run blocked and surface it.

Decide where the recovered run lives. Resume in place when the run's session and context are intact and the failure was transient. Start a fresh agent with the checkpoint when the session is gone, the context is poisoned by the loop, or the run has been dead long enough that its in-memory state cannot be trusted. Either way, the checkpoint is what carries the work forward.

4. Verify the recovery

After the recovery move, confirm the run actually made progress. The check is simple: the next checkpoint entry must show a done step whose outputs differ from the pre-failure state, or a changed state at the real destination. If the recovered run just re-entered the same loop or re-failed the same call, you did not recover it. Reclassify and pick a different move.

Output Contract

A recovery plan names three things, in order:

1. Failure class: one of crashed, stuck-loop, partial-state, tool-blocked, plus the evidence (checkpoint timestamps, repeated call signatures, zero-delta cycles, error strings).
2. Recovery move: resume-in-place or fresh-agent-with-checkpoint, the exact step to re-run or skip, and what changed versus the failed attempt (new input, new scope, backoff schedule).
3. Verification check: how you will confirm progress after the move (next checkpoint diff, destination state to inspect).

A checkpoint entry looks like this, inline in the run's checkpoint file:

json
{
  "step_id": "post-batch-3",
  "status": "done",
  "started_at": "2026-10-05T07:10:00-05:00",
  "finished_at": "2026-10-05T07:11:22-05:00",
  "inputs": {"batch": 3, "items": ["a", "b", "c"]},
  "outputs": {"posted_ids": ["88112", "88113"], "skipped": ["c"]},
  "next_step": "post-batch-4"
}

Operating Rules

• A failed report is not proof nothing happened. Check the destination before repeating any irreversible action.
• Checkpoint before every irreversible step, not after. You cannot verify what you never recorded.
• Distinguish stuck from slow. One long tool call looks like a hang; poll for status or progress output before calling it dead.
• Never stack identical retries on a stuck loop. Change the input, narrow the scope, or split the step. Same call, same result, more times, is not a strategy.
• When the checkpoint file itself is missing or corrupt, rebuild from the destination state (files on disk, records in the system), not from memory of what the run was supposed to do.
• Prefer resume-in-place for transient failures and fresh-agent-with-checkpoint for dead sessions or poisoned context. Pick deliberately; the default is not always the right one.
• If two different recovery moves have both failed, stop the run and surface it. A third guess is rarely cheaper than a human look.