--- name: "autonomous_loop_ops" description: "Operate long-running agent loops: define progress signals before launch, monitor them on a schedule, detect stalls, intervene on an escalation ladder, and shut down gracefully. Trigger when launching a background agent loop, checking whether one is still healthy, deciding whether a stuck loop should be nudged or killed, or winding a loop down on purpose." --- # Autonomous Loop Ops ## Purpose A long-running agent loop fails silently more often than it fails loudly. This skill is the operator discipline for any loop that runs minutes to days in the background: cron-style agents, worker sweeps, conversion jobs, scheduled checks. It tells you how to watch a loop so you catch stalls, how to tell a stall from normal slowness, what to try before killing anything, and how to shut down cleanly when the work is done or the run is beyond saving. This covers live loops. If the run is already dead and you need checkpoints, classification, and resume strategies, that is post-mortem recovery, a different job. ## Workflow ### 1. Define progress before launch A loop is not healthy because its process is alive. It is healthy because something is advancing. Before starting, write down: - **The progress artifact**: the log file, output file, counter, or queue-depth metric that must move. One concrete thing, not a feeling. - **The expected cadence**: how often that artifact should change when things are normal (every 30 seconds, every few minutes, hourly). - **The stall threshold**: 3 to 5 times the expected cadence. Ordinary slowness must never trip it. - **The abort limit**: max iterations, max wall-clock time, max cost. This is the last line of defense and it must exist even for loops that should run forever. If the job prints nothing for long stretches, make it emit a periodic timestamp or progress line. Silence you cannot measure is silence you cannot diagnose. ### 2. Arm the guard at launch A background loop is not launched until its monitor is launched. Both start in the same move. The guard is a separate process that watches the progress artifact, not the loop's code: ```bash # launch the loop and capture the real pid, not a name ./the-loop.sh >> loop.log 2>&1 & LOOP_PID=$! # arm the guard: pid, stall threshold in seconds, artifact(s) that must move bin/loop-watch.sh "$LOOP_PID" 1800 loop.log progress.json >> watch.log 2>&1 & ``` Your launch report states three things: the pid, what the guard watches, and what threshold kills the job. If you cannot say all three, the loop is unguarded. ### 3. Check health with evidence A health check answers one question: **what moved since the last check?** Accept any of these as evidence: - the progress artifact grew, or its mtime advanced - the log gained lines - a counter the loop prints went up - the queue drained or the work item in flight changed If nothing moved, the loop is stalled however busy it looks. CPU pegged at 100% can be a tight loop going nowhere; a live process can be fully deadlocked. Report stalled and diagnose, never report healthy on liveness alone. Name the state precisely. Active work, expected delay, stall, failure, completion, and intentional pause are six different states. A paused process and a hung one both show frozen artifacts, but only one of them may be restarted. ### 4. Diagnose before you touch anything When the guard fires or a check finds a stall, follow the triage tree in `references/stall-recovery-playbook.md`. The rules that matter most: - **Confirm the stall first.** One quiet check is a data point, two consecutive no-progress checks is a stall. A single threshold trip on a shared machine can be contention. - **Read what the loop was chewing on** with an independent tool before relaunching, so a bug in the loop is not misdiagnosed as a corrupt file or a failing disk. - **Partial progress is still a stall.** A run that did 539 items and then stopped looks healthy: long log, real outputs, busy process list. Compare the artifact's mtime against *now*, not against zero. A counter that moved every few minutes and has not moved in an hour means the run is stalled even though most of it succeeded. - **Progress signals plateau.** An archive reaches full size while the tool still builds its index, so a size-only watcher reads a finished write as a stall. Pair size with a second signal (process still exists, a fresh log line, the temp file still present) and confirm the job is really gone before acting. ### 5. Intervene on the ladder Do not jump to the kill. Work the escalation ladder in order, and stop at the first rung that works: 1. **Nudge**: ask the loop to report or continue (a message, a flag file it polls). Cheap, safe, often enough for a loop that lost its way briefly. 2. **Escalate the instruction**: give it a bounded new directive, such as abandon the current item and move to the next, or checkpoint and pause. Bound the new instruction the same way you bounded the loop. 3. **Quarantine**: switch the loop's tools and writes to read-only, preserve its state, stop it accepting new work. This freezes the damage without destroying the evidence. 4. **Terminate**: signal the whole process group (`kill -TERM -`, then `kill -KILL -` if needed), verify by pgid that nothing survived, clear the run's lock and any partial files. Signal the group, never just the script pid: a shell blocked in a pipeline ignores signals to the script until the foreground command returns. Never restart a paused task without the owner's word. Never launch a duplicate of an uncertain job; one orphaned worker plus a fresh run writing the same destination produces bogus duplicate outputs. Never kill a loop another session started without first looking for its guard and reading its notes. ### 6. Shut down gracefully A graceful shutdown is an ordered teardown, not a signal. Work the checklist in `references/graceful-shutdown-checklist.md`. The shape of it: stop accepting new work, let the in-flight item finish or checkpoint, flush and checkpoint state, write a shutdown reason code, release resources and locks, verify by pid and pgid that nothing survived. After shutdown, record what happened: root cause, how long detection took, what it cost. That record is what makes the next threshold better. ### 7. Keep the evidence Whatever happens, the logs, the guard's history, the progress artifacts, and the shutdown reason survive. Safety controls that destroy their own evidence make the next incident unknowable. A kill switch that has never been tested is an assumption, not a control: exercise the guard on a short test run before trusting it on a long one. ## Output Contract Every monitoring report, whether a scheduled check or a final teardown report, carries: - **Loop identity**: what it is, pid and pgid, when it started, who launched it - **State**: exactly one of active, delayed, stalled, failed, complete, paused - **Evidence**: what moved since the last check and when (artifact name, mtime, counter values) - **Guard config**: threshold, polling interval, abort limits - **Actions taken**: which ladder rung, why, and the outcome - **Next check**: when the next scheduled check runs, or that the loop is closed If you cannot fill in the evidence line, you do not have a health report yet. ## Operating Rules 1. Never call a loop healthy on liveness, CPU, or elapsed time. Name the artifact that moved. 2. The guard launches with the loop, in the same move, every time. A guard that exists but was not armed is an unguarded loop. 3. Stall thresholds come from expected cadence: 3 to 5 times the normal interval between progress signs. Thresholds from vibes generate pager fatigue or miss real stalls. 4. Bound per-item work, not only the run as a whole. Wrap each unit of work in `timeout --kill-after=10s` so one pathological input dies instead of the batch (exit code 124 means the timeout fired). 5. Watch the pid captured at launch. Pattern-matching a process by name can match the guard itself, a stale shell, or the wrapper after it has `exec`ed away. 6. Confirm before acting on idleness: pair every quiet signal with a second one, and verify the process is really gone before killing by pgid. 7. Interventions follow the ladder: nudge, escalate, quarantine, terminate. Termination is the last rung, not the first. 8. Write shutdown reason codes. "Loop stopped" explains nothing; "stalled: artifact idle 62m over 30m threshold, killed by pgid" explains everything. 9. Relieving load is not a reason to pause. A paused loop's frozen artifacts read as a stall and the guard will kill it on schedule. Use `renice` to yield CPU; there is no safe unilateral move for disk contention, report it to the loop's owner. 10. After every incident, update the record: detection latency, root cause, cost, and whether the thresholds need to change.