---
name: "schedule_design"
description: "Design reliable scheduled jobs for agents: idempotency, watermarks, single-flight locks, failure alerting, and safe retries."
---

# Schedule Design

A cron job that runs fine for a month and then double-sends, silently
stops, or corrupts state on a retry is worse than no job at all. Design
for the failure modes, not the happy path.

## Workflow

1. **Make every run idempotent.** The job will run twice someday
   (overlap, retry, manual re-run). Design so the second run is a no-op:
   - Key writes by a natural idempotency key (date, entity id, run id),
     not by "insert a new row."
   - Check-then-act: "if not already done for 2026-09-17, do it."
   - Prefer upserts over inserts; prefer "ensure state" over
     "append event."

2. **Track progress with watermarks, not memory.**
   - Persist `last_seen_id`, `last_run_at`, `last_success_at` to disk (a
     JSON state file, a DB row) — never a variable.
   - Each run reads the watermark, processes only what's new, and
     advances the watermark only after the work is durably done. A crash
     between "work done" and "watermark advanced" must re-do work safely
     (see step 1).
   - Never advance the watermark on a failed run.

3. **Prevent overlapping runs.**
   - Use a lock: lockfile with PID, a DB advisory lock, or the
     scheduler's own single-flight option.
   - Decide the overlap policy up front and write it down: *skip* (new
     run exits if one is active), *queue*, or *kill-and-restart*.
     Default to skip for polling jobs.
   - Stale-lock handling: if the lock holder's PID is dead, the lock is
     stale — break it and log loudly. A job that never runs again
     because of a lock left by a crashed run is a silent outage.

4. **Fail loudly, succeed quietly.**
   - Every run appends to a run log: start time, end time, items
     processed, outcome. Keep the last N runs; this is the first
     diagnostic when something looks wrong.
   - Alert on: consecutive failures (2–3 in a row, not one — transients
     happen), runs that produce zero output when they usually produce
     some, and runs that take 3x their normal duration.
   - A job that fails silently for a week is the most common scheduled-
     job failure. The alerting is the job; the work is secondary.

5. **Schedule sanely.**
   - Double-check cron syntax before committing: `0 */6 * * *` is every
     6 hours; `*/5 * * * *` is every 5 minutes. Dry-run the expression
     first.
   - Avoid top-of-hour thundering: `:07` or `:23` past the hour spreads
     load and dodges everyone else's `:00` jobs.
   - Respect quiet hours for anything user-visible. Batch overnight
     pings into one morning digest.
   - Timezone explicit, always. Server-local time is a bug report from
     the future.

6. **Retries belong inside the job, not just around it.**
   - Transient step fails (one API call 503s) → retry that step with
     backoff, don't fail the whole run.
   - Persistent step fails (auth rejected) → fail the run fast and
     alert. Retrying a 401 fifty times is noise.
   - Cap total run time. A run that hangs forever holds the lock forever.

## Output Contract

Every scheduled job ships with: the schedule expression and timezone,
the idempotency key, the watermark location, the overlap policy, the
run-log location, and what triggers an alert. If any of these is missing,
the job isn't done.

## Operating Rules

1. No scheduled job without alerting. "I'll notice" is not alerting.
2. State on disk, never in memory, between runs.
3. Test the failure paths: kill -9 the job mid-run and confirm the next
   run recovers cleanly.
4. One job, one responsibility. A cron entry that does five things is
   five jobs sharing one failure domain.
5. Review schedules periodically. Dead jobs accumulate; each one is a
   future 3am page for no reason.