schedule_design v1.0.0

name: "schedule_design" description: "Design reliable scheduled jobs for agents: idempotency, watermarks, single-flight locks, failure alerting, and safe retries."

Schedule Design

A cron job that runs fine for a month and then double-sends, silently stops, or corrupts state on a retry is worse than no job at all. Design for the failure modes, not the happy path.

Workflow

1. Make every run idempotent. The job will run twice someday (overlap, retry, manual re-run). Design so the second run is a no-op:

• Key writes by a natural idempotency key (date, entity id, run id), not by "insert a new row."
• Check-then-act: "if not already done for 2026-09-17, do it."
• Prefer upserts over inserts; prefer "ensure state" over "append event."

1. Track progress with watermarks, not memory.

• Persist last_seen_id, last_run_at, last_success_at to disk (a JSON state file, a DB row) — never a variable.
• Each run reads the watermark, processes only what's new, and advances the watermark only after the work is durably done. A crash between "work done" and "watermark advanced" must re-do work safely (see step 1).
• Never advance the watermark on a failed run.

1. Prevent overlapping runs.

• Use a lock: lockfile with PID, a DB advisory lock, or the scheduler's own single-flight option.
• Decide the overlap policy up front and write it down: skip (new run exits if one is active), queue, or kill-and-restart. Default to skip for polling jobs.
• Stale-lock handling: if the lock holder's PID is dead, the lock is stale — break it and log loudly. A job that never runs again because of a lock left by a crashed run is a silent outage.

1. Fail loudly, succeed quietly.

• Every run appends to a run log: start time, end time, items processed, outcome. Keep the last N runs; this is the first diagnostic when something looks wrong.
• Alert on: consecutive failures (2–3 in a row, not one — transients happen), runs that produce zero output when they usually produce some, and runs that take 3x their normal duration.
• A job that fails silently for a week is the most common scheduled- job failure. The alerting is the job; the work is secondary.

1. Schedule sanely.

• Double-check cron syntax before committing: 0 /6 is every 6 hours; /5 * is every 5 minutes. Dry-run the expression first.
• Avoid top-of-hour thundering: :07 or :23 past the hour spreads load and dodges everyone else's :00 jobs.
• Respect quiet hours for anything user-visible. Batch overnight pings into one morning digest.
• Timezone explicit, always. Server-local time is a bug report from the future.

1. Retries belong inside the job, not just around it.

• Transient step fails (one API call 503s) → retry that step with backoff, don't fail the whole run.
• Persistent step fails (auth rejected) → fail the run fast and alert. Retrying a 401 fifty times is noise.
• Cap total run time. A run that hangs forever holds the lock forever.

Output Contract

Every scheduled job ships with: the schedule expression and timezone, the idempotency key, the watermark location, the overlap policy, the run-log location, and what triggers an alert. If any of these is missing, the job isn't done.

Operating Rules

1. No scheduled job without alerting. "I'll notice" is not alerting.
2. State on disk, never in memory, between runs.
3. Test the failure paths: kill -9 the job mid-run and confirm the next run recovers cleanly.
4. One job, one responsibility. A cron entry that does five things is five jobs sharing one failure domain.
5. Review schedules periodically. Dead jobs accumulate; each one is a future 3am page for no reason.