queue_design v1.0.0

name: queue_design description: Design and review durable job queues: at-least-once delivery semantics, lease and visibility-timeout mechanics, retry policy with backoff and jitter, dead-letter queues, and backpressure handling between producers and consumers. Triggers when someone builds, audits, or debugs a background job queue, work queue, or task worker system.

Queue Design

Purpose

Give agents a correct, compact mental model and an actionable checklist for building or auditing a durable job queue: how delivery guarantees actually work (ack timing decides them), how to size leases and retries so failures heal instead of loop, how dead-letter queues isolate poison messages, and how to keep producers from outrunning consumers (backpressure).

Workflow

1. Name the guarantee. Decide what failure modes are acceptable. Default to at-least-once delivery plus idempotent workers. Only choose at-most-once for work where loss is cheap (low-value telemetry). Do not claim exactly-once from queue machinery alone.
2. Set the lease. Pick a visibility timeout that comfortably exceeds expected processing time (a common starting point: 2x the p99 processing time). For long jobs, extend the lease while work continues (SQS ChangeMessageVisibility; max 12 hours). Keep ack latency low; sub-second is a good target.
3. Define retries. Set a max attempt count, then dead-letter. Use exponential backoff with full jitter between attempts so a recovering downstream is not hit by a thundering herd.
4. Wire the dead letter queue. After max attempts, move the job to a DLQ carrying the original payload, the attempt history, and the final error. Alert on DLQ growth, since growth means something systemic is wrong, not one flaky job.
5. Design backpressure both ways. Producers must slow down or shed load (return 429) when queue depth crosses a threshold. Consumers must scale with queue depth and stop pulling when the downstream is sick (circuit breaker), extending leases on in-flight work while paused.
6. Instrument the four signals. Queue depth, age of oldest message, redelivery rate, DLQ depth. Rising oldest-message age means consumers cannot keep up; rising redelivery means leases are too short or workers are crashing mid-job.

Output Contract

When asked to design or review a queue, produce:

• The chosen delivery guarantee and why it fits (at-least-once + idempotent workers is the default answer).
• Lease/visibility timeout value with the reasoning (relative to measured processing time).
• Retry policy: max attempts, backoff formula, jitter, per-attempt cap.
• DLQ policy: trigger threshold, payload contents, retention, alert condition.
• Backpressure plan: producer-side threshold behavior, consumer scaling rule, circuit-breaker behavior.
• The four metrics and their alert thresholds.

Operating Rules

1. Ack AFTER processing for at-least-once. Ack BEFORE processing is at-most-once: work is lost on crash. There is no ack ordering that gives exactly-once.
2. Every factual claim about a specific broker (SQS limits, retention windows, defaults) must come from the vendor docs or a verified source. See references/queue-patterns.md for broker-neutral guidance.
3. Never tune a queue from memory alone. Base timeouts on measured processing times and re-check them when the workload changes.
4. A DLQ without an alert is a trash can. If nothing pages on DLQ growth, the DLQ does not count as a design feature.
5. Poison messages must be isolated, not retried forever. Any design that can loop the same failing job indefinitely is broken by construction.