---
name: queue_design
description: Design and review durable job queues: at-least-once delivery semantics, lease and visibility-timeout mechanics, retry policy with backoff and jitter, dead-letter queues, and backpressure handling between producers and consumers. Triggers when someone builds, audits, or debugs a background job queue, work queue, or task worker system.
---

# Queue Design

## Purpose

Give agents a correct, compact mental model and an actionable checklist for building or auditing a durable job queue: how delivery guarantees actually work (ack timing decides them), how to size leases and retries so failures heal instead of loop, how dead-letter queues isolate poison messages, and how to keep producers from outrunning consumers (backpressure).

## Workflow

1. **Name the guarantee.** Decide what failure modes are acceptable. Default to at-least-once delivery plus idempotent workers. Only choose at-most-once for work where loss is cheap (low-value telemetry). Do not claim exactly-once from queue machinery alone.
2. **Set the lease.** Pick a visibility timeout that comfortably exceeds expected processing time (a common starting point: 2x the p99 processing time). For long jobs, extend the lease while work continues (SQS `ChangeMessageVisibility`; max 12 hours). Keep ack latency low; sub-second is a good target.
3. **Define retries.** Set a max attempt count, then dead-letter. Use exponential backoff with full jitter between attempts so a recovering downstream is not hit by a thundering herd.
4. **Wire the dead letter queue.** After max attempts, move the job to a DLQ carrying the original payload, the attempt history, and the final error. Alert on DLQ growth, since growth means something systemic is wrong, not one flaky job.
5. **Design backpressure both ways.** Producers must slow down or shed load (return 429) when queue depth crosses a threshold. Consumers must scale with queue depth and stop pulling when the downstream is sick (circuit breaker), extending leases on in-flight work while paused.
6. **Instrument the four signals.** Queue depth, age of oldest message, redelivery rate, DLQ depth. Rising oldest-message age means consumers cannot keep up; rising redelivery means leases are too short or workers are crashing mid-job.

## Output Contract

When asked to design or review a queue, produce:

- The chosen delivery guarantee and why it fits (at-least-once + idempotent workers is the default answer).
- Lease/visibility timeout value with the reasoning (relative to measured processing time).
- Retry policy: max attempts, backoff formula, jitter, per-attempt cap.
- DLQ policy: trigger threshold, payload contents, retention, alert condition.
- Backpressure plan: producer-side threshold behavior, consumer scaling rule, circuit-breaker behavior.
- The four metrics and their alert thresholds.

## Operating Rules

1. Ack AFTER processing for at-least-once. Ack BEFORE processing is at-most-once: work is lost on crash. There is no ack ordering that gives exactly-once.
2. Every factual claim about a specific broker (SQS limits, retention windows, defaults) must come from the vendor docs or a verified source. See `references/queue-patterns.md` for broker-neutral guidance.
3. Never tune a queue from memory alone. Base timeouts on measured processing times and re-check them when the workload changes.
4. A DLQ without an alert is a trash can. If nothing pages on DLQ growth, the DLQ does not count as a design feature.
5. Poison messages must be isolated, not retried forever. Any design that can loop the same failing job indefinitely is broken by construction.