webhook_design_patterns v1.0.0

name: "webhook_design_patterns" description: "Design and debug webhooks on both sides of the wire: verify HMAC signatures (Stripe, GitHub, Shopify, Slack, Svix), deduplicate at-least-once deliveries, ack fast and queue slow work, retry with exponential backoff, and dead-letter the failures. Triggers on 'webhook signature', 'duplicate webhooks', 'webhook retries', 'dead letter queue', or 'debugging webhook delivery'."

Webhook Design Patterns

Purpose

Make webhooks trustworthy and lossless. A webhook system has two halves: the receiver (verify, dedupe, ack fast) and the sender (sign, retry, dead-letter). This skill covers both, with tested code in examples/webhook_toolkit.py.

Workflow

Receiver side: you take webhooks from a provider.

1. Verify before parsing. Capture the RAW body bytes and check the signature over those exact bytes. Never json.loads first. Framework body-parsers that deserialize before you see the bytes will silently break verification; disable auto-parse for the webhook route.
2. Enforce the replay window. Where the signature carries a timestamp (Stripe, Svix, Slack), reject anything older than 5 minutes. Where it does not (GitHub, GitLab), rely on delivery-ID dedup.
3. Dedupe before doing work. Webhooks are at-least-once. Claim the event ID (provider's event/delivery ID) in a table with a UNIQUE constraint before touching side effects. Duplicate arrival returns 200 without re-running the work.
4. Ack fast, work later. Return 2xx immediately; anything over ~200ms goes to a background queue. Slow handlers cause provider timeouts, which cause retry storms.
5. Fail bad signatures loudly. Log headers + body, return 400, alert. Never process unverified payloads.

Sender side: you deliver webhooks to subscribers.

1. Sign everything. HMAC-SHA256 over the raw payload with a per-subscription secret; include a timestamp so receivers can reject replays. Send a ping event when a subscription is created so the receiver can verify setup.
2. Retry with backoff and jitter. Schedule: immediate, 30s, 5m, 30m, 2h, 12h. 2xx = success. Retry 429/5xx/timeouts. 410 Gone = permanent: auto-unsubscribe the endpoint. Timeout each attempt (~10-30s).
3. Dead-letter after exhaustion. Park the raw payload, event ID, attempt history, and error in a DLQ. Alert on DLQ growth. Support inspect -> fix -> replay, and replay stays safe because every step is idempotent.
4. Protect yourself. SSRF-check subscriber URLs (refuse loopback/private/link-local/cloud-metadata), set per-endpoint rate limits, log every attempt with the webhook ID for tracing.

Output Contract

An agent applying this skill produces one of:

• A webhook receiver: signature verification middleware + idempotency claim + fast ack + queue handoff, with per-provider header/algorithm config.
• A webhook sender: signing + per-endpoint retry schedule + DLQ + replay path.
• A debugging session: replayable test events, signature recomputation against a failing payload, and a diagnosis rooted in one of the rules above.
• Each includes the subscription/verification config the provider needs (headers, secret location, subscribed event types), not just code.

Operating Rules

• Every number, header name, algorithm, and signed-content layout in this skill was verified against provider docs (see references/provider-schemes.md). Do not improvise new schemes from memory; check the reference table.
• Use hmac.compare_digest / timing-safe comparison, never ==. Compare against the FULL header value, including prefixes like sha256= and v1=.
• Never verify over re-serialized JSON. If a framework only exposes parsed bodies, add middleware that captures raw bytes first.
• Idempotency keys must come from the provider (event ID, delivery ID), not from content hashes you invent, unless the provider genuinely supplies none.
• Cap retries and always dead-letter. "Retry forever" is a slow outage.
• examples/webhook_toolkit.py is the canonical snippet source: copy from it, don't re-derive the crypto.