---
name: "webhook_design_patterns"
description: "Design and debug webhooks on both sides of the wire: verify HMAC signatures (Stripe, GitHub, Shopify, Slack, Svix), deduplicate at-least-once deliveries, ack fast and queue slow work, retry with exponential backoff, and dead-letter the failures. Triggers on 'webhook signature', 'duplicate webhooks', 'webhook retries', 'dead letter queue', or 'debugging webhook delivery'."
---

# Webhook Design Patterns

## Purpose

Make webhooks trustworthy and lossless. A webhook system has two halves: the receiver (verify, dedupe, ack fast) and the sender (sign, retry, dead-letter). This skill covers both, with tested code in `examples/webhook_toolkit.py`.

## Workflow

**Receiver side: you take webhooks from a provider.**

1. **Verify before parsing.** Capture the RAW body bytes and check the signature over those exact bytes. Never `json.loads` first. Framework body-parsers that deserialize before you see the bytes will silently break verification; disable auto-parse for the webhook route.
2. **Enforce the replay window.** Where the signature carries a timestamp (Stripe, Svix, Slack), reject anything older than 5 minutes. Where it does not (GitHub, GitLab), rely on delivery-ID dedup.
3. **Dedupe before doing work.** Webhooks are at-least-once. Claim the event ID (provider's event/delivery ID) in a table with a UNIQUE constraint before touching side effects. Duplicate arrival returns 200 without re-running the work.
4. **Ack fast, work later.** Return 2xx immediately; anything over ~200ms goes to a background queue. Slow handlers cause provider timeouts, which cause retry storms.
5. **Fail bad signatures loudly.** Log headers + body, return 400, alert. Never process unverified payloads.

**Sender side: you deliver webhooks to subscribers.**

1. **Sign everything.** HMAC-SHA256 over the raw payload with a per-subscription secret; include a timestamp so receivers can reject replays. Send a `ping` event when a subscription is created so the receiver can verify setup.
2. **Retry with backoff and jitter.** Schedule: immediate, 30s, 5m, 30m, 2h, 12h. 2xx = success. Retry 429/5xx/timeouts. 410 Gone = permanent: auto-unsubscribe the endpoint. Timeout each attempt (~10-30s).
3. **Dead-letter after exhaustion.** Park the raw payload, event ID, attempt history, and error in a DLQ. Alert on DLQ growth. Support inspect -> fix -> replay, and replay stays safe because every step is idempotent.
4. **Protect yourself.** SSRF-check subscriber URLs (refuse loopback/private/link-local/cloud-metadata), set per-endpoint rate limits, log every attempt with the webhook ID for tracing.

## Output Contract

An agent applying this skill produces one of:

- A webhook receiver: signature verification middleware + idempotency claim + fast ack + queue handoff, with per-provider header/algorithm config.
- A webhook sender: signing + per-endpoint retry schedule + DLQ + replay path.
- A debugging session: replayable test events, signature recomputation against a failing payload, and a diagnosis rooted in one of the rules above.
- Each includes the subscription/verification config the provider needs (headers, secret location, subscribed event types), not just code.

## Operating Rules

- Every number, header name, algorithm, and signed-content layout in this skill was verified against provider docs (see `references/provider-schemes.md`). Do not improvise new schemes from memory; check the reference table.
- Use `hmac.compare_digest` / timing-safe comparison, never `==`. Compare against the FULL header value, including prefixes like `sha256=` and `v1=`.
- Never verify over re-serialized JSON. If a framework only exposes parsed bodies, add middleware that captures raw bytes first.
- Idempotency keys must come from the provider (event ID, delivery ID), not from content hashes you invent, unless the provider genuinely supplies none.
- Cap retries and always dead-letter. "Retry forever" is a slow outage.
- `examples/webhook_toolkit.py` is the canonical snippet source: copy from it, don't re-derive the crypto.