webhook-reliability v1.0.0

name: webhook-reliability description: Design and harden webhook integrations on both ends. Covers HMAC signature verification, timestamped replay defense, idempotent receivers, sender retry schedules, and manual replay testing. Use when building, fixing, or reviewing a webhook endpoint or a webhook sender.

Webhook Reliability

Purpose

Webhooks break in a handful of predictable ways: forged deliveries hitting an unverified endpoint, captured payloads replayed later, the same event processed twice because a retry arrived during an outage, and silent data loss after the provider exhausts its retries and disables the endpoint. This skill covers both sides of the wire. The receiver side verifies, dedups, and acknowledges fast. The sender side signs, retries on a published schedule, and offers redelivery. The four documents here are the whole skill; nothing else is needed to ship.

Workflow

1. Name the direction. Receiving from a provider (Stripe, GitHub, a custom sender) is the usual case. Sending to customer endpoints is the second. If you own both ends, do both checklists.
2. Receiver. Work through references/receiver-checklist.md in order: capture raw bytes, verify the signature, check the timestamp, dedup by event id, acknowledge fast, process from a queue. Do not reorder these steps.
3. Sender. Adopt a signing scheme (prefer Standard Webhooks: webhook-id, webhook-timestamp, webhook-signature; keep a provider's legacy scheme if one already exists), pick a retry schedule from references/retry-schedules.md, and ship a manual redelivery path. Publish the retry window to customers; it is a promise about how long they can be down before data is at risk.
4. Prove it. Replay a captured delivery against a staging endpoint and confirm four distinct, logged outcomes: accepted, bad signature, stale timestamp, duplicate. Use references/verify.py as the oracle; its self-test demonstrates all four.

Output Contract

• A receiver that verifies signatures against the raw request bytes, enforces a timestamp window (5 minutes is the sane default), dedups by event id in durable storage, and returns 2xx inside the provider timeout. Slow work happens in a background worker, never in the request handler.
• A sender with a documented retry schedule (multi-day exponential backoff with jitter), a fresh timestamp and signature on every attempt, a delivery id that stays stable across retries of the same delivery, and a manual redelivery mechanism for operators.
• A staging replay endpoint that reproduces the four verification outcomes on demand, so the next incident gets debugged against history instead of a partner's production traffic.

Operating Rules

1. Verify against the exact bytes off the wire. Frameworks parse JSON before your handler runs; re-serializing it changes key order, whitespace, or number formatting, and the signature will never match. Parse only after the signature checks out.
2. Compare in constant time (hmac.compare_digest and equivalents). == leaks how much of the signature was right.
3. Fail closed on every verification problem: missing header, bad signature, stale timestamp all get a 4xx. Never log the mismatch and continue; that is the same as having no verification.
4. A signature proves who sent the payload, not when. Without a signed timestamp inside the digest and a tolerance window on the receiver, any captured delivery replays forever.
5. Acknowledge in seconds. Provider timeouts run 3 to 10 seconds. If the handler writes to a database or calls an API before responding, that work goes behind a queue.
6. The dedup key must outlive the retry span. A provider that retries for 3 days needs days of stored event ids, not a 10-minute in-memory cache. The idempotency record and the side effect belong in the same transaction.
7. An unrecognized event type is not an error. Providers add types over time; answer 2xx and ignore it.
8. Secrets live in a manager, never in code or logs. Rotate with an overlap window where both the old and new secret verify. Use a distinct secret per environment and per endpoint.
9. Retries are part of the contract, so delivery is at-least-once and unordered. Retried events can arrive out of order; refetch canonical state or carry a version field instead of assuming sequence.