---
name: "webhook_handling"
description: "Receiving webhooks safely: signature verification, retry semantics, idempotency keys, and dead-letter handling for agents building inbound event endpoints. Triggers on: accept webhooks, verify webhook signature, handle Stripe/GitHub webhooks, idempotent webhook handler, webhook retries."
metadata: { "includeInPrompt": true }
---

## Purpose

Give an agent a safe, correct pattern for building an **inbound** webhook endpoint — one that accepts events from a provider, verifies they are authentic, survives provider retries, never processes the same event twice, and fails gracefully instead of silently. The skill covers four load-bearing pieces: signature verification (the generic HMAC-SHA256 pattern), acknowledge-fast/process-async, idempotent handling, and dead-letter routing. This is about *receiving* webhooks; outbound event emission and API scraping are different problems with different patterns.

## Workflow

1. **Verify the signature before doing anything else.** The generic pattern across providers: the sender computes HMAC-SHA256 over the **raw request body bytes** with a shared secret and places the result in a header. You recompute it and compare with a constant-time compare. Providers vary the header name and may add a prefix (e.g. `sha256=`) or a multi-part scheme — strip the documented prefix, then compare the raw digests. Verified locally (stdlib only: `hashlib` + `hmac`):

```python
import hashlib, hmac

def verify_webhook(body: bytes, received_sig: str, secret: bytes) -> bool:
    # Hash the RAW body bytes. Do NOT parse JSON and re-serialize first:
    # whitespace/key order can change and break the digest.
    expected = hmac.new(secret, body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, received_sig)
```

Tested end to end: valid signature passes; the same signature on a tampered body fails; the same signature verified against the wrong secret fails; a provider-style `sha256=`-prefixed header passes after stripping the prefix. If verification fails, return an HTTP 4xx and drop the request — it is unauthenticated input, not a retryable failure.

2. **Acknowledge fast, process async.** Return a 2xx HTTP response as soon as the event is *durably accepted* (verified + persisted to a queue or outbox table). Do the real work in a background consumer. Reason: providers treat a timeout as a delivery failure and will retry, which means your slow endpoint converts latency into duplicate deliveries. A 2xx means "I have it and own it now" — never "I processed it"; return 2xx only after durable acceptance, never on partial failure.

3. **Make the handler idempotent.** Providers retry aggressively (typically exponential backoff with jitter across minutes/hours), so your endpoint **will** see duplicates. Two layers:
   - **Dedupe on event ID.** Every webhook carries a unique event ID (header or payload). Store it with a unique constraint in your DB; a duplicate insert that conflicts on the event ID is a replay — acknowledge 2xx and skip. Set a TTL on dedupe records (a few days is a sensible default) so the table stays bounded; events arriving past the TTL are rare and must be treated as new, so your business logic needs layer two anyway.
   - **Naturally idempotent handlers.** Prefer upserts (`INSERT ... ON CONFLICT DO UPDATE`) and state-machine transitions guarded by current state ("only transition order X from `pending` → `paid`") over blind inserts or counters. If the business operation can't be made idempotent, the dedupe record is the gate — acquire it (insert event ID) *inside the same transaction* as the work, so a crash can't process the event without recording it.

4. **Route poison events to a dead-letter queue (DLQ), don't retry them forever.** Distinguish outcomes: **retryable** = transient 5xx, timeout, downstream unavailable → return 5xx so the provider retries. **Poison** = signature failure (4xx), schema validation failure, unknown event type, unprocessable business state → record and stop. After a small number of failed attempts (a common convention is ~3–5, tuned to your provider's retry budget), move the event to a DLQ table/queue with the raw body, headers, attempt count, and error, then **alert** (page or notify) — silent DLQ entries rot. Replay procedure: fix the bug or data, then re-drive DLQ entries through the same handler so the dedupe and idempotency logic applies; mark entries replayed, succeeded, or permanently discarded.

## Output Contract

After applying this skill, an agent should be able to produce:

- An endpoint handler that (a) verifies the HMAC signature on raw body bytes with constant-time compare before any other work, (b) returns 2xx only after durable acceptance, (c) dedupes on event ID with a unique constraint + TTL, and (d) routes unprocessable events to a DLQ with alerting.
- A short replay runbook: where DLQ entries live, what metadata they carry, and the command or procedure to re-drive them through the handler.

## Operating Rules

- **Signature over payload.** Never trust a field inside the payload (an `authenticated: true` flag, a matching account ID) as proof the event came from the provider. The HMAC over the raw body with the shared secret is the only authenticity check.
- **Constant-time compare, raw bytes.** Always use `hmac.compare_digest` (or your language's equivalent) for signature comparison — a plain `==` leaks timing information. Always hash the exact raw body bytes you received; parsing and re-serializing JSON before hashing is a classic silent breakage.
- **Freshness check blunts replays.** Prefer verifying a timestamp the provider includes (sent in the header or as part of the signed payload) and rejecting events older than a short window (a few minutes is a common default; match what your provider supports). This shrinks the window where a stolen signature can be replayed.
- **Secrets rotate, code doesn't.** Store the webhook secret in your secret manager / env (see `env-management`), never in code. When rotating, support the old and new secret briefly (try new, fall back to old) so the switchover doesn't drop events.
- **5xx retries, 4xx stops.** Return 5xx only for transient failures you expect a retry to fix; return 4xx for validation/auth failures so the provider does not burn retries on an event that will never succeed. Log both with the event ID.
- **Prefer not to accept what you can't verify.** If the provider's signature scheme is undocumented or you can't get the shared secret, do not expose the endpoint publicly — use a polling fallback or an allowlisted sender instead of an unverified webhook.