webhook_handling v1.0.0

name: "webhook_handling" description: "Receiving webhooks safely: signature verification, retry semantics, idempotency keys, and dead-letter handling for agents building inbound event endpoints. Triggers on: accept webhooks, verify webhook signature, handle Stripe/GitHub webhooks, idempotent webhook handler, webhook retries." metadata: { "includeInPrompt": true }

Purpose

Give an agent a safe, correct pattern for building an inbound webhook endpoint — one that accepts events from a provider, verifies they are authentic, survives provider retries, never processes the same event twice, and fails gracefully instead of silently. The skill covers four load-bearing pieces: signature verification (the generic HMAC-SHA256 pattern), acknowledge-fast/process-async, idempotent handling, and dead-letter routing. This is about receiving webhooks; outbound event emission and API scraping are different problems with different patterns.

Workflow

1. Verify the signature before doing anything else. The generic pattern across providers: the sender computes HMAC-SHA256 over the raw request body bytes with a shared secret and places the result in a header. You recompute it and compare with a constant-time compare. Providers vary the header name and may add a prefix (e.g. sha256=) or a multi-part scheme — strip the documented prefix, then compare the raw digests. Verified locally (stdlib only: hashlib + hmac):

python
import hashlib, hmac

def verify_webhook(body: bytes, received_sig: str, secret: bytes) -> bool:
    # Hash the RAW body bytes. Do NOT parse JSON and re-serialize first:
    # whitespace/key order can change and break the digest.
    expected = hmac.new(secret, body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, received_sig)

Tested end to end: valid signature passes; the same signature on a tampered body fails; the same signature verified against the wrong secret fails; a provider-style sha256=-prefixed header passes after stripping the prefix. If verification fails, return an HTTP 4xx and drop the request — it is unauthenticated input, not a retryable failure.

1. Acknowledge fast, process async. Return a 2xx HTTP response as soon as the event is durably accepted (verified + persisted to a queue or outbox table). Do the real work in a background consumer. Reason: providers treat a timeout as a delivery failure and will retry, which means your slow endpoint converts latency into duplicate deliveries. A 2xx means "I have it and own it now" — never "I processed it"; return 2xx only after durable acceptance, never on partial failure.

1. Make the handler idempotent. Providers retry aggressively (typically exponential backoff with jitter across minutes/hours), so your endpoint will see duplicates. Two layers:

• Dedupe on event ID. Every webhook carries a unique event ID (header or payload). Store it with a unique constraint in your DB; a duplicate insert that conflicts on the event ID is a replay — acknowledge 2xx and skip. Set a TTL on dedupe records (a few days is a sensible default) so the table stays bounded; events arriving past the TTL are rare and must be treated as new, so your business logic needs layer two anyway.
• Naturally idempotent handlers. Prefer upserts (INSERT ... ON CONFLICT DO UPDATE) and state-machine transitions guarded by current state ("only transition order X from pending → paid") over blind inserts or counters. If the business operation can't be made idempotent, the dedupe record is the gate — acquire it (insert event ID) inside the same transaction as the work, so a crash can't process the event without recording it.

1. Route poison events to a dead-letter queue (DLQ), don't retry them forever. Distinguish outcomes: retryable = transient 5xx, timeout, downstream unavailable → return 5xx so the provider retries. Poison = signature failure (4xx), schema validation failure, unknown event type, unprocessable business state → record and stop. After a small number of failed attempts (a common convention is ~3–5, tuned to your provider's retry budget), move the event to a DLQ table/queue with the raw body, headers, attempt count, and error, then alert (page or notify) — silent DLQ entries rot. Replay procedure: fix the bug or data, then re-drive DLQ entries through the same handler so the dedupe and idempotency logic applies; mark entries replayed, succeeded, or permanently discarded.

Output Contract

After applying this skill, an agent should be able to produce:

• An endpoint handler that (a) verifies the HMAC signature on raw body bytes with constant-time compare before any other work, (b) returns 2xx only after durable acceptance, (c) dedupes on event ID with a unique constraint + TTL, and (d) routes unprocessable events to a DLQ with alerting.
• A short replay runbook: where DLQ entries live, what metadata they carry, and the command or procedure to re-drive them through the handler.

Operating Rules

• Signature over payload. Never trust a field inside the payload (an authenticated: true flag, a matching account ID) as proof the event came from the provider. The HMAC over the raw body with the shared secret is the only authenticity check.
• Constant-time compare, raw bytes. Always use hmac.compare_digest (or your language's equivalent) for signature comparison — a plain == leaks timing information. Always hash the exact raw body bytes you received; parsing and re-serializing JSON before hashing is a classic silent breakage.
• Freshness check blunts replays. Prefer verifying a timestamp the provider includes (sent in the header or as part of the signed payload) and rejecting events older than a short window (a few minutes is a common default; match what your provider supports). This shrinks the window where a stolen signature can be replayed.
• Secrets rotate, code doesn't. Store the webhook secret in your secret manager / env (see env-management), never in code. When rotating, support the old and new secret briefly (try new, fall back to old) so the switchover doesn't drop events.
• 5xx retries, 4xx stops. Return 5xx only for transient failures you expect a retry to fix; return 4xx for validation/auth failures so the provider does not burn retries on an event that will never succeed. Log both with the event ID.
• Prefer not to accept what you can't verify. If the provider's signature scheme is undocumented or you can't get the shared secret, do not expose the endpoint publicly — use a polling fallback or an allowlisted sender instead of an unverified webhook.