Webhook Reliability: Idempotency & Retry Reference

Webhook reliability comes down to a single uncomfortable fact: every major provider delivers events at least once, never exactly once, which means your consumer will eventually receive the same event twice and must handle it without double-charging, double-shipping, or double-emailing. Idempotency is not a nice-to-have — it is the load-bearing wall of any production webhook integration.

The cost of getting this wrong is concrete. A payment event processed twice becomes a duplicate charge. An order-created event processed twice becomes a duplicate fulfillment. And the most common cause is not a network glitch you can see — it is your own handler finishing the work but answering the provider a few milliseconds too late, so the provider assumes failure and retries an operation that already succeeded.

This reference assembles the four reliability layers — deduplication, retry with backoff, dead-letter queues, and signature verification — into decision tables you can build against. We compare Stripe, Shopify, Svix, and Amazon SQS side by side, then give a status-code decision tree, recommended DLQ thresholds, and the accept-then-queue architecture that absorbs all of it.

Before designing anything, internalize the constraint: no webhook provider can guarantee exactly-once delivery, because exactly-once delivery is a proven impossibility in distributed systems, not a feature waiting to be built. The result traces back to the Two Generals Problem and the FLP impossibility theorem published in 1985, which showed that consensus cannot be guaranteed in an asynchronous network where even one participant may fail.

The practical consequence is a clean distinction worth carrying into every design review: exactly-once is achievable as a processing guarantee, never as a delivery guarantee. The wire will sometimes deliver a duplicate. What you control is whether processing that duplicate produces a second side effect. That is the entire job of idempotency, and it sits on the consumer side — the receiver — not the sender.

Both major-vendor documentation and the queueing layer agree explicitly. Stripe's webhook docs warn that an endpoint "might occasionally receive the same event more than once." Amazon SQS Standard queues describe the same behavior at the infrastructure level: because messages are stored redundantly across multiple servers, a copy can reappear if the server holding a given message is unavailable during deletion. AWS's guidance is blunt — design applications to be idempotent so that processing the same message more than once does no harm.

The single most useful artifact for a multi-vendor webhook integration is a matrix that puts retry policy, signature scheme, and deduplication semantics side by side. No two providers agree on header names, signed-payload composition, or retry windows — so the integration cost is real, and it compounds per provider. Truto, a multi-SaaS integration platform, reports in-house integration maintenance running upward of $50,000 per integration as the hidden driver behind these inconsistencies. Treat that figure as vendor-stated context rather than an independent benchmark, but the direction is right: heterogeneity is expensive.

The mechanism is simple and the discipline is everything: every well-designed webhook carries a unique event ID that stays the same across retries. That identifier is precisely what distinguishes a retry from a genuinely new event. On receipt, you record the ID; if you have seen it before within your dedup window, you acknowledge and do nothing. The provider gets its 200, your business logic runs exactly once.

Two storage strategies cover almost every case. A database unique constraint on the event ID suits transactional operations — the insert either succeeds (first time) or violates the constraint (duplicate), and you branch on that atomically. A Redis key with a TTL suits high-throughput streams where a database round-trip per event is too costly. The choice is about throughput and consistency needs, not correctness — both work.

Sender-side idempotency keys are the mirror image and worth understanding even as a consumer. Stripe's API accepts an Idempotency-Key header (max 255 characters, a V4 UUID or other high-entropy string, never containing sensitive data such as an email). Stripe saves the resulting status code and body of the first request for that key — including a 500 — and replays it for repeat requests, with a 24-hour TTL after which the key becomes purgeable. The idempotency layer also compares incoming parameters to the original request and errors if they differ, preventing accidental key reuse across different operations.

When a delivery fails, the sender retries on an increasing delay — exponential backoff — so a struggling consumer is not hammered. But naive exponential backoff has a failure mode of its own: if a thousand events fail at the same instant, they all retry at the same computed delay, producing a synchronized thundering herd that knocks the recovering endpoint over again. Jitter randomizes those delays so the load spreads out.

On the receiving end of a managed gateway, the configurable budget is generous. Production webhook systems typically cap individual retry intervals between 6 and 12 hours with a total window of 1 to 3 days, and disable an endpoint after roughly 3 to 5 days of sustained failure. As a point of reference for what is achievable, Hookdeck's gateway documents up to 50 delivery attempts over as long as a week — that is a product maximum, not an industry norm, and it is far more aggressive than what Stripe or Shopify do natively.

Most guides collapse this into "2xx good, 4xx don't retry, 5xx retry." That heuristic is wrong in two important places: a 408 Request Timeout and a 429 Too Many Requests are both 4xx codes that should be retried. The table below is the version that survives production. Note that most providers, including Stripe, treat 3xx redirects as non-retriable failures — point your webhook at the final URL, never a redirect.

For outbound senders, pair this tree with a per-endpoint circuit breaker: open the breaker when, say, 5 of the last 10 requests fail, hold it open for a cooldown of roughly 30 to 120 seconds, then half-open to test recovery before resuming full traffic. Per-endpoint scope matters in multi-tenant systems — one customer's broken endpoint should never throttle deliveries to everyone else.

A dead-letter queue is where events go after exhausting their retries, so a single un-processable "poison" event does not wedge the main pipeline forever. In Amazon SQS the mechanism is a redrive policy with a maxReceiveCount: once a consumer has received a message that many times without deleting it, SQS moves the message to the DLQ. AWS guidance is to set maxReceiveCount high enough to permit genuine retries — at least 3 for standard queues — so transient errors are not misclassified as poison.

Two SQS subtleties bite teams in production. First, DLQ message expiration is computed from the original enqueue timestamp, not DLQ arrival — a message that spent a day in the source queue before failing has only the remaining retention left once it lands in the DLQ, so always set DLQ retention longer than source retention. Second, attaching a DLQ to a FIFO queue breaks strict ordering for the affected messages; decide whether ordering or poison-isolation matters more for that stream.

The DLQ is not a graveyard — it is a replay buffer. Pair it with a tool that lets an operator inspect a dead-lettered event, fix the underlying handler bug, and redrive the event back through processing. Because every event is keyed on its idempotent ID, replaying a DLQ event that was actually processed before failing is safe: the dedup layer absorbs the duplicate.

Signature verification proves an event genuinely came from the provider and was not forged or tampered with. The algorithm is nearly universal — HMAC-SHA256 keyed on a shared secret — but the surrounding conventions are a babel. Stripe signs {timestamp}.{raw_body} and delivers it in Stripe-Signature as t=<ts>,v1=<sig>. Shopify signs the raw body, base64-encodes the digest, and sends it in X-Shopify-Hmac-SHA256 — no timestamp prefix. GitHub uses X-Hub-Signature-256; Slack uses X-Slack-Signature. There is no shared standard to code against, which is exactly why a per-provider matrix earns its keep.

Two rules are non-negotiable. First, always verify against the raw request body bytes, before any JSON parsing or framework deserialization re-serializes and changes the bytes — a single whitespace difference breaks the HMAC. Second, compare the computed and received signatures with a constant-time comparison function — hmac.compare_digest in Python, crypto.timingSafeEqual in Node.js. A naive == comparison returns faster on an earlier-mismatching byte, leaking timing information an attacker can use to guess the signature byte by byte.

The single most dangerous failure mode in webhook handling is the timeout-induced duplicate. Your handler receives the event, does the real work — charges the card, ships the order — and then takes a beat too long to respond. The provider's timeout fires (often 5 to 15 seconds; Shopify is stricter still at a 1-second connection timeout and a 5-second full-request timeout), it concludes delivery failed, and it retries an operation that already completed. The duplicate is born not from a network problem but from your own slowness.

The fix is counter-intuitive for anyone trained on synchronous HTTP: do not do the work in the request. Verify the signature, deduplicate on the event ID, persist the raw event to a durable queue, and return 200 or 202 immediately. Then process asynchronously from the queue. This decoupling is what lets a system absorb thousands of events per second; a well-built ingestion gateway can keep added latency under a few seconds for the overwhelming majority of events while doing all of this.

Once you adopt accept-then-queue, the other layers slot in cleanly. Signature verification happens at the edge before anything is enqueued. Idempotency is checked against the dedup store before the work runs. Failed processing increments a retry count and eventually lands in the DLQ. Each concern lives in one place. This is the same reliability layer that makes any event-driven integration production-safe — the kind of architecture we build into custom web and application development engagements and reuse when wiring real-time triggers in CRM automation workflows. If you are evaluating a build-versus-buy decision for the ingestion layer itself, our total-cost-of-ownership analysis for workflow automation walks through where webhook reliability lands on the ledger.

Stripe states plainly that it does not guarantee events arrive in the order they were generated. A customer.subscription.updated can land before the createdthat logically preceded it. Building state machines that assume ordered arrival is a quiet recipe for corrupted state. The robust pattern is to treat each webhook as a trigger, not a source of truth: on receipt, fetch the object's current state from the provider's API and act on that, rather than trusting the event payload to reflect the latest reality. The same reliability layer underpins event-driven tools we have written about elsewhere — the inbound deliveries in our Slack event-subscriptions tutorial and the dispatch path in our MCP server TypeScript walkthrough both depend on exactly these idempotency and retry guarantees.

Looking ahead, the standards layer is slowly converging even as provider conventions stay fragmented. The CNCF's CloudEvents specification — version 1.0.2, released in 2022 and graduated as a CNCF project on January 25, 2024 — defines a common event envelope with fields such as id, source, type, time, and datacontenttype, and is adopted across Amazon EventBridge, Azure Event Grid, Google Cloud Eventarc, and Knative. That id field is, conveniently, the same stable identifier your dedup layer wants. As CloudEvents adoption widens, the per-provider header babel should narrow — though pragmatically, expect to keep a provider matrix for years yet.