Webhook Reliability: Idempotency & Retry Reference

Webhook reliability starts with two uncomfortable possibilities: the same event can arrive again, and an event can exhaust its delivery budget without reaching your application. A successful HTTP response also says nothing about whether the eventual business operation succeeded. The consumer needs a durable record of accepted work and an explicit rule for safely repeating that work.

A timeout after fulfillment creates an ambiguous result: the sender sees failure while the receiver may already have shipped an order. Acknowledging before persistence creates the opposite problem: the sender sees success while a process crash erases the only copy. Design and test both boundaries. Signature verification, duplicate handling, retries, and dead-letter queues each solve a different part of the problem; none supplies an end-to-end guarantee by itself.

This reference compares HTTPS webhook delivery with the Amazon SQS Standard queue layer, then connects them through a durable inbox pattern. Provider facts were checked on October 4, 2026. Configuration examples are illustrative engineering choices, not measured benchmarks or vendor defaults. The original publication date remains May 28, 2026.

Treat an acknowledgement as a statement about one boundary: this receiver has durably accepted responsibility for the event. It should not claim that a payment, inventory update, or email already happened. Keep delivery state, processing state, and business outcome separate in logs and operator tools. Otherwise an HTTP success graph can hide a growing backlog of unprocessed orders.

Stripe documents duplicate webhook events. Shopify documents duplicate deliveries and a finite retry budget. Svix marks delivery as failed after its retry schedule is exhausted. SQS Standard explicitly offers at-least-once queue delivery; that is a queue contract, not proof that every upstream webhook reaches the queue. The four systems belong at different points in the pipeline, so their guarantees should not be flattened into one slogan.

Idempotency and polling need an explicit effect boundary and recovery assumptions. An atomic transaction can suppress repeated changes inside its own database boundary. Once the operation crosses into another service, a crash can occur after that service acts but before your ledger records completion. Solve that ambiguity with a stable downstream operation key, an outcome lookup, or a compensating workflow.

The table records current documented behavior, with SQS shown as infrastructure after ingress. Stripe’s Receive Stripe events documentation, Shopify’s Verify webhook deliveries documentation, and Svix’s Retry Schedule and Manual Verification documentation were checked on October 4, 2026. Their automatic retry limits do not define a safe universal ledger expiry. Keep a separate policy for operator replay and recovery.

The Svix intervals add to 27 hours, 35 minutes, and 5 seconds before allowing for request durations. They are relative delays after each preceding failure, not timestamps measured from the first attempt. A 24-hour deduplication cache therefore expires too early even for this automatic schedule. Do not replace that mistake with another fixed TTL: inspect replay availability and the lifetime of the business operation.

Stripe permits manual resend through its Dashboard for up to 15 days and through its CLI for up to 30 days after event creation. Those controls can overlap automatic retries. The operational lesson is to retain or reconstruct processed-operation evidence for the replay horizon you actually support. A short-lived cache may accelerate lookups, but it should not be the only durable record of a financial effect.

An unsafe pattern is to mark an event as seen and then enqueue it in a separate operation. If the enqueue fails, a retry sees the marker and gets discarded; work is permanently lost. Reversing the operations can enqueue duplicates. Use an inbox table as the durable work queue, or commit an inbox row and an outbox row in the same transaction. A relay can publish the outbox later and retry after a crash.

Define a unique key that includes the provider, trusted account or tenant scope, and event ID. Resolve the tenant from the verified endpoint configuration and authenticated payload; do not trust an arbitrary tenant header. The persistence adapter must atomically return either inserted or already present, with an existing row always representing retained work or completed processing. A duplicate of pending work is safe to acknowledge because the worker can still discover it.

A Redis lock alone is not the same contract. Expiry, eviction, persistence settings, and a worker crash can reopen the operation or suppress unfinished work. For transactional state, keep the idempotency ledger and the local change in the same transaction. For an external effect, retain the stable operation key across retries and uncertain outcomes; do not generate a fresh payment or fulfillment key after a timeout.

Stripe’s Idempotent requests API reference describes a separate sender-side feature: keys can be up to 255 characters; the first saved result, including a 500 response, is reused. Keys become eligible for pruning after at least 24 hours. That policy concerns requests you send to Stripe and is not the retention contract for webhook events you receive.

A local worker retry policy and a provider’s delivery policy are separate loops. If both retry aggressively while a downstream dependency fails, traffic can multiply. Give the worker a bounded attempt budget and a deadline, record the next eligible attempt durably, and move unresolved work into a recoverable failure state when the budget ends. A retry must reuse the same business-operation key.

For a sender or worker you control, capped exponential backoff makes later attempts less frequent. Jitter spreads otherwise synchronized requests. The AWS Architecture Blog comparison of jitter strategies explains full jitter, equal jitter, and decorrelated jitter. Choose from workload evidence rather than presenting one delay range as an industry norm. A cap and total deadline still matter with decorrelated jitter.

Do not retry an ambiguous side effect merely because a network error looks transient. First determine whether the downstream service supports an idempotency key or an outcome query. If it supports neither, a blind retry may repeat the effect; suspend and investigate or reconcile. Retryability is a property of the operation and its failure, not just of an HTTP status class.

A receiver cannot stop a managed provider from retrying by choosing a status code that a generic guide calls permanent. Stripe treats redirects as failed deliveries; Svix treats every non-2xx response as failure unless its documented delivery-control mechanism applies. Shopify treats responses outside the 200 range as errors. Inspect the actual provider contract instead of assuming that all 4xx responses stop retries.

The following is an illustrative decision tree for an outbound sender you own. It is not a prediction of those providers’ retry schedules. A 429 response can contain Retry-After; if your sender supports it, parse seconds or an HTTP date, enforce bounds, and respect the endpoint’s rate limit. Keep retry budgets finite even when the response asks for a long wait.

Scope concurrency and circuit breakers per endpoint or tenant so one customer’s failure does not stall everyone. A breaker can pause attempts while leaving durable work pending. When it probes recovery, send bounded traffic and resume gradually. Choose failure thresholds and cooldowns from error rates and service objectives. Test recovery behavior with a bounded probe before allowing the full pending backlog to resume.

A DLQ isolates messages that cannot be processed within the configured retry budget. Amazon SQS uses a redrive policy with maxReceiveCount; its dead-letter queue guide advises allowing enough attempts for transient failures. It does not prescribe a universal minimum of three. Select the budget from processing duration, visibility timeout, failure modes, and your recovery deadline.

For Standard queues, retention expiration remains anchored to original enqueue time after movement to the DLQ. The DLQ age metric instead reflects time since that movement. FIFO resets the enqueue timestamp when moved. A DLQ can also break the ordering required by a FIFO workflow. Account for those different semantics before treating a metric or redrive as a correctness guarantee.

The SQS queue attribute reference limits MessageRetentionPeriod to 14 days and documents a 4-day default. A 30-day SQS DLQ retention setting is not available. If recovery requires longer storage, design a separate protected archive and replay process with its own data-retention policy. Retention is a ceiling on recoverability, not proof that an operator will act in time.

A replay tool should show validation failures, attempt history, tenant scope, effect outcome, and who authorized the recovery. Rate-limit redrive so a recovered dependency is not overwhelmed. A completed ledger can suppress an already finished effect; a pending or failed row must remain eligible for recovery. If expiry deletes that evidence, replay is no longer automatically safe. The background job queue patterns reference provides broader worker-system context.

Preserve the exact request bytes before JSON parsing. Stripe’s signature troubleshooting documentation explains how body mutation breaks verification; use its official library and the endpoint signing secret. A secret from a CLI listener differs from a Dashboard endpoint secret. Do not choose verification keys using untrusted request fields, and do not log secrets or complete sensitive payloads.

Shopify uses a base64 HMAC-SHA256 digest over the raw body. Svix signs the message ID, timestamp, and raw body together and can send multiple versioned signatures. These are different protocols even though they use related cryptography. Use the provider SDK rather than reusing a generic HMAC function across them. Signature verification authenticates the signed material; it does not automatically establish the trust of every unsigned header.

For a manual implementation, validate input encoding and signature length before constant-time comparison. Node’s timingSafeEqual documentation requires equal-length inputs and warns that surrounding code can still introduce timing leaks. A freshness check needs an authenticated timestamp and a synchronized clock. Stripe’s documented default tolerance is 5 minutes; do not disable tolerance to make an old captured request pass.

Shopify’s X-Shopify-Triggered-At can inform payload-age diagnostics, but its body HMAC does not authenticate that header. It is not a substitute for signed replay protection, and delayed legitimate delivery should not be rejected solely because an unsigned time looks old. Retained business-operation evidence protects against repeated effects even when a valid request can be replayed.

Keep the request path short: authenticate the raw body, validate the supported event envelope and trusted account context, commit an inbox row, then return the provider’s success response. Shopify documents a 1-second connection timeout and 5-second whole-request timeout. Use the actual endpoint budget when measuring ingress latency; there is no universal five-to-fifteen-second deadline or unmeasured throughput guarantee.

The example below is framework-independent JavaScript for an asynchronous notification endpoint. It assumes a verifier adapter that throws on invalid authentication and an inbox adapter with an atomic insert-if-absent contract. It deliberately has no HTTP server, database implementation, or payment call. The inbox is the durable queue: a worker must recover pending records. Adapter tests alone do not prove production durability or distributed concurrency.

// Illustrative adapters; no production server or database.
export async function accept(rawBody, headers, config, verify, inbox) {
  let event;
  try {
    event = await verify(rawBody, headers, config);
  } catch {
    return { status: 400 };
  }
  if (!event || typeof event.id !== "string" || !event.id.trim()) {
    return { status: 400 };
  }
  const key = JSON.stringify([config.provider, config.account, event.id]);
  try {
    const result = await inbox.insertIfAbsent(key, event);
    if (result !== "inserted" && result !== "present") {
      throw new Error("Unexpected persistence result");
    }
    return { status: 200 };
  } catch {
    return { status: 503 };
  }
}

Here, present means the original inbox record is still durably recoverable or completed. An implementation that stores a transient lock and calls it present violates the contract. If the commit succeeds but its acknowledgement is lost, returning failure is acceptable: the provider retry finds the same row. If persistence fails before commit, the endpoint returns failure and leaves the next delivery eligible.

The processing side needs a second explicit contract. In one local transaction, update application state and mark processing complete. If another system performs the effect, use an outbox plus a stable downstream idempotency key or an outcome query. A worker lease prevents competing execution but is not evidence that the effect completed. Heartbeats, lease expiry, and recovery must retain work without granting a new business identity.

Test concurrent duplicate acceptance, distinct account scopes, persistence failure, commit-with-lost-response, and recovery of pending work. Test a crash after a downstream effect but before completion is recorded. The latter cannot be fixed by the ingress function alone. These boundaries are central to custom web and application development and CRM automation integrations; define them before selecting a gateway.

Stripe does not guarantee event order. Treat an event as a notification to inspect the relevant state when that suits the workflow. A current-resource fetch is useful for converging a local projection, but cannot reconstruct every historical transition. Audit workflows may need the immutable event record as well as the current resource. Serialize or version updates per entity so two workers cannot overwrite newer state with older results.

A reconciliation job can compare provider records against your ledger using persisted checkpoints, overlapping retrieval windows, pagination, and operation keys. It reduces gaps only when the API exposes the necessary data, the credentials remain authorized, and the job actually finishes. Alert on checkpoint lag and failed pages. A scheduled poll by itself is not a guarantee that nothing is permanently lost.

The CloudEvents 1.0.2 specification defines an event envelope, not delivery or authentication guarantees. Its event identity is scoped by source plus id. Optional time metadata does not create a global sequence. Translating a webhook into that envelope can simplify downstream handling, but retain the original provider identity and provenance rather than assuming a normalized field solves deduplication.

For the FIFO distinction, see AWS’s exactly-once processing explanation. Keep the application ledger even when the queue deduplicates sends. A manual recovery outside the queue’s deduplication interval still needs an operation identity. Related REST API design patterns help specify these contracts at service boundaries.