Webhooks: How They Work and 6 Failure Points

This article explains what a webhook is, how the underlying mechanism works, and why production systems that rely on them often fail. You will learn the HTTP-based flow, the difference between webhooks and polling, and the six most common ways webhook integrations break in real-world deployments.

Definition and Core Concept

A webhook is a user-defined HTTP callback: one system sends a simple HTTP request (usually POST) to a specific URL on another system when a particular event occurs. The receiving system then processes the payload (the body of that request) and takes an action. The term is standard in software engineering, appearing in the documentation of GitHub, Stripe, Slack, and most major API providers. It is not slang; it is a precise technical term that has been in widespread use since around 2007, when Jeff Lindsay proposed the concept and the word “webhook” as a portmanteau of “web” and “hook” (a callback mechanism in programming).

The crucial contrast is with polling. In polling, a client repeatedly asks a server “is there anything new?” at fixed intervals. With a webhook, the server itself pushes the notification the moment the event happens. This makes webhooks more efficient and lower-latency, but it also inverts the control flow: the receiving system must be publicly reachable, and it must accept unsolicited inbound requests.

How a Webhook Works

The lifecycle of a webhook has three distinct phases: registration, delivery, and processing.

1. Registration: The consuming application (the receiver) gives the producer (the event source) a URL. This is usually done in a dashboard or via an API call. The producer may also ask for a secret token to sign the payloads.
2. Delivery: When a triggering event occurs (a payment succeeds, a repository is pushed to, a form is submitted), the producer constructs a JSON or XML payload describing the event and sends an HTTP POST request to the registered URL. The request usually includes headers that identify the event type and a signature for verification.
3. Processing: The receiver’s server must respond with a 2xx HTTP status code (typically 200 OK) to acknowledge receipt. The receiver then parses the payload and executes its business logic, such as updating a database or sending a notification.

If the receiver does not return a 2xx response quickly, the producer will usually retry the delivery with an exponential backoff strategy (e.g., after 1 minute, then 5, then 30, then hours). This retry mechanism is a core part of the webhook contract, and it is the source of many production incidents.

Why Webhooks Break in Production

Webhooks are simple in principle but notoriously fragile in practice. The six most common failure modes in production are:

• No Signature Verification: If the receiver does not validate the HMAC signature (a cryptographic hash of the payload using a shared secret), any attacker who discovers the URL can send fake events. This leads to data corruption, unauthorized actions, and security incidents. The fix is to always verify the signature header before processing the payload.
• Timeout and Slow Processing: The producer expects a quick 2xx response, usually within 5-10 seconds. If the receiver’s handler performs a slow database query or calls an external API synchronously, the request times out. The producer then retries, and the receiver processes the same event multiple times, causing duplicate records. The correct pattern is to acknowledge the webhook immediately and process the payload asynchronously in a background job queue.
• Duplicate Delivery: Producers guarantee “at least once” delivery, not “exactly once.” Network glitches, timeouts, and retries mean the same event can arrive multiple times. Without idempotency (making the operation safe to run repeatedly), the receiver will create duplicate orders, duplicate emails, or double-charge customers. The receiver must store a unique event ID and reject or ignore events it has already seen.
• Payload Schema Drift: The producer adds a new field, renames an existing one, or changes a data type. The receiver’s parser, which was written against the old schema, either crashes or silently drops the new data. This is especially common when the producer and consumer are managed by different teams with different release cycles. The mitigation is to treat the payload as a versioned contract and to write defensive parsers that ignore unknown fields.
• Network and DNS Failures: The receiver’s endpoint goes down, its TLS certificate expires, or its DNS record changes. The producer’s retries pile up, and events are lost if the retry queue expires. Monitoring alerting on the delivery failure rate is essential, as is having a dead-letter queue (a storage area for events that could not be delivered after all retries).
• Ordering and Concurrency Issues: Most producers do not guarantee that webhooks arrive in the exact order the events occurred. If event B depends on event A having been processed first, and B arrives before A, the receiver’s state becomes inconsistent. Concurrent processing of multiple webhooks for the same resource makes this worse. The receiver must either buffer and reorder events or design its logic to be order-independent.

Security Considerations

The most critical security practice is signature verification. The producer computes an HMAC-SHA256 hash of the raw request body using a secret key that is shared only with the receiver. The receiver recomputes the hash from the body it received and compares it to the value in the X-Signature header. If they do not match, the request must be rejected. This prevents attackers from forging events.

A second security concern is the content of the payload itself. A webhook payload is untrusted input, just like a form submission. The receiver must sanitize any data from the payload before using it in SQL queries, HTML output, or shell commands to prevent injection attacks. Never assume the payload is clean because it came from a trusted producer.

Finally, the receiver should use HTTPS for the endpoint, and the producer should enforce it. Sending webhooks over plain HTTP exposes the payload and the signature to network sniffing, allowing replay attacks.

Alternatives to Webhooks

Webhooks are not always the right tool. The main alternatives are:

• Polling: The client asks the server for changes at a regular interval. It is simpler to implement and does not require a publicly reachable endpoint, but it wastes resources and adds latency. It is appropriate when events are rare or when the client is behind a firewall.
• Server-Sent Events (SSE): A one-way channel over a single long-lived HTTP connection where the server pushes events to the client. It is simpler than WebSockets and works over ordinary HTTP, but it only supports server-to-client communication, not bidirectional.
• WebSockets: A full-duplex, persistent connection. It allows real-time bidirectional communication but requires a more complex server setup and does not use the standard request/response model of HTTP.
• Message Queues (e.g., RabbitMQ, Kafka): The producer publishes events to a queue, and consumers subscribe. This decouples the systems and provides strong delivery guarantees, but it requires the consumer to maintain a persistent connection to the queue infrastructure, which is not always possible for external SaaS integrations.

For internal microservice communication, a message queue is often a better fit than webhooks. For external integrations with third-party SaaS providers, webhooks are usually the only option offered.

How to Choose a Webhook Provider

When evaluating a service that sends webhooks, look for these features:

• Signing: Does the provider sign every request with a secret? Is the algorithm documented?
• Retry Policy: How many times will it retry? What is the backoff schedule? Can you configure it?
• Delivery Logs: Can you see a history of all deliveries, including the request and response bodies, in the provider’s dashboard?
• Replay Capability: Can you manually re-send a past event from the dashboard or API?
• Dead-Letter Queue: Does the provider store events that failed all retries, and can you retrieve them later?

These features are not optional extras; they are the difference between a debuggable integration and a black box. A provider that offers none of them is likely to cause significant pain in production.

Frequently Asked Questions

What is the difference between an API and a webhook? An API is a general interface for requesting data or actions from a server; the client initiates the request. A webhook is a specific type of API call that the server initiates to notify the client about an event. All webhooks are API calls, but not all API calls are webhooks.

Can webhooks be sent over HTTPS? Yes, and they should be. HTTPS encrypts the payload in transit and prevents tampering. Most major providers require HTTPS for the receiver URL and will refuse to send to plain HTTP endpoints.

How do I test a webhook locally? Use a tool like ngrok or a similar tunnel that exposes your local machine to the internet with a temporary public URL. Register that URL as the webhook endpoint, trigger a test event in the producer, and watch the request arrive in your local server logs.

What happens if my server is down when the webhook is sent? The producer will retry according to its retry policy, typically with exponential backoff over several hours or days. If all retries fail, the event is usually moved to a dead-letter queue or dropped. You should monitor the delivery failure rate and have a process to recover events from the dead-letter queue.

Is a webhook a push or a pull technology? It is a push technology. The producer pushes the data to the receiver’s URL without the receiver asking for it. This is the opposite of polling, which is a pull technology where the receiver repeatedly asks for updates.

Can I use a webhook to send data between two of my own servers? Yes, but it is often better to use a message queue for internal communication. Webhooks are designed for external integrations where you do not control the other system. A message queue gives you better delivery guarantees, ordering, and replay capabilities for internal traffic.

Related

• Sony Bravia Theatre Bar 7 Sub Pairing Checks
• Public and Private Keys & Hashing Explained
• Poco X7 Pro vs Motorola Edge 70 Pro Specs