keepcalculator

06 Numbers and Powers

Alertmanager enrichment sidecar vs webhook service

Card No. 06·81904 · Updated · Standard formula, shown below

When you need to add context to Prometheus Alertmanager alerts before they reach a chat or ticketing system, the two common architectures are an enrichment sidecar that runs alongside Alertmanager and a separate webhook service that Alertmanager calls. The sidecar fits when you want a single, tightly coupled process that intercepts alerts in place, while a webhook service fits when you want a reusable, independently deployable endpoint that multiple Alertmanagers or other systems can call. This article explains how each works, what they cost in operational terms, and how to choose based on your team’s scale and deployment style.

Background: why Alertmanager needs enrichment

Alertmanager receives alerts from Prometheus and handles grouping, inhibition, and routing to receivers like email, Slack, PagerDuty, or a generic webhook. Out of the box, an alert contains labels and annotations from the Prometheus rule, but it often lacks context that operators need: the current deployment version, a link to a runbook, the on-call engineer’s name, or a summary of recent changes. Enrichment adds that data before the alert is delivered.

The two main patterns emerged because Alertmanager’s built-in templating can only use data already present in the alert or in static files. If you need to query an external API (like a deployment tracker or a CMDB) or compute something from a time series, you need a process that sits between Alertmanager and the receiver.

How the sidecar pattern works

A sidecar is a separate container or process that runs on the same host or pod as Alertmanager. It typically listens on a local port and is configured as Alertmanager’s webhook receiver. When Alertmanager sends a notification, the sidecar receives it, enriches the payload, and then forwards it to the actual destination (Slack, email, etc.).

Concretely, you might run Alertmanager and an enrichment sidecar in the same Kubernetes pod. Alertmanager’s route sends all alerts to http://localhost:8080/enrich. The sidecar parses the alert, queries an internal API for deployment metadata, adds a field like version: 2.4.1 to each alert’s annotations, and then posts the enriched payload to your Slack webhook. Because the sidecar is local, there is no network hop between Alertmanager and the enricher, and the sidecar can share the pod’s service account and secrets.

How the webhook service pattern works

A webhook service is a standalone HTTP endpoint, often deployed as its own microservice or serverless function. Alertmanager is configured to send alerts to this service’s URL (e.g., https://alerts.example.com/enrich). The service can be written in any language, scale independently, and be reused by multiple Alertmanagers or even other monitoring systems that emit webhooks.

For example, a company running several Prometheus clusters in different regions might point each Alertmanager to the same enrichment webhook. The webhook service can add region-specific context by looking at the alert’s region label, query a central asset database, and then route the enriched alert to a team-specific Slack channel. Because the service is separate, it can be load-balanced and updated without restarting Alertmanager.

Choosing between sidecar and webhook

The decision hinges on deployment topology, operational ownership, and failure tolerance. Consider these factors:

  • Number of Alertmanagers: If you have one or two, a sidecar per instance is simple. If you have many, a shared webhook service avoids duplicating enrichment logic.
  • Team boundaries: If the team that runs Alertmanager also writes the enrichment logic, a sidecar keeps everything in one deployable. If a separate team owns alert routing, a webhook service gives them a stable API.
  • Availability requirements: A sidecar fails with its Alertmanager (if the pod dies, both go down). A webhook service can be scaled and made highly available independently, but it introduces a network dependency.
  • Latency and data locality: A sidecar can access local files or Unix sockets that a remote service cannot. A webhook service may need to be in the same network to reach internal APIs.
  • Upgrade cadence: A sidecar is upgraded together with Alertmanager (often in the same Helm chart). A webhook service can be versioned and deployed on its own schedule.

In practice, small teams with a single cluster often prefer the sidecar because it is one less thing to operate. Larger organizations with multiple clusters and a central SRE team tend to build a webhook service that becomes a shared internal tool.

Common problems and pitfalls

Both patterns have known failure modes. With a sidecar, the most common issue is silent failure: if the sidecar crashes, Alertmanager keeps sending notifications to a dead endpoint, and alerts are lost. You need a health check and a fallback (for example, Alertmanager’s built-in retry, but that only helps if the sidecar is temporarily down, not permanently).

With a webhook service, the main issues are latency and timeouts. Alertmanager has a default webhook timeout of 10 seconds (configurable via http_config.timeout). If your enrichment service queries a slow API, you may exceed that timeout and Alertmanager will mark the notification as failed and retry, potentially causing duplicate alerts. You should design the service to respond quickly, even if it means returning the original alert with a note that enrichment failed.

Another common pitfall is authentication. Alertmanager can send a bearer token in the webhook header, but the sidecar pattern often relies on localhost security, which is fine on a single host but not if you run Alertmanager in a shared cluster where other pods can reach the sidecar’s port. Always bind the sidecar to 127.0.0.1 or use network policies.

Comparison: sidecar vs webhook service

Here is a side-by-side view of the typical characteristics:

  • Deployment unit: Sidecar runs as a container in the same pod or host; webhook runs as a separate service or function.
  • Discovery: Sidecar is localhost, no service discovery needed; webhook needs a stable URL, DNS, and load balancer.
  • Scalability: Sidecar scales with Alertmanager (one per instance); webhook scales independently, often horizontally.
  • Failure isolation: Sidecar failure takes down alert delivery; webhook failure can be mitigated by Alertmanager retries and multiple replicas.
  • Reusability: Sidecar is tied to its Alertmanager; webhook can serve many consumers.
  • Operational overhead: Sidecar adds a process to an existing deployment; webhook adds a new service with its own CI/CD, monitoring, and on-call.
  • Latency: Sidecar has near-zero network overhead; webhook adds a network hop, which can be significant if the service is in another region.

There is no universal winner. The sidecar is simpler for a single cluster; the webhook is better for multi-cluster or multi-team environments.

Operational costs and safety considerations

Running a sidecar means you now have two processes to monitor instead of one. You need to ensure the sidecar has the same configuration lifecycle as Alertmanager (e.g., secrets, environment variables). If you use Kubernetes, you must update the sidecar image in the same pod template, which can be convenient or annoying depending on how you manage releases.

A webhook service requires you to operate an always-on endpoint. That means setting up TLS, authentication, rate limiting, and monitoring for the service itself. You also need to think about capacity: if Alertmanager sends a burst of notifications (e.g., after a large incident), the webhook service must handle the spike or queue them. A sidecar naturally inherits Alertmanager’s concurrency limits, so it is less likely to be overwhelmed.

For safety, always ensure that enrichment never blocks alert delivery. If the enrichment source is down, the sidecar or webhook should still forward the original alert, possibly with a flag like enrichment_failed: true. This prevents a dependency from hiding critical alerts.

Alternatives and related patterns

Before building either pattern, consider whether you actually need enrichment. Alertmanager templates can do simple lookups from static files (e.g., a map of team to Slack channel) without any extra service. For dynamic data, you might also use a receiver like webhook that points to a chat bot, and let the bot enrich the message. Some teams use a message queue (e.g., Kafka) between Alertmanager and a consumer that enriches and then posts to multiple destinations. That is more robust but adds significant infrastructure.

Another alternative is to enrich at the Prometheus rule level: have the rule query an external source and add an annotation at evaluation time. This only works for data that is available when the rule evaluates, and it can slow down rule evaluation if the external query is slow. It is rarely a good idea for per-alert dynamic data.

Frequently Asked Questions

Can I run both a sidecar and a webhook service?

Yes, you can. For example, a sidecar could do local enrichment (adding a timestamp or hostname) and then forward to a central webhook service that adds global context (like the current incident commander). This layered approach is common in large organizations, but it adds complexity and more failure points, so only do it if each layer has a clear purpose.

How do I test enrichment without sending real alerts?

Use Alertmanager’s amtool to send a test alert to your webhook receiver. For a sidecar, you can send a curl request directly to the sidecar’s port with a sample Alertmanager webhook payload (the JSON structure is documented in the Alertmanager webhook receiver docs). This lets you verify the enrichment logic in isolation.

Does Alertmanager retry if my webhook service returns an error?

Yes, Alertmanager retries failed notifications, but the retry behavior depends on the receiver type. For webhook receivers, Alertmanager will retry with an exponential backoff, but it does not guarantee exactly-once delivery. If your enrichment service is slow, you may get duplicates after a retry. Design your service to be idempotent where possible.

What is the best language for writing an enrichment service?

Any language that can handle HTTP and JSON. Go is a natural choice because Alertmanager itself is written in Go, and you can reuse the same libraries for parsing alerts. Python is also common for quick scripting. The language matters less than the deployment model and how you handle timeouts and errors.

How do I secure the sidecar port?

Bind the sidecar to 127.0.0.1 or to the pod’s loopback interface. In Kubernetes, you can also use a NetworkPolicy to restrict ingress to only the Alertmanager pod. If you must expose it on a network, require a shared secret in the header and use TLS.

Can I use a serverless function as a webhook service?

Yes, a serverless function (like AWS Lambda or Google Cloud Functions) can work as a webhook receiver. The main caveat is cold start latency, which can exceed Alertmanager’s 10-second timeout. Use a warm-up mechanism or set the timeout high enough, but be aware that serverless platforms often have a maximum execution time that may be too short for complex enrichment.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →