keepcalculator

06 Numbers and Powers

Monitoring Small Go Services: What to Choose and Why

Card No. 06·71656 · Updated · Standard formula, shown below

For small Go services in production, the best monitoring choice is a lightweight metrics library like Prometheus client_golang combined with Grafana dashboards, because it gives you the most value for the least operational overhead. You push metrics from your service, store them in Prometheus, and visualize them in Grafana; this stack is the de facto standard in the Go ecosystem and works well even on a single small server. Below I’ll explain the reasoning, the alternatives, and the practical details you need to make the right call for your specific situation.

Why Go Services Have Specific Monitoring Needs

Go is a compiled, statically typed language that produces a single binary with no external runtime dependencies. This makes deployment simple, but it also means that the usual operating-system-level metrics (CPU, memory, disk) are not enough to understand what your service is doing. You need application-level metrics: request counts, latencies, error rates, in-flight requests, and internal state like queue depths or cache hit ratios. Go’s standard library provides net/http/pprof for profiling and runtime/metrics for runtime statistics, but these are not designed for long-term storage or alerting. For production monitoring, you need a system that collects, stores, and queries these metrics over time.

The Core Options: Pull vs. Push

Monitoring systems generally fall into two camps based on how they collect data:

  • Pull-based (Prometheus, Grafana Mimir, Thanos): The monitoring server scrapes an HTTP endpoint on your service. Your Go service exposes a /metrics endpoint that returns the current values in a text format. The server periodically fetches that endpoint.
  • Push-based (InfluxDB, Graphite, StatsD, Telegraf): Your service sends metrics to a central collector over the network. The collector then stores them in a time-series database.

Pull-based has a major advantage for Go services: it’s the default. The Prometheus client library for Go is maintained by the same community that builds Prometheus, and it integrates seamlessly with Go’s concurrency primitives. You can register metrics as package-level variables and update them from any goroutine without worrying about locks. Push-based systems often require you to manage batching, retries, and network failures, which adds complexity to your service code.

Prometheus + Grafana: The Default Choice

For most small Go services, the combination of Prometheus and Grafana is the best starting point. Here’s why:

  • Minimal code changes: You add the client library, expose a handler, and you’re done. The library handles concurrency and formatting.
  • Rich ecosystem: Prometheus has a wide range of exporters for databases, caches, and other infrastructure. If you later add a PostgreSQL database, you can scrape its exporter with the same Prometheus instance.
  • Powerful query language (PromQL): You can compute rates, percentiles, and aggregations on the fly. For example, histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) gives you the 95th percentile latency over the last 5 minutes.
  • Grafana dashboards: Grafana connects to Prometheus and provides beautiful, shareable dashboards. You can start with a community dashboard template and customize it.

Operationally, you can run Prometheus as a single binary on a small VM. It doesn’t need a cluster. For a handful of services, a single Prometheus instance with local storage is perfectly fine. You can even run it in Docker on the same machine as your service.

Instrumenting Your Go Service

The Prometheus client library offers four main metric types:

  • Counter: A monotonically increasing value (e.g., total requests).
  • Gauge: A value that can go up and down (e.g., current number of goroutines).
  • Histogram: Samples observations (like request durations) and counts them in configurable buckets. This allows you to calculate percentiles.
  • Summary: Similar to histogram but calculates quantiles on the client side. Prefer histograms because they allow aggregation across instances.

Here’s a minimal example of instrumenting an HTTP handler:

var (
    httpRequests = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total number of HTTP requests.",
        },
        []string{"method", "path", "status"},
    )
    httpDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name: "http_request_duration_seconds",
            Help: "Duration of HTTP requests.",
            Buckets: prometheus.DefBuckets,
        },
        []string{"method", "path"},
    )
)

func init() {
    prometheus.MustRegister(httpRequests, httpDuration)
}

func instrument(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        next.ServeHTTP(w, r)
        httpRequests.WithLabelValues(r.Method, r.URL.Path, strconv.Itoa(w.StatusCode())).Inc()
        httpDuration.WithLabelValues(r.Method, r.URL.Path).Observe(time.Since(start).Seconds())
    })
}

Then expose the metrics endpoint:

http.Handle("/metrics", promhttp.Handler())

That’s it. Your service now exposes metrics that Prometheus can scrape.

When to Consider Alternatives

Prometheus+Grafana isn’t always the right answer. Here are some scenarios where you might choose something else:

  • You already use a cloud provider’s monitoring: AWS CloudWatch, Google Cloud Monitoring, and Azure Monitor all have Go SDKs. If your service runs on a single cloud and you don’t want to run your own server, using the cloud-native monitoring can reduce operational overhead. The downside is vendor lock-in and often higher cost at scale.
  • You need tracing in addition to metrics: If your service is part of a distributed system and you need to trace requests across services, you might consider OpenTelemetry. OpenTelemetry can export metrics, logs, and traces. You can still use Prometheus as the backend for metrics, but you’d add a tracing backend like Jaeger or Tempo.
  • You have a very short-lived batch job: For a cron job that runs for a few seconds, scraping won’t work because the process exits before Prometheus can scrape it. In that case, you can push metrics to Prometheus using the Pushgateway, or use a push-based system like InfluxDB.
  • You need long-term storage: Prometheus’s local storage is not designed for years of data. If you need to keep metrics for more than a year, you can use Thanos or Mimir, but that’s probably overkill for a small service.

Common Pitfalls and How to Avoid Them

Even with a good stack, there are several mistakes that can undermine your monitoring:

  • Not defining SLOs: Metrics without targets are just noise. Decide on service level objectives (e.g., 99% of requests under 300ms) and set up alerts that fire when you’re about to breach them.
  • Using too many labels: Prometheus performance degrades with high cardinality. If you put the request path as a label, a path like /users/12345 will create a new time series for every user ID. Use a limited set of labels, and if you need high-cardinality data, use logs instead.
  • Not alerting on symptoms: Alert on user-facing symptoms (error rate, latency) rather than causes (CPU usage). Causes can be many, but symptoms are what matter.
  • Forgetting about logs: Metrics tell you that something is wrong, but logs tell you why. Use structured logging (e.g., with log/slog) and aggregate logs in a system like Loki or ELK. Metrics and logs are complementary.

Cost and Operational Complexity

For a small service, the cost of monitoring is often more about time than money. Running Prometheus and Grafana on a single small VM costs a few dollars per month in cloud fees, but the main cost is the time to set up and maintain them. However, that time is usually less than the time you’d spend debugging without metrics. If you’re running on Kubernetes, you can use the Prometheus Operator to automate deployment, but that adds complexity. For a single binary, a simple systemd service running Prometheus is enough.

If you want to minimize cost further, you can use a managed Prometheus service (e.g., Grafana Cloud’s free tier) that offers a limited amount of metrics for free. That eliminates the need to run your own server.

Frequently Asked Questions

Is Prometheus too heavy for a small service?

No. Prometheus itself is a single Go binary that uses modest resources (typically less than 100MB RAM for small workloads). The client library adds only a few kilobytes to your binary and a tiny overhead per request. For a small service, the resource usage is negligible.

Should I use OpenTelemetry instead of Prometheus client?

OpenTelemetry is a good choice if you need traces and logs as well as metrics, or if you want to avoid vendor lock-in. However, it adds more complexity. For a small service that only needs metrics, the Prometheus client is simpler and more direct. You can always migrate to OpenTelemetry later.

How do I monitor a service that runs as a cron job?

For short-lived jobs, use the Prometheus Pushgateway. Your job pushes metrics to the Pushgateway before exiting, and Prometheus scrapes the Pushgateway. Be careful to clean up old metrics to avoid stale data.

What metrics should I always track?

At minimum: request rate (count), error rate (count with status code), latency histogram (duration), and in-flight requests (gauge). Also track runtime metrics like goroutine count and memory usage, which you can get from the Go runtime collector.

Can I use Grafana without Prometheus?

Yes, Grafana supports many data sources, including InfluxDB, CloudWatch, and even SQL databases. But Prometheus is the most common and best-supported for metrics.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →