Choosing the right sampling strategy directly reduces tracing overhead by limiting how many requests generate and transmit trace data, while still capturing enough information to diagnose problems. You control the trade-off between observability detail and performance cost by deciding what to sample, when to sample, and how much to sample. This article explains the main sampling approaches, how to pick one for your system, and the pitfalls that can undermine your efforts.
Formula
The calculator above converts a configured sampling percentage into the expected number of traces you will retain per 1,000 requests. The formula is simple:
Expected traces per 1000 = (sampling rate / 100) × 1000
Where:
- Sampling rate (%) – the percentage of requests you choose to trace, from 0 to 100.
- Expected traces per 1000 – the number of traces you can expect to collect for every 1,000 requests that pass through your system.
This gives you a quick way to estimate volume before you configure your tracing backend. For example, a 10% sample on a service handling 10,000 requests per minute yields 1,000 traces per minute, which may be far more than you need if you only look at traces when investigating an incident.
Worked Example
Suppose you run an API that receives 100,000 requests per hour. You set your sampling rate to 5%. Using the formula:
Expected traces per 1000 = (5 / 100) × 1000 = 50 traces per 1000 requests.
Over one hour, that is 100,000 / 1000 × 50 = 5,000 traces. If each trace averages 20 KB of exported data, you are sending about 100 MB per hour to your tracing backend. If that volume is acceptable, 5% might be fine. If not, you need a lower rate or a smarter strategy.
Why Sampling Exists
Tracing every request in a high-throughput system is often impractical. Each trace involves generating span IDs, recording timestamps, attaching attributes, and exporting the data to a collector. At thousands of requests per second, the CPU time, memory allocation, and network bandwidth add up. Sampling reduces that load by only capturing a subset. The key is to keep enough traces to answer the questions you actually ask: where is latency coming from, which services are erroring, and what does a typical request path look like?
How Sampling Works in Practice
Sampling happens at two levels: head-based and tail-based. Head-based sampling decides whether to trace a request when it starts, before any spans are created. It is simple and cheap, but it cannot know whether the request will be interesting later. Tail-based sampling waits until the trace is complete and then decides whether to keep it, which allows you to keep all traces that contain an error or a slow span, but it requires buffering all spans temporarily, which itself consumes memory.
Most tracing libraries support a fixed probability (e.g., 10% of all requests). More advanced setups use a rate limiter that allows a certain number of traces per second, or a priority sampler that always keeps traces from specific routes or for specific users. Some systems use a hybrid: head-based for most traffic, plus a tail-based sampler that reviews buffered traces for error conditions.
How to Choose a Sampling Strategy
There is no universal best strategy; the right one depends on your traffic volume, your storage budget, and what you need to observe. Consider these factors:
- Traffic volume – low-traffic services can often trace 100% without noticeable overhead. High-traffic services need a lower percentage.
- Error rate – if errors are rare, you want to capture all of them, not a random 1% sample. Tail-based sampling is ideal for this.
- Latency outliers – if you care about slow requests, a fixed sample might miss them. Consider sampling based on duration thresholds.
- Storage cost – every trace you keep costs storage and query time. Estimate your daily trace volume and multiply by the average size to see the cost.
- Team workflow – if developers only look at traces when debugging a specific ticket, a low random sample may be enough. If they routinely explore traces to find performance issues, you need a richer sample.
A common starting point is to trace 100% of requests on low-traffic services (under a few hundred requests per minute) and 5–10% on high-traffic services. Then adjust based on how often you find yourself missing a trace you needed.
Common Problems and Misconceptions
One frequent misconception is that a higher sampling rate always gives better visibility. In reality, if you never look at most of the traces you collect, you are paying overhead for data you do not use. Another issue is sampling without considering the whole request path: if you sample 10% at the edge but each downstream service independently samples 10%, the chance that all spans of a given request are kept is 0.1 raised to the number of services. For a 3-service call, that is only 0.1% of requests having complete traces. To avoid this, propagate the sampling decision in the trace context so all services follow the same choice.
Another problem is ignoring errors. A fixed 1% sample on a service with a 0.1% error rate will only capture one in ten errors. If you need to see every error, you must either trace 100% or use a sampler that always keeps error spans.
Comparing Sampling Strategies
Here is a quick comparison of the three main approaches:
- Fixed probability – simplest, low overhead, but may miss rare events. Good for general performance monitoring.
- Rate limiting – caps the number of traces per second, preventing overload during traffic spikes. Good for keeping costs predictable.
- Tail-based – keeps all traces that meet criteria (errors, slow, specific routes). Higher memory overhead, but best for debugging.
Some systems combine these: use a low fixed rate for all traffic, plus a tail-based sampler that retains any trace with an error or a duration above a threshold. This gives you both a representative sample and complete coverage of anomalies.
Costs and Safety Considerations
Sampling reduces tracing overhead, but it does not eliminate it. The instrumentation itself (creating spans, adding attributes) still runs on every request; sampling only controls whether the data is exported. If the instrumentation code is expensive, even a 0% sample will still cost CPU. Make sure your tracing library is efficient and that you are not adding heavy attributes to every span.
Also consider the cost of under-sampling. If you reduce the rate too much, you may lose the ability to diagnose a rare issue. A good rule is to keep enough traces that you can answer “what happened around this time?� for at least the last hour. If you cannot, your sampling rate is too low.
Finally, remember that sampling is not a security feature. Traces often contain sensitive data (user IDs, query parameters), so ensure your sampling and retention policies comply with your data protection obligations. Do not rely on sampling to hide sensitive information; use redaction.
Alternatives to Sampling
If sampling is not enough, consider other ways to reduce overhead:
- Reduce span granularity – trace only the services you care about, not every library call.
- Use metrics instead – for high-cardinality data like request counts and latency histograms, metrics are far cheaper than traces.
- Aggregate spans – combine multiple spans into one summary span when you do not need individual timings.
- Increase export interval – batch spans and export less frequently to reduce network I/O.
These can complement sampling, giving you visibility without the full cost of tracing everything.
Frequently Asked Questions
What is the best sampling rate for a typical web service?
There is no single best rate. For a service with under 100 requests per second, 100% sampling is often affordable. For higher traffic, start at 10% and adjust based on how often you need traces for debugging. If you rarely look at traces, a lower rate like 1–5% may be enough.
Does sampling affect latency measurements?
Sampling itself does not change the latency of the requests you do trace, but it can hide slow requests if they are not sampled. If you need accurate latency percentiles, use a high sampling rate or a tail-based sampler that captures slow traces.
How do I ensure all spans of a trace are kept together?
Propagate the sampling decision in the trace context (e.g., via a trace header). When a service decides to sample a request, it should pass that decision to downstream services so they all keep their spans. This avoids the “partial trace� problem.
Can I sample based on error status?
Yes. Many tracing libraries support a sampler that always keeps traces with an error status. This is a form of tail-based sampling and is highly recommended if errors are rare but important.
What is the overhead of tail-based sampling?
Tail-based sampling requires buffering all spans temporarily until the trace completes, which uses memory. The overhead depends on the number of concurrent requests and the span size. For high-throughput systems, this can be significant, so it is often used only for a subset of traffic.