If you’ve ever opened your tracing dashboard after an incident and discovered the failing trace simply isn’t there, you know the exact problem: trace sampling stripped away the errors. The fix is a pattern called tail-based sampling—use it to keep every failure and cut volume at the same time. In this article, we’ll break down why head-based sampling is the culprit and how to implement tail-based sampling without blowing up your observability budget.
Why Head-Based Sampling Misses the Failures
Head-based sampling makes a decision at the start of a request. The tracer assigns a sampling decision based on a fixed probability—say 10%—and propagates that decision across all spans. This is cheap and easy, but it has a critical flaw: the sampling decision doesn’t know whether the request will eventually fail. A rare error that happens 0.1% of the time will be sampled at the same rate as a healthy request. If your sampling rate is 5%, you will lose 95% of your errors.
In practice, this means error rates and root-cause investigations suffer. The traces that could explain a timeout, a bad cache stampede, or a database connection leak never make it to the backend. Head-based sampling is still useful for broad traffic visibility, but it cannot guarantee error coverage. The more aggressive your sampling rate, the more likely you are to be blind when an incident occurs.
The Tail-Based Sampling Pattern: Decide After the Fact
Tail-based sampling flips the decision point. Instead of choosing at the beginning, a collector or sampler buffers the spans from a trace until the trace is complete. Then it evaluates the whole trace and decides whether to store it. This allows the sampler to use much richer criteria, and the most important rule is: if any span contains an error, keep the entire trace.
How It Works in a Modern Observability Pipeline
When a service emits spans, they are forwarded to a trace-aware backend or a dedicated sampling agent. The agent groups spans by trace ID and waits for a completion signal—usually a timeout or an explicit end flag. Once the trace is complete, the sampler checks for error status codes, exception events, or latency outliers. If the trace qualifies, it’s stored; otherwise, it can be discarded or downsampled. This is the core of tail-based sampling.
The key difference is that you can now write a policy like “keep 100% of traces with HTTP 5xx or error spans” and “keep 10% of successful traces.” This way, tail-based sampling keeps every failure and cuts trace volume significantly for the paths that are operating normally.
Keeping Every Failure Without Losing Context
One concern teams raise is that keeping every failure might still be too much volume, especially in a healthy system where errors are frequent but not interesting. That’s where tail-based sampling gets more nuanced. You can define what counts as a “failure” per service or per span: a 4xx response may be a client problem; a 500 is a server problem; a timeout is an infrastructure problem. The sampling rule can be as precise as your telemetry allows.
For a large e-commerce platform, for example, the login service might have a 2% error rate, and the checkout service has a 0.5% error rate. With tail-based sampling, you can keep 100% of traces from both services that contain an error span, while healthy checkout traffic can be sampled at 5%. The result is that every failure is retained, and the volumes remain manageable.
Consider an inventory service that experiences a spike in database connection errors. With head-based sampling at 10%, the failing trace might not be stored, leaving engineers with only a counter alert. With tail-based sampling, the collector notices the error span and preserves the entire trace, including the database client span, the retry logic, and the original request context. This complete view turns a vague alarm into a direct pointer to the connection pool misconfiguration. That is the pattern’s real value.
Designing a Tail-Based Sampling Strategy in 2026
As microservices become more dynamic, sampling strategies need to adapt. Here’s a practical pattern for implementing tail-based sampling without drowning in data or missing edge cases.
1. Start with an Error Budget for Traces
Just as site reliability engineering uses error budgets for uptime, you can define an error budget for trace storage. Decide how many error traces you can afford to store per minute or hour. Then set your tail-based sampling policy to preserve all error traces up to that budget, and use a dynamic rate for the rest. This prevents a sudden burst of failures from flooding your storage while still giving you representative coverage.
2. Combine Head and Tail Sampling in a Hybrid Model
Tail-based sampling can be resource-intensive because it requires buffering spans. A full transition away from head-based sampling may not be practical for high-volume systems. A common pattern is to use head-based sampling to filter the obvious noise, then use tail-based sampling on the remaining stream. For example, set a coarse head sampling rate of 50% for all traffic, then apply tail-based sampling to retain only errors and high-value traces. This reduces buffering overhead while preserving failure coverage.
3. Use Dynamic Sampling Decisions Based on Cardinality
In 2026, many trace pipelines support “smart” sampling that considers trace properties like endpoint, service, user ID, or error type. Rather than a single fixed rate, you can set a dynamic policy that fires when a new or unusual error appears. This is useful for catching rare failures that your static rules might miss. The sampler can maintain a digest of recently seen error signatures and always keep the first occurrence of a new signature.
4. Monitor Sampling Effectiveness
Once tail-based sampling is in place, don’t assume it’s working. Look at metrics like “sampled error traces” versus “estimated total error traces” to check coverage. If the number of stored error traces is consistently close to the error count reported by your services, your policy is healthy. If there is a gap, inspect the batch timeout, policy precedence, and buffer limit. Also track the storage ratio: how many stored traces are error traces versus successful traces. A healthy ratio is not fixed, but it will help you tune your dynamic rates over time.
Pitfalls to Avoid When Shifting to Tail-Based Sampling
Tail-based sampling seems straightforward, but there are several traps that can sabotage your rollout.
- Buffering latency: Waiting for the “complete” signal means you need to hold spans in memory or temporary storage. Ensure your buffer size can handle a long-running request, such as an async job that takes minutes.
- Trace ID collisions: If your trace ID generation is not unique enough, you might merge unrelated spans. Use a 128-bit trace ID and validate your instrumentation library.
- Partial trace races: A trace might be considered complete by the collector before a late-running span arrives. Use a sufficiently long completion timeout, but not so long that it creates feedback delay.
- Storage pressure from negative results: If you keep every error trace, a noisy service can generate thousands of identical errors. Consider grouping duplicate errors and keeping a representative sample after the first N occurrences.
Tooling and Open Source Options for Tail-Based Sampling
Tail-based sampling is not a hypothetical pattern. It’s supported by many open source and commercial observability platforms. If you operate your own stack, the OpenTelemetry Collector now includes a tail_sampling processor that can apply complex policy combinations, like “keep if status is error” or “keep if latency is above threshold.” The processor buffers spans by trace ID and evaluates them according to your policy file.
Other tools, such as Grafana Tempo and Jaeger, offer trace storage and query features that work well with tail-based sampling. On the commercial side, observability vendors provide trace sampling controls that can be configured to retain errors and unusual traces while dropping routine traffic. The key is to choose a pipeline that lets you configure policies across multiple services without embedding sampling logic into every application.
Conclusion
Trace sampling stripped away the errors is a common pain, but it’s not an unsolvable one. Tail-based sampling gives you a robust pattern to keep every failure and cut volume by evaluating traces after they complete. By combining head-based filtering, error budgets, and dynamic policies, you can build an observability pipeline that never misses the trace that matters. In a distributed world, that visibility is worth the extra buffering cost.
