As engineering teams scale distributed systems in 2026, telemetry volume continues to balloon faster than infrastructure budgets. Every microservice emits spans, metrics, and logs, and storage vendors charge per gigabyte ingested. The result is an uncomfortable trade-off: pay more or drop data and lose visibility. Adaptive sampling in OpenTelemetry offers a middle path, keeping the traces that matter while trimming the long tail that drives up cost. This guide walks through how to design a sampling strategy that protects signal without lighting money on fire.
Why static sampling quietly drains your budget
Most teams start with a fixed head-based sampler. They pick a rate, say 10 percent, and apply it uniformly to every incoming request. It is simple, predictable, and easy to reason about during a postmortem. Unfortunately, it is also one of the worst cost-versus-signal deals you can make.
- Important failures are rare by definition, so a uniform sampler discards most of them.
- Noisy health checks and crawler traffic get the same treatment as user-driven checkout flows.
- Sampling decisions happen before the system knows whether the trace will contain an error or a slow span.
The net effect is that you pay full ingestion cost for the traces you keep, but you keep a mostly random slice. Tail-based and adaptive approaches flip this around: defer the decision until more context is available.
How OpenTelemetry represents sampling decisions
Before tuning anything, it helps to understand the moving parts. In the OpenTelemetry data model, every span carries a trace state and a sampling flag propagated through context. The SDK offers several sampler implementations out of the box:
- AlwaysOn / AlwaysOff: trivial, useful for tests and local debugging.
- TraceIDRatioBased: deterministic head-based sampling by trace ID hash.
- ParentBased: honors upstream decisions so a trace is sampled consistently across services.
- Composite: chains samplers so different rules apply at different stages of the pipeline.
Adaptive sampling builds on these primitives by inserting logic that observes the span in flight, evaluates attributes such as status code, latency, and HTTP route, and then decides whether to keep, drop, or upgrade the trace. The decision is recorded in the span itself, which keeps downstream processors honest.
Head-based versus tail-based sampling at a glance
Head-based sampling happens at the edge, the moment a request enters your service. It is cheap, stateless, and fast. Tail-based sampling runs after the trace is assembled, usually inside a collector or a vendor gateway, where the full picture is available. Adaptive strategies usually combine both: a permissive head sampler that keeps a small fraction by default, paired with a tail sampler that upgrades or discards based on outcome.
Designing an adaptive policy that matches real workloads
A useful sampling policy mirrors how your system actually fails. Generic rules such as “keep five percent” rarely map onto incident patterns. Instead, anchor your rules to observable signals that correlate with user pain.
1. Always keep errors and slow traces
Errors are rare, expensive to debug, and directly affect revenue. Configure your tail sampler to retain every trace where any span reports an error status, an HTTP 5xx response, or a latency exceeding the 95th percentile of recent traffic. This single rule usually captures the bulk of incidents and pays for itself.
2. Sample successful traffic by route priority
Not every endpoint deserves the same sampling budget. Critical paths like checkout, signup, and search deserve higher fidelity than internal health checks, asset fetches, and batch backfills. Express this with a route-based allowlist in your sampler. The collector can match on span attributes such as http.route or rpc.service and apply different rates per group.
3. Bound volume with reservoir and quota rules
Even with good targeting, a traffic spike can overwhelm the collector. Add a probabilistic reservoir layer that caps retained traces per second for each service. This protects downstream storage and keeps dashboards responsive when something goes wrong upstream.
4. Preserve representativeness for analytics
Error-heavy sampling biases analytics. If you only keep failures, latency dashboards become meaningless. Pair your high-priority retention with a small but consistent uniform sample across all traffic so that aggregate metrics stay statistically valid.
Putting it together: a reference configuration
The OpenTelemetry Collector ships a tail_sampling processor that captures exactly the workflow above. A pragmatic policy might look like this conceptually:
- Layer one: a head sampler at the SDK that retains 1 percent uniformly and forwards all traces to the gateway collector.
- Layer two: a tail sampler with composite policies that keep traces containing errors, high latency, or high-priority routes at 100 percent.
- Layer three: a probabilistic catch-all that retains up to a configurable traces-per-second budget for everything else.
- Layer four: a final rate limiter per service that prevents any single noisy neighbor from monopolizing storage.
This layered approach costs a little more in compute at the collector, but pays off quickly when ingest bills drop by half or more. Many teams find that the savings dwarf the additional CPU by an order of magnitude.
Measuring whether you actually kept signal
Cost reduction is meaningless if you lose the traces you need during incidents. Build a feedback loop before declaring victory. Useful metrics to track include:
- Error trace retention rate: percentage of failed requests that appear in your backend, compared with the same metric pre-change.
- Slow trace retention rate: traces above your latency SLO that survive sampling.
- Dashboards statistical drift: p50, p95, and p99 latency across uniform samples versus your historical baseline.
- Incident mean time to detection: whether engineers still spot regressions quickly after deploys.
Run the new policy in shadow mode for a week if your vendor supports it. Compare what would have been retained against what was actually retained. Iterate the policy until retention of high-value traces sits above 99 percent while overall ingest drops to your target.
Common pitfalls when adopting adaptive sampling
Adaptive sampling is not free of footguns. Watch for a few patterns that bite teams in production:
- Inconsistent parent decisions: if some services sample at the SDK while others defer to the collector, traces can split. Use the
ParentBasedsampler consistently. - Overcounting retries: retried spans can look like independent failures and inflate retention. Normalize on trace ID before counting errors.
- Missing context propagation: if your services do not propagate
traceparentcorrectly, tail sampling sees partial traces and drops useful data. - Vendor lock-in: some backend-specific tail samplers are not portable. If multi-cloud portability matters, prefer the standard
tail_samplingprocessor.
The takeaway
Adaptive sampling is less about clever math and more about matching retention to actual user impact. Keep every error, keep every slow request, keep the routes that drive revenue, and drop the rest probabilistically. With a layered head-and-tail configuration in the OpenTelemetry Collector, plus a measurement loop that watches both cost and signal, most teams can comfortably halve their observability bill without weakening their incident response. The trick is to treat sampling as a product surface that evolves with your system, not a knob to set once and forget.
