Prometheus is endlessly forgiving until it suddenly isn’t. For most SRE teams, the tipping point isn’t a spike in scrape volume or a disk filling with WAL files. It’s the moment a single metric quietly begins producing thousands of distinct time series because one label became unbounded. The real unit of cardinality is not a label’s unique value count, but every combination of label values attached to a metric. If you haven’t built a process to audit label value combinations before they hit production, you are one configuration change away from a monitoring meltdown. In this article, we’ll show you how to turn that chaotic surface area into a finite, reviewable inventory.
Why a Label Combination Audit Is Now an SRE Discipline
Modern observability pipelines collect data from service meshes, Kubernetes controllers, serverless functions, and API gateways—each adding its own opinionated labels. It is tempting to reason about cardinality by counting unique label values. But that’s like estimating the size of a city by counting street names instead of intersections. A metric with three labels and 100 distinct values per label can produce up to one million series, even if each individual label looks harmless. The consequence is not just increased storage cost. It creeps into every Prometheus query, causing rule evaluations to time out and dashboards to render slowly. In the worst cases, a single high-cardinality metric can exhaust the memory of a Prometheus instance and crash the entire service delivery pipeline for observability.
Cardinality explosions are increasingly triggered by business logic entering telemetry. Teams add labels with account IDs, request IDs, order numbers, or feature flags. These labels are useful for debugging but catastrophic for time-series storage. An SRE practice that treats cardinality as a deployment blocker, not an after-the-fact fire drill, is essential. Auditing label combinations before release is the only way to keep Prometheus fast, predictable, and useful.
The Anatomy of a Label Value Combination
Every time a metric is ingested, Prometheus combines the label set into a unique identity. For example, http_requests_total{method="GET", endpoint="/admin", status="500"} is one series. If your service has 10 methods, 50 endpoints, and 5 status codes, the theoretical maximum is 2,500 series. Add a label called user_id with only 100 concurrent users, and that becomes 250,000 series—assuming every user actually triggers every endpoint, which is often worse than you think.
The dangerous labels have a few common traits. They tend to be:
- Unbounded: Values are generated per request or per session, such as
trace_id,request_uuid, orlease_id. - User-controlled: Values come from query parameters, headers, or JWT claims, like
org_slugorcustomer_tier. - Multi-dimensional: They combine with other unbounded labels to produce combinatorial explosion, e.g.,
source_ip+target_endpoint. - Rarely aggregated: Cleanup or re-labeling rules remove them only after the damage is done.
If you see a label name ending in _id, _key, _token, or _name, treat it as a high-risk candidate for your audit.
How to Audit Label Value Combinations Before Deployment
The goal is not to eliminate all labels. The goal is to know exactly how many combinations are possible and to gate that number against a budget. Here is a repeatable, four-step process you can implement in 2026 without buying another tool.
Step 1: Inventory Your Label Universe
Start by extracting every metric used in your environment and every label attached to it. You can do this by querying the prometheus_tsdb_head_series metric or by exporting a snapshot from /api/v1/status/tsdb. For each metric, list the label names and the approximate number of unique values per label. You don’t need exact counts—just enough to spot combinatorial risk. Write the result into a static file that is reviewed in every change request that touches instrumentation.
Step 2: Set a Cardinality Budget
Define a maximum number of series per metric, per target, and per job. For example, a runtime metric like process_cpu_seconds_total should never have more than 10 series. An application metric like http_request_duration_seconds might be allowed 500 series if you aggregate status and endpoint into finite buckets. If a new label would push a metric above its budget, the change must be redesigned.
A useful technique is to document the budget inside the global config file as a comment, then use a script to compare the actual series count against it. You can automate this with a simple job that runs promtool query and fails if the result exceeds the threshold.
Step 3: Build an Audit Query or Script
PromQL can give you the exact series count for a metric. The query count({__name__="http_requests_total"}) returns the number of active series. To see label value counts, use count by (method) ({__name__="http_requests_total"}). But that only gets you halfway. To detect combinatorial explosion, you need to simulate the full cross product. Write a script that reads a metric’s labels and prints the product of all label value counts. If the product is larger than the actual count, you have a high potential for explosion when traffic patterns change.
Store that script in your infrastructure repo and run it against every staging environment. A cardinality regression is easier to catch when the label set is still under your control.
Step 4: Integrate with CI/CD
Cardinality controls should be a first-class gate in your deployment pipeline. Add a step that runs an audit against a temporary Prometheus instance loaded with the new rules and metric definitions. This catches the common mistake of adding a label in code without realizing it is attached to a high-volume metric. In a Kubernetes environment, you can also enforce it with a validating admission webhook for your custom Prometheus operator manifests.
Detecting Cardinality Explosion with Proactive Alerts
Audits are a release-time defense, but the curve can still bend after go-live. Alert on the rate of series growth, not just the absolute count. A useful alerting rule looks at the percentage of available series consumed by the current prefix of the label set. If the TSDB reports a 50% increase in series within five minutes, raise a page.
While you can write an alert like sum by (job) (count by (job, __name__) ({__name__=~".+"})) > 100000, be careful with its own cardinality. Aggregate with topk in a recording rule to avoid an expensive query in the alerting path. The key is to detect the growth trend, not to identify the exact offending label in real time.
Case Study: The Default Label Trap
Consider the classic story of a payments team that added a client_id label to a metric tracking login attempts. The label had fewer than 20 values in staging, so the CI check passed. In production, every third-party integration got its own client ID. Within an hour, the number of series exploded from 5,000 to 95,000 because client_id combined with endpoint, status_code, and error_type. Monitoring dashboards became white panels, and alerts misfired because rule evaluations timed out.
An audit would have flagged that client_id was unbounded and semantically attached to a metric that had four other labels. The fix was to replace client_id with client_tier (a low-cardinality bucket) and keep the raw client ID in logs, not in metrics. This is the fundamental trade-off: you can always enrich a series later by joining it with a log line, but you cannot recompress an exploded TSDB without painful downtime.
Tools and Techniques to Make Audits Repeatable
Prometheus itself exposes useful telemetry for cardinality audits. The prometheus_tsdb_head_series metric gives you a global count. The /api/v1/status/tsdb endpoint provides a per-metric breakdown of series counts, along with label value counts. If you use a multi-tenant long-term storage system like Thanos or Mimir, you can query their cardinality APIs to identify top series by freshness and size. Add a recurring job to export these statistics into a separate meta-monitoring bucket.
- Promtool: Use
promtool check metricsto validate the metric metadata, and extend it with custom scripts that compare against your budget. - Recording rules: Create a recording rule that calculates series count per metric every five minutes, then alert on the top offenders.
- Label linters: Integrate a linter into your IDE or pre-commit hook that rejects labels with suspicious names like
id,uuid, oreventunless explicitly whitelisted. - Cortex/Mimir cardinality APIs: If your architecture has moved to a horizontally scalable store, use the API to scan for series that have seen no writes for 24 hours (an indicator of label chaos).
The more automated the audit, the less chance a human has to accidentally approve a bad instrumentation pattern. Make the audit a bot that leaves a comment on your pull request with the estimated series count for each component. That bot is worth more than ten monitoring postmortems.
Conclusion
Prometheus cardinality explosion is not a problem you solve once and forget. It is a discipline of continuously auditing the label value combinations that define each metric, setting explicit budgets, and catching risky patterns before they reach a production scrape. By treating label combinations as the core unit of cardinality, you give your SRE team a practical way to keep metrics fast, reliable, and meaningful—even as systems become more distributed and labels multiply.
