Back to Monitoring & Logging
Q7Question 7 of 9

Cardinality explosion in Prometheus metrics: what it is, the incident it causes, and how to avoid it

Key Concepts
  • Prometheus creates one time-series per unique combination of metric name and label values
  • Total series count is the product of distinct values of each label: 5 services x 3 statuses x 2M accounts = 30M series
  • High-cardinality labels (userId, orderId, traceId, email, URL with IDs) are the classic offenders
  • The incident is cascading: TSDB OOM, crashed Prometheus, failed scrapes, slow dashboards, lost monitoring
  • Keep labels low-cardinality and bounded; put identifiers in logs and traces, not metric labels
  • Use metric_relabel_configs to drop dangerous labels at scrape time and sample_limit to cap series per target

Definition

One time-series per unique label combination means an unbounded label like account ID or trace ID explodes series into the millions and crashes Prometheus: keep those identifiers in logs and traces, never in metric labels.
  • Cardinality explosion is the uncontrolled growth of time-series caused by attaching a high-cardinality or unbounded label to a metric
  • Prometheus creates one distinct time-series for every unique combination of metric name and label values
  • High-cardinality labels have large or infinite value spaces: user ID, order ID, email, trace ID, URL paths with embedded IDs
  • The incident is memory exhaustion in the TSDB, crashed or OOM-killed Prometheus, failed scrapes, and slow dashboards
  • The fix is keeping labels low-cardinality and bounded, and putting high-cardinality identifiers in logs and traces where they belong

Prometheus keeps one notebook page for every unique kind of measurement. A sensible label is like "transaction type": there are only a few values (transfer, deposit, withdrawal), so only a few pages. But imagine someone decides to label every measurement with the customer's account number. The National Bank of India has millions of customers like Rajesh SAV2001 and Priya CUR3001, so Prometheus now needs millions of pages, one per account, forever. The notebook grows until the supervisor's desk collapses under its weight. That collapse is cardinality explosion.

Understanding the Concepts

What causes cardinality explosion?

The root mechanic is that Prometheus identifies a time-series by its full label set. The total number of series for a metric is the product of the number of distinct values of each label.

TEXT
# Safe: bounded labels
transfer_requests_total{service="transfer", status="200"}
transfer_requests_total{service="transfer", status="500"}
# 5 services x 3 statuses = 15 series (fine)

# Dangerous: add account_id label
transfer_requests_total{service="transfer", status="200", account_id="SAV2001"}
transfer_requests_total{service="transfer", status="200", account_id="CUR3001"}
# ... 2 million more
# 5 services x 3 statuses x 2M accounts = 30 MILLION series (explosion)

Each series carries overhead in memory for its index and recent samples, so millions of series exhaust RAM.

Which labels are high-cardinality?

High-cardinality labels are ones with large or unbounded value spaces. These are the classic offenders:

| Label | Cardinality | Why Dangerous |
|---|---|---|
| account_id | Millions | One per customer |
| user_id | Millions | One per user |
| order_id / txn_id | Unbounded | Grows with every transaction |
| email | Millions | One per user |
| trace_id | Unbounded | One per request |
| session_id | Unbounded | One per session |
| URL path with ID | Unbounded | /accounts/SAV2001/txns/TXN-99817 |

JAVA
// WRONG: account_id in a metric label
Counter.builder("transfer.requests")
    .tag("service", "transfer")
    .tag("accountId", request.getAccountId())   // Millions of unique values!
    .register(registry)
    .increment();

// CORRECT: account_id in a log line, not a metric label
Counter.builder("transfer.requests")
    .tag("service", "transfer")
    .tag("accountType", "savings")   // Bounded: savings, current, loan
    .register(registry)
    .increment();
log.info("Transfer processed for account {}", request.getAccountId());

What incident does cardinality explosion cause?

The incident is severe and cascading. As series count climbs, Prometheus memory grows until the process is OOM-killed.

DIAGRAM
Cascading Failure Timeline:

1. Bad deploy adds account_id label to transfer counter
2. Series count climbs: 100K → 1M → 5M → 15M
3. Prometheus memory: 4GB → 8GB → 16GB → OOM-killed
4. Prometheus restarts, replays huge WAL (write-ahead log)
5. Restart is slow or fails entirely
6. Scrapes fail -- no new data ingested
7. Dashboards time out -- aggregations over millions of series
8. Alerts stop firing -- monitoring is blind
9. You lose monitoring exactly when you need it most

How do you keep labels low-cardinality?

The rule is that metric labels should only have values from a small, stable set. Anything that looks like an identifier belongs in logs and traces.

JAVA
// Good labels (bounded, stable):
// service: transfer, account, fraud, notification (< 20 values)
// status: 200, 400, 500 (< 10 values)
// method: GET, POST, PUT, DELETE (< 10 values)
// accountType: savings, current, loan (< 5 values)
// environment: prod, staging, dev (< 5 values)

// Bad labels (unbounded):
// accountId: SAV2001, CUR3001, ... (millions)
// transactionId: TXN-1, TXN-2, ... (grows forever)
// traceId: abc123, def456, ... (one per request)

How do you aggregate before exposing?

Instead of per-account counters, expose a per-account-type counter. The detailed breakdown belongs in logs.

JAVA
// WRONG: Per-account metric (millions of series)
Counter.builder("transfer.amount")
    .tag("account", accountId)   // 2M unique values
    .register(registry)
    .increment(amount.doubleValue());

// CORRECT: Aggregate by account type (bounded)
Counter.builder("transfer.amount")
    .tag("accountType", account.getType().name())  // savings, current, loan
    .register(registry)
    .increment(amount.doubleValue());

// Put the per-account detail in the log
log.info("Transfer amount={} account={} type={}",
    amount, accountId, account.getType());

How do you apply Prometheus guardrails?

Prometheus itself provides several guardrails to prevent and contain cardinality explosion.

YAML
# prometheus.yml -- guardrails
scrape_configs:
  - job_name: 'transfer-service'
    sample_limit: 10000    # Cap series per target; scrape fails if exceeded
    label_limit: 20        # Max number of labels per series
    label_value_length_limit: 128   # Max label value length

    metric_relabel_configs:
      # Drop any metric that has an account_id label
      - source_labels: [account_id]
        regex: .+
        action: drop

      # Drop a specific high-cardinality metric entirely
      - source_labels: [__name__]
        regex: 'transfer_per_account_total'
        action: drop

How do you detect cardinality explosion?

Monitor your own cardinality with self-monitoring queries to find which metric is exploding before it crashes the system.

PROMQL
# Top 10 metrics by series count
topk(10, count by (__name__)({__name__=~".+"}))

# Total active series (watch for sudden spikes)
prometheus_tsdb_head_series

# Series created per second (a spike means a new high-cardinality label)
rate(prometheus_tsdb_head_series_created_total[5m])

# Memory used by the TSDB
process_resident_memory_bytes{job="prometheus"}

What are common mistakes?

The most common mistake is adding a label that seems bounded but is not at scale.

JAVA
// WRONG: URL path looks bounded but includes IDs
Counter.builder("http.requests")
    .tag("path", request.getRequestURI())  // /accounts/SAV2001/transactions/TXN-99817
    .register(registry)
    .increment();
// Every unique URL creates a new series!

// CORRECT: Use the route template, not the actual URL
Counter.builder("http.requests")
    .tag("path", "/accounts/{accountId}/transactions/{txnId}")  // One value
    .register(registry)
    .increment();

What should a senior engineer know?

The fix for an already-happening explosion is to drop the offending label at scrape time with metric_relabel_configs, redeploy the instrumentation to stop emitting it, and if needed delete the bad series from the TSDB. The preventive culture is a rule in code review: any new metric label must have a provably bounded value set. Histograms also quietly add cardinality through their le bucket label, so you keep bucket counts reasonable.

BASH
# Emergency: delete bad series from TSDB (requires admin API)
curl -X POST 'http://prometheus:9090/api/v1/admin/tsdb/delete_series' \
  -d 'match[]=transfer_per_account_total'

# Then compact to reclaim space
curl -X POST 'http://prometheus:9090/api/v1/admin/tsdb/clean_tombstones'

What are common interview follow-up questions?

  • Can you fix cardinality explosion without redeploying the app? Yes, use metric_relabel_configs to drop the label at scrape time.
  • Where should high-cardinality identifiers go? In logs and traces, not metric labels.
  • What is the recommended maximum series count? It depends on hardware, but a single Prometheus instance typically handles 1-10 million series; beyond that use Thanos or Cortex for horizontal scaling.

Visual Overview

DIAGRAM
  Cardinality Multiplication
  ┌────────────┐   ┌────────────┐   ┌──────────────┐
  │ service    │ x │ status     │ x │ account_id   │ = Total Series
  │ (5 values) │   │ (3 values) │   │ (2M values)  │   = 30,000,000
  └────────────┘   └────────────┘   └──────────────┘

  Cascading Failure
  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐
  │ Bad label    │───▶│ Series count │───▶│ Memory       │
  │ deployed     │    │ explodes     │    │ exhaustion   │
  └──────────────┘    └──────────────┘    └──────┬───────┘
                                                  │
                                                  ▼
  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐
  │ Monitoring   │◀───│ Dashboards   │◀───│ Prometheus   │
  │ blind        │    │ timeout      │    │ OOM-killed   │
  └──────────────┘    └──────────────┘    └──────────────┘

  Where Identifiers Belong
  ┌────────────────────────────────────────────┐
  │  Bounded labels ──▶ METRICS                │
  │  (service, status, method, accountType)    │
  │                                             │
  │  Unbounded identifiers ──▶ LOGS + TRACES   │
  │  (accountId, txnId, traceId, email)        │
  └────────────────────────────────────────────┘

Key Takeaway

Cardinality explosion is what happens when you attach a high-cardinality or unbounded label to a Prometheus metric. Prometheus creates one separate time-series for every unique combination of a metric name and its label values, and the total series count is the product of the distinct values of each label. A label like status with three values is harmless, but a label like account ID, user ID, order ID, or trace ID has millions of possible values, multiplying across your other labels to produce millions of series, each consuming memory essentially forever.

The incident is severe and cascading. Memory in the time-series database climbs until Prometheus is OOM-killed. On restart it has to replay a huge write-ahead log, so it comes back slowly or not at all. Scrapes fail, dashboards timeout, and you lose your monitoring exactly when you need it most. A single deploy that adds one unbounded label can take down the whole monitoring stack.

Avoiding it is about discipline and guardrails. Keep metric labels low-cardinality and bounded: service name, environment, HTTP method, status code class. Anything that looks like an identifier goes into logs and traces, because a log line is the right place to record an account ID and a trace ID. Aggregate before exposing, so instead of a per-account counter use a per-account-type counter. Put hard guardrails in Prometheus: sample_limit on the scrape job so a misbehaving target fails safely, and metric_relabel_configs to drop a dangerous label at scrape time.

Remember one sentence: metrics are for bounded aggregates and identifiers belong in logs and traces, never in metric labels.

---

Share
Use ← → arrow keys to navigate