Skip to content

CONCEPT Cited by 3 sources

Metric cardinality

Metric cardinality is the number of unique combinations of label values a metric has — e.g., cpu_usage{pod="...", tenant="..."} has one series per distinct (pod, tenant) pair. Cardinality is the primary scaling factor for a TSDB: memory cost, query cost, storage cost, and block-compaction cost all scale with the count of active series, not with the raw sample rate.

Why it dominates TSDB scaling

A TSDB indexes every (metric_name, label_set) pair it has ever seen in memory; queries touch the index to resolve series before fetching sample data. A 10× increase in pod count with a pod_id label means 10× more series, 10× more index memory, 10× more query fan-out, and 10× more blocks to compact. Sample rate is linear; cardinality is the knob that multiplies.

Serverless amplifies the problem: short-lived workloads (serverless functions, ephemeral VMs, Kubernetes pods on autoscalers) mean each label value has a vanishingly short lifetime. The cardinality keeps growing even if the instantaneous fleet size is stable, because label values churn — each new pod ID is a new series that the TSDB must index before it disappears. See serverless-workload-churn-cardinality.

Why engineers still add high-cardinality labels

Metric owners add labels like node_id, pod_id, tenant_id, request_id because those dimensions are exactly what makes a metric useful during incidents — aggregated metrics tell you "region CPU is elevated" but can't tell you "which tenant is causing swap pressure, which node crashed, which shard is isolated." High cardinality is a feature, not a bug; the scaling problem is real.

Responses

Three families of responses:

  • Aggregation shield — drop expensive labels before they reach the TSDB, trading incident-debugging detail for bounded TSDB scaling. See concepts/metric-cardinality.
  • Limit / reject at ingestion — enforce per-tenant or per-metric cardinality caps at the router / ingestion tier. Simple but rejects data.
  • Store raw data elsewhere — keep raw high-cardinality data in a lakehouse / OLAP store that scales horizontally with object storage, and let the TSDB serve only aggregated views. See patterns/telemetry-to-lakehouse.

Most hyperscale observability platforms end up combining all three, with tight interface-stability discipline to make the split invisible to users (see unified-metric-semantics).

Seen in

  • sources/2026-05-05-databricks-10-trillion-samples-a-day-scaling-beyond-traditional-monitoring — canonical production framing. "Cardinality is the primary scaling factor for a TSDB, and growth in the cardinality of existing metrics increases costs and scaling pressure on Pantheon." At Databricks: serverless compute launches "tens of millions of VMs daily," which together with the >5B active-timeseries fleet motivated a three-pronged response (Pantheon tiered-storage TSDB, Telegraf aggregation shield, Hydra raw-lakehouse tier).

  • sources/2026-08-13-aws-adobe-firefly-simplified-observability-with-amazon-managed-prometheus — GPU training makes cardinality operationally valuable: GPU, job, memory, network, pod, node, resource-allocation, and health dimensions are needed to diagnose bottlenecks. Adobe's representative 2,000-node / 16,000-GPU fleet, scraped every 30 seconds, can yield more than one billion data points in a query window. Moving a critical 2-million-time-series tier to AMP made the 24-hour GPU-utilisation query 28.8× faster without requiring the team to discard its self-managed Prometheus workflow. This is a managed-TSDB response that preserves high-dimensional telemetry rather than reducing labels first. (Source: sources/2026-08-13-aws-adobe-firefly-simplified-observability-with-amazon-managed-prometheus)

  • sources/2026-10-01-grafana-tempo-31-release-kafka-traceql-metrics-trace-redaction — per-span label-set construction as the generator cost floor. The cost of deriving metrics from traces is set by per-span work, because a metrics-generator processes every ingested span, and building the (metric_name, label_set) tuples was a dominant share of that work. Grafana Tempo 3.1's span-metrics and service-graphs processors borrow pooled label buffers from the registry instead of allocating a new label set per span, and steady-state service-graphs benchmarks report zero allocations per operation — same metric names/labels/values, lower generator CPU/memory. A trace-side mirror of the TSDB lesson that cardinality (label-set work), not raw sample rate, is the scaling knob.

Merged aliases

  • metric-aggregation-as-cardinality-shield- high-cardinality-aggregation-overload
Last updated · 766 distilled / 2,225 read