CONCEPT Cited by 1 source
Tiered storage (hot / warm / cold)¶
A storage architecture that splits data across three tiers by age and access pattern:
- Hot (memory) — most recent data. Served at the lowest latency. Bounded by RAM.
- Warm (on-disk) — recent-but-not-current data, typically hours to a day. Served quickly from local disk, but can be re-created by replaying from the cold tier if needed.
- Cold (object storage) — all older data. Stored in S3 / GCS / Azure Blob — effectively unbounded capacity, highly durable, at the lowest $/GB.
The architectural payoff: compute and storage are decoupled. Compute nodes can scale up or down independently of the historical data volume, because the historical data lives in shared object storage and is queried on demand.
Why this matters for stateful databases¶
Vertically-scaling a database that keeps historical data local is painful: adding a node means redistributing historical data across nodes, which is slow, risky, and costly. Tiering the historical data out to object storage means adding a compute node is a zero-data operation — the new node just needs to join the cluster and can immediately serve queries against the cold tier.
At hyperscale ingestion rates, where scaling up is a frequent event (see concepts/tsdb-scaling-bottleneck), this decouple- compute-from-storage property is the difference between routine and heroic.
Canonical instances¶
- Thanos / Pantheon — memory (2h or 30m) / on-disk (24h) / object storage (rest). The memory-retention window is tunable per Receive group to match workload lifespan.
- Delta Lake + caching layer — Delta files in object storage are the cold tier; Databricks' caching tiers (disk + memory on compute nodes) form the warm + hot tiers. Delta time-travel + Z-order clustering optimise cold- tier queries.
- Kafka + tiered storage / systems/redpanda — similar split at the log-broker altitude: recent segments on broker- local NVMe, older segments in S3.
Seen in¶
- sources/2026-05-05-databricks-10-trillion-samples-a-day-scaling-beyond-traditional-monitoring — canonical TSDB-altitude instance. "A key element of Thanos is its tiered storage architecture. The most recent timeseries are kept in-memory, the last 24 hours' timeseries are kept on-disk, and all older data is kept on object storage ... leveraging object storage allows the system to essentially decouple compute from storage; a cluster can scale up without needing to rebalance all its historical data across database nodes." Pantheon adds the memory-retention-tier refinement (2h vs 30m for persistent vs ephemeral workloads).