Skip to content

PATTERN Cited by 1 source

Data-placement spreading

Pattern

For each new object, choose a different subset of drives to place it on — even for objects owned by the same tenant, in the same bucket. The result: a single tenant's data is scattered across the majority of the fleet, rather than concentrated on a small set of drives.

Two benefits (Warfield, 2025)

  1. A customer's data only occupies a very small amount of any given disk, which helps achieve workload isolation, because individual workloads can't generate a hotspot on any one disk.
  2. Individual workloads can burst up to a scale of disks that would be really difficult and really expensive to build as a stand-alone system.

(1) Per-disk workload isolation

If a tenant's data spans millions of drives, no drive holds enough of the tenant's objects for that tenant's burst to create a single-drive hotspot. The per-drive load from any given customer is tiny; only aggregate load matters.

(2) Customer burst access to the whole fleet

The post's headline number:

Today, we have tens of thousands of customers with S3 buckets that are spread across millions of drives. […] That burst of requests can be served by over a million individual disks. That's not an exaggeration.

A stand-alone system trying to match that burst profile would have to provision a million-disk fleet for one customer. In S3, it falls out of the placement policy — the customer gets the whole fleet on their burst, everyone else is unaffected because the burst is a vanishing fraction of per-drive load.

Interactions

  • With concepts/aggregate-demand-smoothing: spreading is what translates smooth aggregate demand into smooth per-drive demand. Without spread, aggregate smoothing wouldn't help individual drives.
  • With concepts/erasure-coding: individual objects are split into (k+m) shards that might span tens of drives. The different objects on different drive sets rule applies at the object level, not the shard level.
  • With patterns/redundancy-for-heat: once objects are spread, redundancy provides the steering degree-of-freedom within the spread.
  • With concepts/noisy-neighbor: the EBS lesson is that "spread a hot tenant widens blast radius." Spread works in S3 precisely because concepts/aggregate-demand-smoothing makes any one customer's impact-per-drive negligible. The patterns differ because the scale and workload mix differ.

Preconditions

  • Large enough fleet that any one customer is a small fraction of total drives.
  • Large enough tenant population that aggregate demand smooths.
  • Metadata layer that can resolve many-drive placements efficiently (object → drive-set lookup is on the hot path).

Failure modes

  • Correlated workloads across tenants (e.g., everyone hits the top-of-hour batch) can still cause aggregate peaks.
  • Very large single tenants can erode the aggregation assumption. S3's customer mix has historically stayed broad enough that this doesn't dominate.
  • Small buckets' tax: a bucket with few objects can't fully benefit — the spread is bounded by object count. The post doesn't address this explicitly but implies it's why S3's placement is "across lots of disks" rather than "all disks."

Seen in

Last updated · 200 distilled / 1,178 read