Skip to content

PATTERN Cited by 1 source

Rate controller for asymmetric media

Context

Some storage media have read throughput substantially higher than write throughput on the same device. The archetype is QLC flash — reads are up to 4× or more faster than writes because Quad-Level-Cell programming is slow and iterative while reading is fast and parallel (see concepts/qlc-read-write-asymmetry).

Mixed workloads on asymmetric media expose latency-sensitive reads to being queued behind slow writes. Naive FIFO queueing defeats the read-bandwidth benefit the media was deployed for. The result: workloads that should run read-BW-bound instead run write-tail-latency-bound.

The pattern

Install a rate controller in front of the media that arbitrates admission between read and write streams, preserving read-latency headroom under write load.

Specifics typically include:

  1. I/O classification at admission. Tag each request as read vs write and (optionally) latency-sensitive vs bulk.
  2. Cap concurrent writes at a fraction of the drive's write capacity that leaves enough queue headroom for reads not to stall. The cap is tuned per drive and per workload — no single universal value.
  3. Prioritise reads in the scheduler, or operate separate queues with different service policies.
  4. Pace writes with a token-bucket or leaky-bucket to avoid burst-write-followed-by-read-starvation.
  5. Feedback from the media. If the FTL is host-side (patterns/userspace-ftl-via-io-uring), observe actual queue depth / GC state and adjust the write budget dynamically. Fallback is static tuning.
  6. Iterate. Meta: "requires building, and carefully tuning, rate controllers and I/O schedulers" — the tuning is explicit.

Canonical instance

Meta's 2025-03-04 QLC post:

"Read throughput in the case of QLC can be as high as 4x or more than write throughput. What's more, typical use cases around reads are latency sensitive so we need to make sure that the I/O delivering this massive read BW is not getting serialized behind the writes. This requires building, and carefully tuning, rate controllers and I/O schedulers."

The rate controller here is the software-side primitive that makes QLC deployable on latency-sensitive read-dominant workloads. Without it, QLC's nominal read-BW advantage is eaten by tail-latency from write interference.

Why this can't (usually) live in the drive

Modern NVMe drives expose a single namespace-level queue with limited host visibility into internal scheduling. The drive's firmware may interleave reads and writes, but its policy is vendor-fixed and not workload-aware.

Putting the rate controller in the host — ideally co-located with the FTL in userspace (patterns/userspace-ftl-via-io-uring) — gives the scheduler full visibility into pending writes and full control over admission. This is one of the composition reasons userspace-FTL designs emerge at the QLC density tier.

Trade-offs

  • Write throughput caps mean bulk writes run slower. If the workload has bounded write-BW requirements (true for Meta's target QLC workloads), this is fine. For write-heavy workloads QLC is the wrong tier anyway.
  • Tuning overhead. The right write-budget depends on drive, firmware, workload mix, temperature. Per-workload tuning is expensive; auto-tuning is hard.
  • Interaction with GC. FTL garbage collection also consumes write bandwidth. The rate controller needs to reserve budget for GC, or coordinate with a host-side FTL that schedules GC.
  • Tail still exists. Rate control improves p50 and p99; p99.9 tail events during drive-internal pauses (erase, GC spike) are reduced but not eliminated.

Generalization

The pattern extends to any medium with asymmetric R/W throughput serving mixed workloads — future PLC (5-bit-cell) flash, computational storage devices, tiered-cache hierarchies, even some SMR HDD write-band modes. The QLC instance is the 2025 canonical example because it's the first at hyperscale to have the asymmetry be large enough to require host-side arbitration as a first-class design concern.

Seen in

Last updated · 319 distilled / 1,201 read