Skip to content

PATTERN Cited by 1 source

Full-stack IO instrumentation

Intent

Instrument every IO at multiple points in every subsystem of the storage stack — plus run continuous canary workloads — so that (a) the true source of latency/variance can be localized across layers and (b) every subsequent change is falsifiable.

Motivation (EBS, 2012)

At this point in EBS's history (2012), we only had rudimentary telemetry. To know what to fix, we had to know what was broken, and then prioritize those fixes based on effort and rewards.

EBS had a performance problem but couldn't attribute it. The turnaround began not with a code change but with a measurement change.

Components

  1. Per-IO, per-layer instrumentation. Timestamps and counters at:
  2. the client initiator (software in the EC2 instance/hypervisor),
  3. the network stack,
  4. the storage durability engine,
  5. the operating system (kernel + driver queues).
  6. Continuous canary workloads. Known-shape benchmarks running 24×7 against the live fleet. Any perf regression from a deploy shows up in the canary before customers see it.
  7. Customer-workload monitoring. Shape of real traffic tracked against the canary as a drift detector.

Why it earns every other change its keep

  • Changes ship with a falsifiable hypothesis. "This change should reduce p99 at layer X by Y%" can be checked.
  • Regressions are caught early. Canaries fail before customer tickets land.
  • Cross-layer attribution becomes possible. Combined with patterns/loopback-isolation you can say "the Xen dom0 queue accounts for X µs of our current p99."
  • patterns/loopback-isolation — the method that complements full-stack instrumentation: replace a layer with near-zero-latency stub, remeasure, attribute.
  • concepts/observability — broader pattern of which this is the storage-IO specialization.

Seen in

Last updated · 200 distilled / 1,178 read