Skip to content

CONCEPT Cited by 1 source

Constrained-resource benchmark

Definition

A constrained-resource benchmark is a benchmark run with an explicit resource cap (CPU cores, memory, IOPS, network bandwidth) chosen to represent a specific production tier, rather than a benchmark run with "as much hardware as we can give it" aiming to demonstrate the substrate's ceiling. The purpose is capability disclosure within a consumer-meaningful envelope — what a customer at this tier will actually see — rather than peak-capability-for-PR.

This is a distinct methodology from unconstrained performance benchmarks, which optimise the environment for maximum headline number. A constrained-resource benchmark deliberately leaves resources on the table to match what the customer's tier buys them.

Canonical instance: PlanetScale × TAOBench (2022)

Source: Van Dijk 2022-09-08.

Audrey Cheng's UC Berkeley team ran TAOBench against PlanetScale with a 48-CPU-core resource limit. The PlanetScale deployment under test was their multi-tenant serverless offering — where some infrastructure is shared across tenants. PlanetScale allocated the 48 cores as:

  • 44 cores for the query path — VTGate + VTTablet + backing MySQL — the resources actually doing the customer's work.
  • 4 cores for multi-tenant serverless overhead — edge load balancers, shared components every tenant pays some small fraction of.

Verbatim: "The limit Cheng's team set for their tests was 48 CPU cores. Since the test was performed against PlanetScale's multi-tenant serverless database offering where certain infrastructural resources are shared across multiple tenants, we underprovisioned the 'query path' of the cluster itself to use a maximum of 44 CPU cores out of the requested 48 maximum. The other 4 cores would be used for multi-tenant aspects of the infrastructure, such as edge load balancers."

The 44 + 4 = 48 split is canonical: it's the disclosure shape for capability-vs-allocation transparency on a multi-tenant substrate. A single-tenant deployment would have no such split — all 48 cores would go to the query path.

Why it matters

  • Benchmarks without an explicit cap are not tier-aligned. If a vendor's benchmark uses 200 cores and the customer's tier ships 8, the headline number doesn't translate. Capping the benchmark to a tier-representative envelope makes the result actionable for the customer reading it.
  • Multi-tenant substrates have overhead that benchmarks must subtract. Shared infrastructure (load balancers, routing, authentication, observability taps) consumes non-zero compute. Benchmarks that don't subtract this overhead over-claim the tenant-facing ceiling.
  • Saturation behaviour at the cap is more informative than peak throughput uncapped. A benchmark at the cap can expose graceful saturation vs. congestive collapse — what happens past 100% CPU of what the tenant owns, which is the failure mode the tenant will actually encounter.
  • Tier selection becomes a substrate-design concern. When a benchmark is run at tier X's resource cap, the numbers encode information about what tier X feels like in production — which is the information the customer is trying to buy.

Contrast with unconstrained benchmarks

Dimension Constrained-resource benchmark Unconstrained benchmark
Resource budget Fixed cap (e.g. 48 cores, 32 GB) Whatever hardware available
Purpose Tier-aligned capability disclosure Peak substrate capability
Audience Prospective customers at that tier Architects evaluating substrate ceiling
Overhead disclosed Explicit split (e.g. 44 query + 4 multi-tenant) Aggregate only
Most informative signal Saturation behaviour at the cap Peak throughput achievable
Canonical example PlanetScale × TAOBench (48-core cap, 44+4 split) PlanetScale × sysbench-tpcc 1M-QPS (single-tenant, resources provisioned to hit 1M)

The two shapes are both useful — they answer different questions. The peak-QPS benchmark answers "what can the substrate do?"; the constrained-resource benchmark answers "what can the substrate do at the tier I'm buying?".

When to use it

  • Multi-tenant offerings where headline numbers from uncapped runs would mislead tenants about what they'll actually see.
  • Capability disclosure within a documented tier (e.g. Heroku Hobby vs. Performance; PlanetScale Scaler vs. Team vs. Enterprise).
  • Comparing substrates at the same tier, holding resources constant so the variable is the substrate's efficiency.
  • Demonstrating graceful saturation behaviour — easier to drive a cluster to 100% CPU with a cap than without.

Trade-offs

  • Less marketable headline number. The 44-core-constrained QPS is strictly smaller than the unconstrained QPS. Readers comparing across posts without reading the methodology may miscompare numbers.
  • Cap selection is a judgement call. 48 cores wasn't arbitrary — Cheng's team chose it as the TAOBench test limit. The constrained-resource methodology inherits whatever representativeness (or bias) the cap selection introduces.
  • Requires tenant-tier transparency. A vendor that doesn't publish tier definitions can't publish tier-aligned benchmarks — the methodology assumes the tier is a named, documented product unit.

Seen in

  • sources/2026-04-21-planetscale-taobench-running-social-media-workloads-on-planetscale — Van Dijk canonicalises the 48-core cap with the 44+4 split as the methodology shape for multi-tenant-serverless capability benchmarks. Explicit note: "the numbers themselves are by no means approaching the limits of what can be accomplished on our infrastructure. Rather, they represent what is achieved by imposing an explicit resource limit to the cluster." This is the canonical vendor-disclosure that the constrained-resource number is deliberately smaller than the unconstrained one, because it's measuring a different thing.
Last updated · 550 distilled / 1,221 read