PATTERN Cited by 1 source

Constrained-resource benchmark for shared-tenant capability disclosure

When benchmarking a multi-tenant substrate for customer-facing capability disclosure, cap the total resources at a tier-meaningful number, and publish the explicit split between tenant-facing query path and shared infrastructure overhead.

Canonical instance from PlanetScale × TAOBench (Van Dijk, 2022-09-08): the total cap was 48 CPU cores; PlanetScale allocated 44 cores to the query path (VTGate + VTTablet + MySQL) and 4 cores to multi-tenant serverless overhead (edge load balancers, shared components). Both numbers are disclosed in the post.

Intent

Fix the mismatch between uncapped vendor benchmarks (peak-substrate-capability, not useful to a customer buying a specific tier) and realistic tier-aligned capability disclosure. A customer on a multi-tenant-serverless tier will not see the same numbers as a single-tenant enterprise deployment with dedicated hardware — and a benchmark that doesn't disclose this distinction mis-sells the tier.

Mechanism

1. Pick a cap that represents a tier. Cheng's UC Berkeley team chose 48 CPU cores as the TAOBench test limit. This is the total compute budget the benchmark operates within.
2. Allocate within the cap, reserving overhead for shared infrastructure. For a multi-tenant substrate:
3. Identify shared components that every tenant's traffic touches (edge load balancers, authentication proxies, observability taps, routing layers).
4. Subtract their compute from the cap: "we underprovisioned the 'query path' of the cluster itself to use a maximum of 44 CPU cores out of the requested 48 maximum. The other 4 cores would be used for multi-tenant aspects of the infrastructure, such as edge load balancers."
5. Run the benchmark against the tenant-facing path only. The 44 cores do the query work. The 4 cores of overhead serve the shared infrastructure.
6. Publish both numbers. The overhead split is load-bearing disclosure — readers see "48 total, 44 query, 4 overhead" rather than a bare "48 cores". This enables apples-to-apples comparison with other tiers and substrates.
7. Frame the result as tier-capability, not substrate-ceiling. "the numbers themselves are by no means approaching the limits of what can be accomplished on our infrastructure. Rather, they represent what is achieved by imposing an explicit resource limit to the cluster." (Van Dijk.)

When to use it

• The offering is multi-tenant (serverless databases, BaaS, PaaS) and tenants don't control the shared-infrastructure layer.
• You want to publish a benchmark that customers on a specific tier can read as "what I'll get" rather than "what the substrate can do with unlimited hardware".
• The offering has tiered pricing tied to resource envelopes and marketing materials risk misleading customers by reporting unconstrained peak numbers.
• You want to demonstrate graceful saturation — it's easier to drive a capped cluster past 100% CPU than an uncapped one.
• Independent parties (academics, customers, competitors) are likely to re-run the benchmark and publish their own numbers — transparency about the cap + split prevents adversarial framing.

When NOT to use it

• Single-tenant deployments with dedicated hardware — the overhead split doesn't apply; there's no shared infrastructure to reserve cores for.
• Capability-envelope benchmarks where the goal is genuinely "what can the substrate do at scale?" — for example, PlanetScale's 1M-QPS single-tenant sysbench-tpcc post deliberately uses an unconstrained single-tenant deployment because the question it answers is substrate-ceiling, not tier-capability.
• Academic benchmarks aimed at comparing substrates without tier-specific context — the constraint imposes tier-coupling that may not generalise.

Trade-offs

• Lower headline numbers. The 44-core-capped QPS is strictly smaller than the unconstrained QPS. Marketing teams may prefer the bigger number; this pattern deliberately trades that off for truthfulness about what the tier delivers.
• Tier selection is a judgement call. The cap picked encodes assumptions about the representative tier. A different cap choice (say 16 cores vs. 48) would produce different numbers; the cap choice should be justified in the publication.
• Overhead split requires transparency about shared infrastructure. Not every multi-tenant substrate wants to disclose what its shared components are. This pattern requires the vendor to be specific about what the 4 cores are for, not just that there are 4 cores of overhead.
• Benchmark code must support the cap. The benchmark runner needs a way to enforce the cap — cgroups, resource limits, container quotas, or Kubernetes resource requests. Off-the-shelf benchmarks (like the sysbench binary) may or may not support this natively.

Related patterns

• patterns/reproducible-benchmark-publication — open-source benchmark code + public methodology so third parties can verify. The constrained-resource variant inherits this property and adds the cap + overhead-split disclosure.
• patterns/custom-benchmarking-harness — for when the benchmark code doesn't natively support the resource cap.

Seen in

• sources/2026-04-21-planetscale-taobench-running-social-media-workloads-on-planetscale — canonical instance. The 48-core cap was set by Cheng's Berkeley team; the 44+4 split was PlanetScale's application of it to their multi-tenant serverless tier. Van Dijk's disclosure of the split is the reason this pattern earns a page: "we underprovisioned the 'query path' of the cluster itself to use a maximum of 44 CPU cores out of the requested 48 maximum. The other 4 cores would be used for multi-tenant aspects of the infrastructure, such as edge load balancers."