Skip to content

CONCEPT Cited by 3 sources

Opportunistic capacity harvesting

Definition

Opportunistic capacity harvesting is the practice of aggregating locally-idle, reclaim-at-any-time capacity across a fleet into a shared pool and spending it only on optional, elastic work that can expand and contract with the pool and degrade gracefully the instant the capacity is taken back. The harvested capacity is preemptible (can be reclaimed with little notice), highly volatile (availability fluctuates with the baseline demand it is borrowed from), and carries no long-term guarantees — so it must never be relied on for work whose failure is user-visible.

It is the generalization of a family of industry primitives that all share the same shape: AWS Spot instances, GCP preemptible / Spot VMs, Borg's best-effort / opportunistic allocation tier, and Meta's inference-fleet opportunistic pool. The common bet is that the spare in a fleet provisioned for peak demand is large in aggregate, and can be put to productive use if — and only if — the consuming workload is built to lose it at any moment.

The capacity paradox that motivates it

The forcing observation (Meta's framing) is that a fleet can be globally constrained while meaningful pockets of capacity remain locally idle. Common structural causes:

  • Peak-provisioning vs demand-variation mismatch — capacity sized for peak sits idle off-peak and off-season.
  • Supply-timeline vs model/launch-timeline mismatch — capacity arrives on supply-chain schedules; the workloads that will consume it are developed and launched on independent schedules, leaving a gap.
  • Regional resource-ratio mismatch — a workload needs accelerators and supporting CPU in a fixed proportion; when the ratio is off in a region, one resource strands the other (a bin-packing inefficiency).
  • Accelerator-type mismatch — a heterogeneous fleet spans vendors / generations / architectures, and a workload that isn't portable across them strands capacity on hardware that can't serve current demand.

At scale these transient pockets sum to substantial capacity — enough to be worth harvesting if it can be used safely.

What makes a workload safe to run on it

Harvested capacity can only be consumed without compromising the baseline workload if the consuming work has three structural properties (Source: sources/2026-10-12-meta-turning-opportunistic-ai-inference-capacity-into-better-ads-recommendations):

  1. High workload elasticity — it can add or remove a large amount of optional work quickly as the pool changes, without affecting baseline expectations.
  2. Sustained quality at scale — it keeps meaningfully improving output across the whole range the pool can supply (a workload that plateaus can't absorb the surplus usefully). This tends to correlate with workloads that already consume a lot of dedicated capacity.
  3. Limited cross-service dependencies — a self-contained workload has fewer upstream/downstream systems that must react when the volatile capacity appears and disappears.

Recommendation / ads ranking is the canonical fit: more compute reliably buys more relevant results, the work is self-contained, and it is trivially elastic (rank more candidates, or route to a bigger model).

The design invariant

The load-bearing rule is one-directional: losing opportunistic capacity may reduce the optional work, but it must never overload the capacity reserved for the baseline workload. Everything below exists to enforce that invariant when the volatile pool evaporates faster than the control loops can react.

Mechanisms

Harvesting opportunistic capacity safely is an exercise in isolation + feedback control + fast load shedding:

  • Isolate baseline from optional work. Two shapes appear in Meta's system: physical isolation (optional work runs on separate routing targets / replicas, so contraction simply stops routing to them and falls back to already-provisioned baseline capacity) and logical isolation (optional work shares replicas with baseline but is capped by a global budget sized to what the opportunistic replicas can absorb, so baseline is protected by the cap rather than by physical separation).
  • Close a feedback loop per lever. A control loop expands optional work when a capacity-sensitive signal (e.g. tail latency) has headroom and contracts it when the signal tightens — e.g. a PID controller driving candidate volume against a tail-latency setpoint.
  • Shed value-aware under sudden preemption. Central control loops run on the order of minutes; a preemption can remove capacity in milliseconds. Server-side and client-side value-aware load shedding drop the lowest-value optional work immediately (never baseline), bridging the gap until the slow loops reconverge. This is graceful degradation applied to the optional tier, prioritized by request value.

Relationship to adjacent concepts

  • concepts/elasticity — opportunistic harvesting is elasticity pointed at a volatile, preemptible substrate rather than an autoscaled dedicated one. The elasticity levers (scale optional work up/down) are the same; the substrate's lack of guarantees is what forces the isolation + shedding discipline.
  • concepts/latency-critical-vs-latency-tolerant-workload — the baseline-vs-opportunistic split is a priority/criticality tier: baseline work is protected and never shed; opportunistic work is best-effort and shed first. Harvesting is the capacity-side realization of running two priority classes on one fleet.
  • concepts/fair-sharing — fair-sharing distributes idle capacity among tenants by weight; opportunistic harvesting is the single-consumer case where one elastic workload soaks up whatever idle capacity exists, under a strict "give it back on demand" contract.
  • concepts/bin-packing — regional resource-ratio and accelerator-type mismatches that create idle pockets are packing failures; harvesting is how a workload reclaims value from the slack a perfect packer would have eliminated.

Seen in

  • sources/2026-10-29-meta-catching-bugs-before-they-land-jittestings-human-on-the-loop — LLM test generation as a spare-capacity batch job. Meta's Catching JiTTests "runs overnight on spare capacity" against the previous day's risk-ranked diffs. The workload is latency-tolerant (results are needed before land, not at submit time), so it can use idle capacity instead of sitting on the developer's critical path.

  • sources/2026-10-12-meta-turning-opportunistic-ai-inference-capacity-into-better-ads-recommendations — canonical wiki source. Meta aggregates locally-idle inference capacity across its fleet into a shared opportunistic pool and spends it on better ads recommendations with no additional capacity orders. Two elasticity levers (Rank More Ads — needs CPU + accelerator surplus; Serve Better Models — uses accelerator-only surplus) expand ads work onto the pool; physical + logical isolation protect baseline serving; a PID controller (RMA) and an explore-and-exploit value router (SBM) are the two feedback loops; multi-layer value-aware load shedding reacts in milliseconds to preemption. The post names the four causes of the inference capacity paradox and the three workload properties (elasticity, sustained-quality-at-scale, limited dependencies) required to consume volatile capacity safely.

Merged aliases

  • preemptible-capacity
  • spot-capacity
  • idle-capacity-harvesting
  • best-effort-capacity-tier

Manta: Spot for latency-tolerant ML jobs (2026-10-27)

A cloud-customer-side instance: Manta runs its less-urgent ETL/training jobs on EC2 Spot via dedicated per-instance-type ECS capacity providers, keeping time-sensitive voyage optimizations On-Demand. Interruption handling is not described. (Source: Manta on AWS)

Last updated · 854 distilled / 2,448 read