Skip to content

META 2026-10-12

Read original ↗

Turning Opportunistic AI Inference Capacity Into Better Ads Recommendations

Summary

Meta describes a production system that harvests idle inference capacity across its fleet into a shared opportunistic capacity pool and spends it on better ads recommendations — serving more relevant ads and improving advertiser outcomes without ordering additional capacity. The central observation is the inference capacity paradox: a fleet can be globally constrained while meaningful pockets of capacity sit locally idle (peak-provisioning vs demand variation, model-development vs capacity-delivery timing, regional resource-ratio mismatch, accelerator-type mismatch). This idle capacity is preemptible, highly volatile, and carries no long-term guarantees, so it can only be consumed safely by a workload with three structural properties: high elasticity, sustained quality improvement across a wide capacity range, and limited cross-service dependencies — properties the ads ranking workload already has. Two complementary elasticity levers expand ads work onto opportunistic capacity: Rank More Ads (RMA) increases the number of candidates evaluated by the ranking funnel (needs both CPU and accelerator surplus), and Serve Better Models (SBM) raises the share of requests routed to a more powerful model (needs accelerator-only surplus). Baseline (dedicated-capacity) serving is protected by physical isolation (SBM: separate routing targets/replicas) and logical isolation (RMA: a global candidate-volume budget), two independent feedback loops (a central PID controller for RMA, an explore-and-exploit value-aware router for SBM), and multi-layer value-aware load shedding at both server and client that reacts in milliseconds when opportunistic capacity is suddenly reclaimed.

Key takeaways

  1. The inference capacity paradox is the motivating framing. "Inference fleets face a capacity paradox: They can be globally constrained while meaningful pockets of capacity remain locally idle." Four structural causes: peak-provisioning/demand-variation mismatch, model-development/capacity-delivery timing mismatch, regional resource-ratio mismatch (accelerators idle for want of supporting CPU, or vice versa), and accelerator-type mismatch (workloads not portable across vendors/generations strand capacity on hardware that can't serve current demand). (Source: this article)

  2. Opportunistic capacity has three defining operational constraints that make it hard to use: it is preemptible (reclaimed at any time), highly volatile in availability, and carries no long-term guarantees. A workload can only safely consume it if it exhibits (a) high workload elasticity (add/remove large amounts of optional work quickly without affecting baseline), (b) sustained quality at scale (keeps meaningfully improving output across the range the pool can supply — which also tends to mean it already uses a lot of dedicated capacity), and (c) limited cross-service dependencies (self-contained, so fewer upstream/downstream systems must react to volatile capacity). Ads ranking satisfies all three. (Source: this article)

  3. Two complementary levers consume different resource mixes. Rank More Ads (RMA) widens the retrieval → ranking funnel — evaluating more candidates increases the chance of surfacing a relevant ad — but the extra candidates traverse CPU-heavy upstream (retrieval, feature processing) and accelerator-heavy inference, so RMA needs both CPU and accelerator surplus. Serve Better Models (SBM) keeps the candidate count constant and instead raises the fraction of requests scored by a more powerful model, so upstream CPU demand is stable — letting SBM consume accelerator-only surplus that RMA cannot. "Together, they can use a broader range of opportunistic capacity than either lever alone." The allocation between RMA and SBM is periodically re-tuned by measured performance and resource requirements. (Source: this article)

  4. Baseline traffic is protected by two different isolation mechanisms. For SBM the isolation is physical: baseline and more-powerful models use separate routing targets; dedicated replicas are sized for the baseline model, replicas on opportunistic capacity serve the more-powerful model; when opportunistic capacity contracts, requests fall back to the baseline model "whose capacity is already provisioned for them." For RMA the isolation is logical: the system computes how much extra candidate-ranking work the opportunistic replicas can support and sets a global budget; each replica mixes baseline and additional work, and baseline traffic is protected by capping total incremental work — not by physically separating individual requests. In both cases contraction removes optional work rather than letting it spill onto baseline capacity. (Source: this article)

  5. RMA runs a closed PID feedback loop on tail latency. A central proportional-integral-derivative (PID) controller reads observed tail latency and computes a target number of ads to send to downstream ranking; clients enforce that target on new requests. "When latency is below the target, the controller increases the number of ads ranked. When latency rises, it reduces the number." The latency signal is influenced by the capacity situation, so the loop converges on a sustainable candidate volume as conditions change — a textbook use of PID with latency as process variable and candidate volume as actuator. (Source: this article)

  6. SBM routes by request value with an explore-and-exploit loop. A central controller sets the target coverage for the more-powerful model; a client-side model router decides which eligible requests use it. The router groups requests into value segments by predicted likelihood of user engagement, and an explore-and-exploit process periodically measures the incremental benefit of the powerful model per segment and updates the allocation — "necessary because the incremental benefit of the more powerful model does not always align with the absolute value of a request." Exploratory traffic is preserved so the system keeps collecting observations rather than reinforcing a stale routing decision. A calibration loop keeps the realized traffic share close to target as the mix shifts across regions and time. (Source: this article)

  7. Multi-layer value-aware load shedding bridges the gap the slow control loops can't. The RMA/SBM central controllers operate "on the order of minutes", but a sudden preemption can remove capacity before clients get new targets. Two faster layers absorb it: server-side shedding — when the opportunistic pool loses capacity before clients' view catches up, servers immediately drop lower-value opportunistic work (RMA reduces candidates processed within a request; SBM applies an equivalent value-aware shedding policy); and client-side shedding — clients also observe server status directly and, on observed overload, shed lower-value opportunistic work themselves. "Both forms of load shedding react within a few milliseconds, bridging the interval before the controller-client loops converge." Both use value signals when choosing what to drop, minimizing quality impact during the transition. This is graceful degradation of optional work with value-aware prioritization. (Source: this article)

  8. The design invariant is one-directional. "Losing opportunistic capacity may reduce incremental ranking work, but it must not overload the capacity that serves the baseline workload." Every mechanism — isolation, feedback loops, load shedding — exists to keep baseline latency and reliability within target when the volatile pool evaporates. (Source: this article)

Systems / concepts / patterns extracted

  • Opportunistic / preemptible capacity harvesting — new concept page. Aggregating locally-idle, reclaim-at-any-time fleet capacity into a shared pool and spending it only on optional, elastic work that degrades gracefully when the pool shrinks. (The AWS-Spot / GCP-preemptible / Borg-opportunistic-tier idea, applied here to an inference fleet.)
  • Elasticity — the three structural workload properties (high elasticity, sustained quality at scale, limited cross-service dependencies) that qualify a workload to consume volatile capacity. Elastic optional work expands/contracts with the pool. Enriches concepts/elasticity.
  • PID controller — RMA's central controller: tail latency as process variable, candidate volume as actuator. Enriches concepts/pid-controller.
  • Graceful degradation — value-aware, multi-layer load shedding that drops the lowest-value optional work (never baseline) when capacity is reclaimed. Enriches concepts/graceful-degradation.
  • Retrieval → ranking funnel — RMA widens the funnel by evaluating more candidates when capacity allows (recall-shaping at the funnel's top). Enriches concepts/retrieval-ranking-funnel.
  • Latency-critical vs latency-tolerant workload — the baseline-vs-optional split is a value/priority tiering that maps onto the critical-vs-best-effort distinction: baseline work is protected; opportunistic work is best-effort and shed first. Enriches concepts/latency-critical-vs-latency-tolerant-workload.
  • Backpressure / load shedding — server-side + client-side value-aware shedding as the millisecond-scale boundary response to preemption. Enriches concepts/backpressure.
  • Bin-packing (tag) — the regional resource-ratio mismatch (accelerators idle for want of CPU in the right proportion) is a packing-inefficiency statement; recorded as a tag/prose, not a new page.
  • meta-adaptive-ranking-model, wukong-turbo (systems frontmatter, plain-text) — the LLM-scale ads-ranking serving stack this capacity work sits on top of; named for context, no new page.
  • Not minted (recorded as prose/tags per the taxonomy gate): rank-more-ads (RMA), serve-better-models (SBM), value-aware-load-shedding, physical-vs-logical-isolation-for-optional-work, explore-and-exploit-value-routing, precomputation-elasticity — all article-specific mechanisms or sub-ideas of existing canonical pages (elasticity, graceful-degradation, pid-controller, retrieval-ranking-funnel). value-aware-load-shedding is a Lint-promotion candidate if a second source names it.

Operational numbers / specifics

The post is architecture-overview voice and discloses no absolute numbers — no fleet size, no QPS, no pool size, no latency targets, no measured advertiser-value or capacity-efficiency lift, no accelerator vendor/model mix. Qualitative/structural specifics only:

  • Candidate selection: "select a small number of ads from millions of candidates in less than a second" across a sequence of ranking stages (each later stage scores only the top candidates of the previous).
  • Central RMA/SBM control loops operate "on the order of minutes"; server-side and client-side load shedding react "within a few milliseconds."
  • RMA requires CPU + accelerator surplus; SBM can use accelerator-only surplus.
  • SBM physical isolation = separate routing targets + dedicated-vs-opportunistic replicas; RMA logical isolation = global candidate-volume budget shared across mixed replicas.

Caveats

  • Architecture-overview voice: no production metrics, no system name, no hardware disclosure, no A/B lift. The RMA/SBM mechanisms are described structurally, not with controller gains, segment counts, shedding thresholds, or pool-churn rates.
  • The "more powerful model" and "baseline model" are abstract — no model identities, sizes, or quality deltas given.
  • Future work named: broadening the operating range of RMA/SBM, new levers such as precomputation elasticity (work done ahead of request time, less constrained by live-traffic location/timing), expanding model portability + cross-region inference to grow the usable pool, and an optimization-based system to automatically allocate opportunistic capacity across levers by user/advertiser benefit under regional / hardware-compatibility / durability constraints.

Source

Last updated · 854 distilled / 2,448 read