Skip to content

META

Read original ↗

Modernizing the Meta Ads Service With an Open-Source Kernel Scheduler

Summary

Meta's Ads and Linux Kernel teams describe how they used sched_ext — the upstream, BPF-based extensible scheduling framework (merged in Linux kernel v6.12) — to build a custom scheduling policy tailored to the ads delivery workload. The policy soft-partitions CPUs into two pools (latency-critical request-path threads vs. background threads), dynamically adjusts pool sizes via load-based heuristics, and improves L3 cache locality by keeping related work on the same CPUs. The result: a 28% reduction in ads retrieval p99 latency, 3.28 MW power savings across the fleet, and a +1.1% increase in weighted-ads-ranked. Two follow-on policy updates delivered an additional 60% p99 latency reduction and 18% fewer timeout errors — all shipped as userspace BPF program restarts in days rather than kernel releases spanning months.

Key Takeaways

  1. Kernel upgrade caused latency regression: Upgrading from Linux 6.4 (CFS) to 6.9 (EEVDF scheduler) introduced a latency regression in ads serving that reduced ads ranked per response, forcing a subset of hosts to remain on the older kernel — creating technical debt and operational fragmentation.

  2. sched_ext as regression fix and optimization platform: What started as a targeted fix for the EEVDF regression became a continuous-optimization platform. sched_ext's BPF-program-as-scheduler-policy model turned scheduling from a monolithic kernel concern into an iteratable, per-workload concern.

  3. Domain knowledge encoded into the scheduler: The policy encodes which threads are on the latency-critical request path and which are background work. General-purpose schedulers (CFS, EEVDF) balance threads with no understanding of workload semantics — sched_ext closes that gap.

  4. Soft CPU partitioning improves cache locality: By keeping latency-critical threads on a dedicated pool of CPUs, the policy improves last-level cache (L3) locality and reduces costly DRAM accesses. Pool sizes adjust dynamically via load heuristics.

  5. Userspace deployment model transforms iteration velocity: The policy is a user-space binary that loads the BPF program. Rolling out a change = restarting the scheduler process to unload old / load new policy — no kernel rebuild, no reboot, no fleet-wide kernel upgrade. Iteration cadence: days rather than months.

  6. Compounding improvements without kernel dependency: Two follow-on scheduler-policy updates shipped purely in user space, extending the initial win to a cumulative ~88% p99 reduction (28% initial + 60% follow-on) and 18% fewer timeout errors on the critical path.

  7. Open-source ecosystem benefit: sched_ext was upstreamed into Linux v6.12; Meta co-developed it with Google's ghOSt team. Any operator with a workload that doesn't fit general-purpose scheduling can now ship workload-specific policies without forking the kernel.

  8. Scale context: Meta's ads serving fleet handles >5 million requests/second at the serving platform entry point (>400 billion/day across all monetized surfaces). A few milliseconds of latency degradation has significant business impact.

Operational Numbers

Metric Value
Ads platform entry-point QPS >5M rps (>400B/day)
p99 latency reduction (initial launch) 28%
p99 latency reduction (follow-on updates) additional 60%
Timeout error reduction 18%
Fleet power savings 3.28 MW
Weighted-ads-ranked improvement +1.1%
Kernel version (old) 6.4 (CFS)
Kernel version (new) 6.9 (EEVDF)
sched_ext upstream version Linux v6.12

Caveats

  • Architecture-overview voice — no fleet size, host count, CPU vendor/model, or per-host QPS disclosed.
  • No detail on how threads are classified as latency-critical vs. background (likely a cgroup or thread-naming convention, undisclosed).
  • Load-based heuristics for pool-size adjustment not described beyond "dynamic."
  • No comparison against alternative scheduling approaches (e.g. cpuset pinning, cgroup CPU controllers).
  • Acknowledgements name the teams but no code or scheduler-policy implementation details are shared (open-source framework, but Meta's ads-specific policy is presumably internal).

Source

Last updated · 585 distilled / 1,765 read