Skip to content

CONCEPT Cited by 1 source

Buddy allocator shard grouping

Definition

Buddy allocator shard grouping is the adaptation of the classic OS buddy-allocator algorithm (power-of-two memory block splitting/merging) to the problem of dynamically adjusting upload parallelism across CPU shards in a shared-nothing, thread-per-core system.

Instead of memory blocks, the "resource" being managed is a pool of CPU shards that collectively feed an upload pipeline. Shards are organized into power-of-two-sized groups. Each group uploads independently; within a group, shards take turns via round-robin.

Key property: local-only decisions

The defining advantage over centralized coordination is that split and merge decisions require only local state inspection:

  • A group leader checks its own group's batcher backlog (atomic counter read).
  • It checks its buddy group's backlog (another atomic counter read).
  • If its own backlog exceeds a threshold → split the group in half.
  • If both its own and its buddy's backlogs are empty → merge (only the lower-numbered group can initiate, preventing races).

No global locks, no coordinator shard, no cross-shard synchronization during the upload itself.

Origin

The buddy allocator was originally designed for OS memory management (and is used in Linux and in Seastar itself). Redpanda adapted the algorithm's local-decision property to solve a scheduling problem: how many upload streams should run in parallel when the optimal answer changes continuously with load (Source: sources/2026-06-18-redpanda-adaptive-write-request-scheduling).

Seen in

Last updated · 542 distilled / 1,571 read