PATTERN Cited by 1 source

Scale out for scan-dominated aggregation

Context

Analytical queries that must read the entire dataset (no selective predicate) are bottlenecked on read throughput and aggregate computation, not coordination. In shared-nothing architectures where compute reads from shared object storage, adding nodes directly increases scan parallelism.

Pattern

For full-corpus queries (GROUP BY, COUNT, SUM without a selective WHERE clause), scale out by adding compute nodes. In a shared-nothing architecture over object storage, each additional node scans a proportional slice of the data and performs local partial aggregation. The final merge step is lightweight relative to the scan, so strong scaling approaches linear up to moderate node counts.

Mechanism

1. Each node is assigned a subset of Parquet data files from the Iceberg manifest.
2. Each node scans its files and builds a local partial hash table.
3. A final merge aggregates the per-node results.
4. Per-node peak memory decreases as the hash table is partitioned across more nodes.

Constraints

• No spill to disk: If the hash table exceeds per-node RAM, the query fails rather than slowing down. Skewed keys can overload individual nodes.
• Coordination fraction grows with joins: Shuffle-heavy queries (joins, window functions) scale less cleanly because inter-node communication increases.
• Diminishing returns as node count approaches the number of data files (nothing left to split).

Seen in

• sources/2026-07-29-redpanda-single-query-scaling-in-redpanda-sql: Heavy GROUP BY on 6.07 B rows drops from 298 s (1 node) to 43 s (8 nodes) — 7× speedup. Per-node RSS drops from ~7.1 GB to ~2.9 GB.