PATTERN Cited by 1 source

Size for heaviest query, not total data

Context

Traditional analytical cluster sizing rules say "provision compute proportional to total data volume." This overestimates resource needs when the query engine can prune most data at planning time via file-level statistics.

Pattern

Size your compute cluster for the data scanned by your single heaviest analytical query, not for the total data retained. Time-bounded dashboard queries that leverage Iceberg file-level min/max statistics will be fast on a single node regardless of total corpus size. Only full-corpus scans (no selective predicate) benefit from cluster scale-out.

Mechanism

1. File-level column statistics in Iceberg manifests let the query planner skip entire data files whose value ranges don't overlap the predicate.
2. A 1 TB corpus with a 1-year time filter may read only ~1 GB (76 of 4,961 files).
3. That 1 GB query is unaffected by cluster size — latency tracks the matched slice, not the retained volume.
4. The cluster is sized for the remaining queries that must scan the full corpus.

Trade-offs

• Only works when data is ordered (or clustered) by the predicate column — interleaved data destroys pruning effectiveness.
• If your heaviest query is a full scan, you're back to scaling linearly with data volume.

Seen in

• sources/2026-07-29-redpanda-single-query-scaling-in-redpanda-sql: Redpanda explicitly proposes this as their new sizing rule. Dashboard queries stay at ~4 s from 100 GB to 1 TB; only full-scan queries need more nodes.
• Snowflake's warehouse sizing guidance follows the same logic: size for the heaviest workload, not for total stored data.