Skip to content

Single-query scaling in Redpanda SQL on 1TB

Summary

Redpanda benchmarks single-query scaling of Redpanda SQL — its Postgres-compatible analytical engine that joins live streaming topics with historical Iceberg tables — against a 100 GB–1 TB NYC Taxi dataset (0.61–6.07 billion rows). The key finding: query cost tracks data scanned, not data retained, because Iceberg column statistics enable file-level pruning at planning time. For full-corpus scan queries, adding nodes provides near-linear speedup (7× on 8 nodes for a high-cardinality GROUP BY on 1 TB).

Key takeaways

  1. Predicate pushdown via Iceberg statistics keeps latency constant regardless of retention: A time-bounded query took ~4–5 s whether the total corpus was 100 GB or 1 TB; the planner pruned thousands of non-overlapping files cheaply by reading per-file min/max statistics from Iceberg manifests (Source: Result 1).

  2. Full-scan queries scale near-linearly with node count: A high-cardinality GROUP BY dropped from 298 s on 1 node to 43 s on 8 nodes (7× speedup at 1 TB) — close to ideal for scan-dominated workloads (Source: Result 2).

  3. Memory is partitioned across nodes with no spill-to-disk: Per-node peak RSS drops roughly as total-state ÷ nodes, but skew means you must size for the busiest node. Redpanda SQL has no disk spill — exceeding RAM is a hard failure (Source: Result 3).

  4. Architecture is shared-nothing compute over shared object storage: Adding nodes redistributes scan and aggregation; no data migration is needed. Nodes are stateless readers of Parquet from S3 (Source: architecture section).

  5. The sizing rule changes: Size compute for the heaviest single query scan, not for total retained data volume. Time-bounded dashboard queries are fast on one node because Iceberg pruning dominates latency (Source: What this means for sizing).

  6. Amdahl's law applies: Once pruning cuts the read to ~1 GB, there's almost nothing left to parallelize — the pruned query barely moves across 1–8 nodes (4.8 s → 3.3 s) (Source: Result 2, pushdown row).

  7. Temporal ordering in data is critical: File-level pruning works because records were produced in pickup_datetime order, so each data file spans a narrow time range. Interleaved loading destroys pruning effectiveness (Source: caveat in Result 1).

  8. Join/window queries not yet characterized: The near-linear scaling claim is scoped to scans and aggregations — join and window queries shuffle more data and may scale less cleanly (Source: scope note in Result 2).

Operational numbers

Metric Value
Dataset size 100 GB – 1 TB (0.61 B – 6.07 B rows)
Time-bounded query (1 TB) ~3.9 s (scans ~1.1 GB from 76 of 4,961 files)
Full scan (1 TB, 1 node) 181 s
Heavy GROUP BY (1 TB, 1 node → 8 nodes) 298 s → 43 s (7× speedup)
Light GROUP BY (1 TB, 1 node → 8 nodes) 205 s → 25 s (~8× speedup)
Node type m7i.2xlarge (8 vCPU / 32 GiB)
Per-node peak RSS (heavy rollup, 8 nodes) ~2.9 GB
Per-node peak RSS (heavy rollup, 2 nodes) ~7.1 GB

Caveats

  • Dataset above 100 GB is built by time-shifted replication — skew distribution is real but total volume is extrapolated.
  • All times are warm (OS/storage caches populated); cold-start latency is out of scope.
  • Single-query strong scaling only; throughput (concurrent queries) scaling deferred to follow-up.
  • No spill-to-disk: OOM under high-cardinality GROUP BY on under-provisioned clusters is a hard failure.

Source

Last updated · 604 distilled / 1,840 read