Skip to content

Redpanda — Little's Law in practice with Cloud Topics

Summary

Redpanda's 2026-05-05 post is the production-tuning sequel to the 2026-03-30 Cloud Topics architecture deep-dive. Where the architecture post described the L0-file / placeholder-batch / Reconciler primitives, this post is a performance-engineering retrospective on a production-tuning issue the team hit during internal benchmarking: substituting the per-partition Raft replication phase (~10 ms) with an object-storage upload phase up to 100× slower caused throughput to collapse on a per-connection basis, even though the substrate was correct. The fix — a single extra queueing stage in the write pipeline — is presented as a worked application of Little's Law, framed verbatim as "Throughput = Latency × Concurrency" (the post's preferred algebraic form rather than the textbook L = λW). With the queueing stage in place, OpenMessaging Benchmark "easily push[es] through to the GB/s scale we were targeting without needing to change any producer configurations." The post is also a candid retrospective on the meta-lesson: the team had been so focused on building the functionality of Cloud Topics that they had not pushed real-world workloads through it, and what could have been a serious architectural oversight became a process improvement (more emphasis on production-shape benchmarking earlier in the cycle).

This is the first wiki source ingest where Little's Law appears as the load-bearing architectural lens for a fix, not as a side-note diagnostic. It promotes Little's Law from a passing reference in concepts/queue-length-vs-wait-time / concepts/latency-rises-before-throughput-ceiling to a first-class wiki concept with its own page, and pairs it with two named patterns (patterns/concurrency-buffer-stage-for-high-latency-io and patterns/pipelined-produce-with-position-guarantee) that generalise the Cloud Topics fix beyond Redpanda.

Key takeaways

  1. Throughput = Latency × Concurrency is the load-bearing equation. The post frames Little's Law in its application-friendly form rather than the textbook L = λW. Verbatim: "the latency of the upload phase could be up to 100x slower than the replication phase. This makes it easy to feel the implication of Little's Law, which equates to Throughput = Latency * Concurrency." For a fixed pipeline structure, if latency rises by 100× and concurrency stays fixed, throughput drops 100×. The only recourse — without re-architecting the slow phase — is to raise concurrency. Canonicalised as concepts/littles-law. (Source: this post)

  2. Bottlenecks migrate as storage hardware evolves. The post's opening framing is the meta-observation that Redpanda was built "in an era of shifting bottlenecks" — first the spinning disk, then thread context switches and cache invalidations on fast NVMe (which thread-per-core architectures like Redpanda / Seastar address), and now object-storage I/O latency reintroducing high-latency I/O into systems built for low-latency assumptions. Verbatim: "Demand for low-cost storage solutions, such as cloud object storage, is reintroducing high-latency I/O into systems designed around the assumption of low-latency storage. […] Just as older systems had to confront the introduction of high-performance storage, Redpanda isn't immune either." Canonicalised as concepts/storage-bottleneck-migration. (Source: this post)

  3. Pipelined produce processing releases the next request after position is guaranteed, not after completion. The pre-existing low-latency Redpanda write pipeline already used this technique: "We observed early on that after a request's position in the pipeline has been guaranteed, all of its dependencies have been resolved, and the next queued request can be processed before previous requests have been replicated. This allows pipelined processing of produce requests and was a significant improvement over the early design." This is the pipelining technique that makes a single Kafka producer connection capable of more than 1000ms / replication_latency requests per second — without it, per-connection throughput would be capped at 1 / replication_latency. Canonicalised as patterns/pipelined-produce-with-position-guarantee. (Source: this post)

  4. Per-connection throughput is capped at 1 / replication-latency without pipelining — at 1 / upload-latency without a queue stage. Verbatim worked example: "the first two stages are extremely fast (e.g. < ~1ms). So if replication takes, for example, 10ms, then the system can only process 100 requests per second per connection." The 100 RPS-per-connection figure is the per-connection ceiling that Little's Law sets when concurrency-per-connection is 1 (one request in flight at a time). Pipelining lifts the concurrency multiplier; queueing inflates it further. With Cloud Topics' upload phase reaching 100× the replication latency, even pipelined-from-day-one Redpanda would have been pinned at ~1 RPS per connection without further intervention. (Source: this post)

  5. The Cloud Topics write path is two storage stages, not one. This is the load-bearing structural difference from low-latency topics. Verbatim: "storage in the Cloud Topics write path consists of two stages: one for metadata management and another for application payload data. This is different than standard topics where application data and metadata are stored together. In particular, the existing replication layer handles Cloud Topics metadata, while payload data is written directly into cloud object storage." The article reaffirms the Raft-metadata-plus-object-storage-data split established in the 2026-03-30 architecture post, but re-frames it as the root cause of why a new bottleneck class appeared: an additional queue and object-storage upload phase was inserted before the metadata enters the replication layer. (Source: this post)

  6. The fix: an extra queueing layer at the upload phase. The architectural shape of the fix is named explicitly: "To improve our throughput with the increased latency, we had to figure out how to increase concurrency. Just like last time, we addressed this with an additional layer of queuing. As shown in the image above, this extra queueing is introduced during the upload phase of the write path, before requests are processed by the replication layer, and helps hide the large latencies caused by cloud object storage." The pattern: when a downstream phase becomes 100× slower, insert a queue between the producer-facing ingest and the slow phase, so multiple uploads can be in flight from a single connection. Canonicalised as patterns/concurrency-buffer-stage-for-high-latency-io and concepts/queue-depth-as-latency-hiding-mechanism. (Source: this post)

  7. Order preservation is restored after the queue, not lost. This is the specific subtlety that distinguishes the Cloud Topics write pipeline from a naive pipelining implementation. Verbatim: "Once the data is uploaded, we preserve the ordering from the producer and release it into the replication layer, again waiting for the replication layer to ensure our position meets the needs of idempotent workloads. We still hold the producer acknowledgment until the metadata is fully replicated across the cluster. This means we preserve the correct ordering and data durability requirements at every stage while allowing more concurrency in the latency-bound part of request processing." The pipeline buys concurrency in the upload phase but restores order in the replication phase, and only acks the producer once the metadata is durable. Idempotent producers continue to work correctly because the position-guaranteeing replication phase remains serialised. (Source: this post)

  8. Validation substrate: OpenMessaging Benchmark, no client tuning required. Verbatim: "Running OpenMessaging Benchmark with this change was the key to unlocking throughput, and allowed us to easily push through to the GB/s scale we were targeting without needing to change any producer configurations." This is the deployability win — the broker-side queue stage means no linger.ms / batch.size / max.in.flight.requests.per.connection tuning is required on the producer side to hit GB/s. Cf. OMB is the same benchmark substrate the [[sources/2025-02-11-redpanda-high-availability-deployment-multi-region-stretch-clusters|2025-02-11 stretch-cluster post]] used; canonicalises Redpanda's pattern of validating new substrate features through OMB before publishing first-party throughput numbers. (Source: this post)

  9. The retrospective lesson: prioritise real-world workload testing over functional implementation. Verbatim: "When we discovered this bottleneck in our performance testing a few months ago, we realized we had been so focused on building functionality that we hadn't been focused on pushing real-world workloads through the system. Thanks to the entire Cloud Topics team, our carefully thought-out implementation readily accepted the fixes we needed, and what could have been a serious architectural oversight became an insightful process." This is a candid post-hoc disclosure: the initial Cloud Topics implementation (described in the 2026-03-30 architecture post) would have been throughput-pinned without the extra queue stage, and only an OMB run discovered it. The framing — "a serious architectural oversight became an insightful process" — is the transferrable lesson for any team substituting a slow-storage substrate behind a previously fast-storage abstraction. (Source: this post)

Architectural shape

Pre-Cloud-Topics: low-latency Redpanda write pipeline

producer ──► ingest checks (~<1ms)
              (validation, idempotency)
                       │
                       ▼
              ordering reservation
              (next-in-pipeline released
               once position guaranteed)
                       │
                       ▼
              replication layer (~10ms)
              (Raft replication of
               data + metadata, NVMe)
                       │
                       ▼
              ack producer once durable

Per-connection throughput ceiling without pipelining: 1 / replication_latency = 1 / 10ms = 100 RPS.

With pipelining (next request released after position guarantee, before completion): per-connection throughput rises to whatever the pipeline depth allows; observed RPS becomes network-bandwidth-bound rather than latency-bound.

Cloud Topics write pipeline (initial — broken)

producer ──► ingest checks (~<1ms)
                       │
                       ▼
              upload to object storage (up to ~1000ms)
              (no extra queueing — single in-flight upload
               per request, behaves like the old replication
               phase but 100× slower)
                       │
                       ▼
              replication layer for metadata only (~10ms)
                       │
                       ▼
              ack producer

Per-connection ceiling collapses to 1 / 1000ms = 1 RPS (under worst-case object-storage latency). At GB/s targets this requires thousands of concurrent connections — which client configurations do not deliver by default.

Cloud Topics write pipeline (fixed — Little's Law applied)

producer ──► ingest checks (~<1ms)
                       │
                       ▼
              ┌────────── upload queue ──────────┐
              │  (depth N — multiple uploads      │
              │   in flight from one producer)    │
              └───────────────┬───────────────────┘
                              │ (parallel uploads)
                              ▼
              upload to object storage (up to ~1000ms each,
              N concurrent → effective per-connection
              throughput ≈ N × (1 / upload_latency))
                              │
                              ▼
              ordering restored
              (producer order preserved when
               releasing into metadata replication)
                              │
                              ▼
              replication layer for metadata only (~10ms)
                              │
                              ▼
              ack producer once metadata durable

This is the Little's Law fix: with concurrency N ≈ 100×, the per-connection upload throughput recovers what the upload latency spent. Order is restored before metadata replication, so idempotent producers continue to work. The producer's ack is held until the metadata is durable, preserving Kafka's acks=all durability contract. No producer-side configuration change required.

Operational numbers

Property Value Notes
Pre-Cloud-Topics replication latency (low-latency NVMe path) ~10 ms First-party first-principles example, not measured average
Pre-Cloud-Topics ingest-check + ordering latency < ~1 ms First-party characterisation
Cloud Topics object-storage upload latency up to 100× slower than replication First-party — implies upload latencies up to ~1000 ms in worst case
Per-connection RPS ceiling without pipelining (replication-only path) 100 RPS Worked example: 1000 ms / 10 ms
Per-connection RPS ceiling without pipelining (Cloud Topics initial) ~1 RPS Worked example: 1000 ms / 1000 ms
Validated post-fix throughput target GB/s scale Validated via OpenMessaging Benchmark
Required producer configuration changes for the fix none "without needing to change any producer configurations"

The numbers are illustrative first-principles examples ("e.g.") not measured benchmarks. The team was deliberate not to publish production-shape latency / throughput tables in this retrospective, positioning it as a methodology post rather than a benchmark post.

Caveats

  1. No measured latency / throughput tables published. The ~10 ms replication latency, ~1 ms ingest-check latency, and 100× upload-latency multiplier are first-principles worked examples, not measured first-party numbers. The post does not disclose the actual upload-phase p50 / p99 latency, the chosen queue depth N, or the absolute GB/s number achieved at the validated target. Treat the per-connection 100 RPS / 1 RPS ceilings as illustrative, not as Cloud Topics SLOs.

  2. Object-storage provider not disclosed for the upload-latency range. The article cites "cloud object storage" generically; it does not break out p99 numbers per AWS S3 / GCS / ADLS, nor account for whether the 100× figure includes warm-up, throttling, or PUT-amplification effects. Cf. the small-file problem mitigations described in the architecture post (0.25 s / 4 MB cross-partition batching) influence both the upload-latency curve and the concurrency required.

  3. The queue-depth N is not disclosed. A reader can infer "some N at least near the latency multiplier" from Little's Law, but the article does not publish the chosen depth, nor the knobs Redpanda exposes (if any) for tuning it per-cluster. The silence is consistent with the broker-internal-only nature of the fix — the explicit virtue of the design is that customers don't tune it.

  4. The post does not address back-pressure beyond the queue. What happens when the queue saturates is not described; in principle a sufficiently fast producer could still pin the queue, after which back-pressure must propagate. Whether that propagation closes at the ingest-check stage, blocks at the producer's max.in.flight.requests.per.connection, or surfaces as a buffer-full error is not stated.

  5. Idempotency-preserving order restoration is asserted, not proven. The post claims the post-queue ordering preservation "meets the needs of idempotent workloads", but the actual merge / sequence-number scheme used to re-establish producer order after concurrent uploads is not described. For architecture-curious readers this is the most interesting missing detail — the article gives the what but not the how.

  6. No comparison to alternative architectures. The article does not compare to WarpStream's object-store- native model, to Kora's tiered model, or to direct-to-S3 designs that bypass a broker queue. Reading this article alongside the sources/2026-03-30-redpanda-under-the-hood-redpanda-cloud-topics-architecture|2026-03-30 architecture post gives the most-complete picture; the comparison table on systems/redpanda-cloud-topics holds the cross-system context.

  7. OMB validation methodology not detailed. The article mentions running OpenMessaging Benchmark but does not disclose the workload YAML, partition counts, message sizes, producer / consumer fan-out, or warm-up duration. Compare to the [[sources/2025-02-11-redpanda-high-availability-deployment-multi-region-stretch-clusters|2025-02-11 stretch-cluster post]] which discloses the OMB driver config in full — that level of disclosure is absent here, again consistent with the post's positioning as a methodology / lesson-learned piece rather than a performance-claim piece.

Source

Systems

Concepts

Patterns

Companies

Last updated · 542 distilled / 1,571 read