Skip to content

PATTERN Cited by 1 source

Concurrency buffer stage for high-latency I/O

Definition

A concurrency buffer stage is a queueing stage inserted into a write pipeline immediately upstream of a high-latency I/O step, sized so that multiple operations can be in flight against the slow stage simultaneously without raising the producer-facing in-flight cap. The pattern recovers per-connection throughput that would otherwise be capped at 1 / latency by Little's Law, without requiring client-side configuration changes.

The pattern is distinct from a burstiness-absorbing queue (which sizes for traffic spikes against a steady service rate). Its purpose is latency hiding — the queue's depth is deliberately set so that items spend ~stage_latency worth of time in it, providing the in-flight concurrency the slow stage needs to deliver target throughput. See concepts/queue-depth-as-latency-hiding-mechanism for the conceptual underpinnings.

When to use

Apply this pattern when all of the following hold:

  1. A pipeline stage's latency has risen substantially (≥10×) due to a substrate change (e.g. local disk → object storage; in-region → cross-region replication; in-process → RPC).
  2. Per-connection throughput on the slow stage now caps the producer at 1 / latency, which is below your target.
  3. The slow stage can process multiple operations concurrently — it has spare capacity, just per-operation latency is high (e.g. a single S3 client can have multiple PUTs in flight; a single network connection can have multiple RPCs in flight via pipelining).
  4. The client API contract is something you want to keep stable — you don't want to push the latency increase up to the producer's max.in.flight.requests.per.connection, linger.ms, or buffer.memory settings.
  5. Strict producer-order guarantees can be restored downstream of the queue; the slow stage need not preserve order itself.

If any of these break, this pattern is wrong. If (1) is wrong, you don't have a latency problem yet. If (3) is wrong, the slow stage is intrinsically serialised and queueing won't help — you need to parallelise the stage itself or batch within it. If (5) is wrong (strict order at the slow stage is contractually required), the queue can't reorder, so you're stuck at 1 / latency per partition; you need to scale by adding partitions, not by adding concurrency.

Canonical instance

Redpanda — Little's Law in practice with Cloud Topics (2026-05-05).

The Cloud Topics write pipeline introduced an upload phase between the producer-facing ingest checks and the existing metadata-replication layer. Object-storage upload latency was up to 100× slower than the prior NVMe replication latency — the write pipeline had been designed for the latter, and so the upload stage was implicitly serialised on a per-connection basis.

The fix:

"To improve our throughput with the increased latency, we had to figure out how to increase concurrency. […] we addressed this with an additional layer of queuing. […] this extra queueing is introduced during the upload phase of the write path, before requests are processed by the replication layer, and helps hide the large latencies caused by cloud object storage."

"The additional queue becomes a mechanism for the producer and networking layers to release more batches into the system. And, with more requests queued in this new layer, we can upload more batches from a single producer in parallel."

"Once the data is uploaded, we preserve the ordering from the producer and release it into the replication layer […]. We still hold the producer acknowledgment until the metadata is fully replicated across the cluster. This means we preserve the correct ordering and data durability requirements at every stage while allowing more concurrency in the latency-bound part of request processing."

Validation: OpenMessaging Benchmark hit GB/s scale "without needing to change any producer configurations."

The pipeline shape

producer ──► ingest checks (~<1ms)
                       │
                       ▼
              ┌────────── concurrency buffer queue ──────────┐
              │ (depth N ≈ target_throughput × stage_latency  │
              │  via Little's Law — in Cloud Topics' case      │
              │  ~100× the depth of the prior pipeline)        │
              └────────────────────┬──────────────────────────┘
                                   │ (multiple ops in flight)
                                   ▼
              high-latency I/O stage (object-storage PUT,
              cross-region RPC, etc — N concurrent ops)
                                   │
                                   ▼
              order restoration / merge step
              (producer-order is reconstructed
               using sequence numbers / similar)
                                   │
                                   ▼
              downstream serial stage (metadata replication,
              durability commit) — order-preserved
                                   │
                                   ▼
              ack producer once durable

Three components compose the pattern:

  1. Buffer queue sized via Little's Law to absorb the target_throughput × stage_latency product.
  2. Concurrent slow stage that can process N items in parallel.
  3. Order-restoration step post-stage that reorders completed items into producer order before any downstream order-sensitive step.

Why broker-side, not client-side

The pattern lives inside the broker, not in the client. The client's view is unchanged: it submits requests with the same contract (acks=all, idempotent producer semantics, in-flight cap) and gets the same ack semantics back. The substrate change is absorbed by the broker.

The alternative — pushing the concurrency increase up to the client — has three failure modes:

  1. Configuration burden across the entire client fleet. Every producer would need to raise max.in.flight.requests.per.connection (and idempotency settings to compensate). Doing this fleet-wide is operationally expensive and error-prone.
  2. Memory pressure on every client. Concurrency on the producer side requires per-client buffer memory. A broker-side queue pools the memory cost across the whole connected fleet.
  3. Loss of substrate fungibility. Once the client knows about the concurrency requirement, switching back to a low-latency substrate doesn't reduce the configuration. The broker-side queue is invisible to the client and can be tuned per-substrate without API churn.

The Redpanda team's framing is unambiguous on this: "without needing to change any producer configurations" is the deployability property the pattern delivers.

Sizing the queue depth

From Little's Law:

required queue depth N ≥ target_per_connection_throughput × stage_latency

Worked example for object-storage upload at 1 s p99 latency, targeting 100 RPS per connection: depth ≥ 100. For a target of 1000 RPS per connection, depth ≥ 1000. The choice of "per connection" depth vs "per broker" depth matters — the broker typically operates a per-broker queue shared across connections, which can be smaller than the sum of per-connection requirements because of statistical multiplexing.

Memory cost per slot: typically the size of one batched message set (potentially several MB at Cloud Topics' 4 MB cross-partition batch size). At depth 1000 × 4 MB = 4 GB — non-trivial but feasible on modern broker hardware. Real-world depth is usually substantially lower because of per-connection backpressure and finite producer arrival rates.

Order restoration is non-trivial — bake it into the design

The pattern's main subtlety is the order-restoration step. With N operations in flight against the slow stage, completion order is non-deterministic. A correct restoration step requires:

  • Sequence numbers assigned at queue ingress (in producer order).
  • Out-of-order completion handling: when item k+1 completes before item k, hold k+1's release until k completes.
  • Bounded memory for held-back completions: if item k stalls indefinitely, the held-back completions for k+1, k+2, … accumulate. Backpressure or timeout/retry must engage.

Cloud Topics achieves this by holding the producer's ack until the metadata has been replicated, not until the upload has completed. The metadata-replication layer remains serialised, so once items emerge from the upload queue in producer order, the rest of the pipeline behaves identically to the pre-Cloud-Topics path. This composes cleanly with patterns/pipelined-produce-with-position-guarantee and Kafka's idempotent-producer sequence-number scheme.

Backpressure is mandatory at the queue's input

A latency-hiding queue without backpressure is an unbounded in-memory buffer. The slow stage's processing rate caps the sustainable throughput; if the producer arrival rate exceeds it, the queue grows without bound and the broker eventually OOMs. The correct behaviour at queue saturation:

  • Block producer ingest (delays propagate naturally upstream), or
  • Apply explicit producer rate limits (token bucket, leaky bucket), or
  • Fail explicit with a backpressure error code that producers retry with backoff.

Of these, blocking is most natural in a Kafka-API context (the producer's buffer.memory will fill, and send() will block until space is available). Explicit rate limiting and explicit errors require client-side awareness and are usually a deliberate choice for SLA-isolation reasons (multi-tenant brokers).

Composes well with batching, not as a substitute

The concurrency-buffer-queue pattern composes with — but does not replace — batching:

Mechanism Amortises Latency cost
Batching window Per-operation fixed cost (PUT request, network round-trip) Up to the configured window (linger.ms, batch-size threshold)
Concurrency buffer queue (this pattern) Per-operation variable I/O latency None — items move through the queue at their natural in-flight rate

Cloud Topics uses both. A 0.25 s / 4 MB cross-partition batching window (patterns/object-store-batched-write-with-raft-metadata) amortises the per-PUT cost across many records. The concurrency buffer queue (this pattern) provides the in-flight concurrency to hide the per-PUT latency. Each addresses a different cost axis; neither alone is sufficient.

Failure modes

  1. Queue too shallow → throughput pinned below target. Not a correctness bug, but quietly leaves performance on the table. Diagnosed by comparing observed per-connection RPS to 1 / stage_latency; if they're equal, the queue isn't doing its job.

  2. Queue too deep → memory pressure under saturation. When the queue saturates (arrival > sustainable throughput), the depth at which backpressure engages is critical. Too deep and the broker OOMs before backpressure fires.

  3. Order-restoration buggy → idempotency violations. Concurrent uploads with sequence numbers can be replayed in any order to the metadata layer if restoration is incorrect. Treat the queue as a correctness multiplier on the restoration logic — bugs that were latent at low concurrency become surfaced at high concurrency.

  4. Steady-state queue depth mistaken for saturation. A queue sized for T × W should be at depth ~T × W in steady state. Operators must distinguish "expected steady-state depth" (good) from "saturation, backpressuring" (degraded). See concepts/queue-length-vs-wait-time — wait-time is the better signal here than queue length.

  5. Slow-stage tail-latency amplification. If the slow stage has a long-tail latency distribution, the queue's order-restoration step holds back faster completions waiting for the long-tail one. p99 of the pipeline is p99(slow_stage) not mean(slow_stage). Mitigate via per-request timeout + retry, or parallel-redundant submission (hedged requests).

Seen in

  • Redpanda Cloud Topics (2026-05-05) — canonical instance: extra upload queue inserted between ingest and metadata-replication, sized to hide ~100× latency increase from object-storage substrate change, with order restoration via Kafka idempotent-producer sequence numbers and ack held until metadata-replication completes.
Last updated · 542 distilled / 1,571 read