Skip to content

CONCEPT Cited by 1 source

Parallel broker replication tasks

Definition

Parallel broker replication tasks is the broker-internal cross-cluster replication property where every broker in the shadow / destination cluster runs its own replication tasks that read directly from brokers in the source cluster, in a fully distributed shared-nothing pattern. Replication throughput scales linearly with broker count until the cross-cluster network saturates — adding brokers adds replication capacity without reconfiguration.

Canonical wiki source

Introduced by the 2026-04-21 Shadow Linking deep-dive as the mechanism underneath Redpanda Shadow Linking:

"A shadow link is defined within the shadow cluster and creates tasks internal to the broker that read data from the source cluster and write it locally. These tasks read data from the source using the standard Kafka API."

"Each broker in the shadow cluster runs replication tasks that read directly from the brokers in the source cluster, enabling massively parallel data transfer. This fully distributed approach provides excellent throughput and allows you to scale replication capacity simply by adding more brokers, up to the limit of your network."

"Shadow linking also scales naturally with the cluster, both vertically and horizontally. If you use bigger nodes with more cores, Redpanda's internal shared-nothing architecture can use that to its fullest. If you scale out the cluster and add more nodes, we will use them to increase the shadowing parallelism, all without you needing to tune anything out of the box."

Three load-bearing properties

  1. Fully distributed. Replication work is not centralised in a coordinator, a master task, or an external service — each broker does its share. No single node is the bottleneck.

  2. Scales linearly with broker count. Doubling the shadow cluster's broker count doubles the total replication capacity (up to the network limit). This is the horizontal scaling axis of replication throughput.

  3. Inherits Redpanda's thread-per-core runtime. Replication tasks run on the same Seastar cooperative-scheduling substrate as the rest of the broker. Bigger nodes with more cores absorb more parallelism without reconfiguration — the vertical scaling axis.

Contrast with connector-based replication

MirrorMaker2 runs replication on a separate Kafka Connect cluster whose capacity is planned and scaled independently of the source and destination clusters. The Connect cluster has its own replication-task scheduler, its own workers, its own failure modes, and its own JVM tuning surface.

Parallel-broker-replication-tasks folds this responsibility into the existing broker fleet:

  • No separate cluster to plan capacity for. Replication capacity is a derivative of broker count — if the shadow cluster is sized for its read/write load, replication sizing is automatic.
  • No separate failure mode. Brokers that are up replicate; brokers that are down are replaced by their Raft peers and replication resumes on the replacement. There's no Connect worker to be in a bad state.
  • No JVM tuning. Redpanda is C++; the replication task runs in the same C++ runtime as the rest of the broker.

The trade-off is coupling: cluster-wide broker outages take replication with them, whereas a separate Connect cluster could theoretically keep running if the destination cluster is down (though there'd be nothing to produce to). The 2026-04-21 deep- dive treats this coupling as the favoured trade-off because the common case is "when the shadow cluster is up, replication should be running."

Performance disclosure

The 2026-04-21 post reports a scale-test at 2.5 GiB/s source throughput / 2.5 million messages/second, achieved with "a total lag (across all topics) that was consistently lower than 10,000 messages" — i.e. cluster-wide lag under 10k messages across the entire workload, yielding an effective RPO of ~4 ms on average. This is the first disclosed per-feature Shadow Linking RPO number and is bounded by the parallel-broker-task architecture's throughput ceiling, not by any single-point-of-contention in the replication path.

Where the pattern generalises

Parallel-broker-replication-tasks is not Redpanda-specific in principle — any broker whose per-partition work already lives on shard-owned brokers (shared-nothing) can extend the same mechanism to cross-cluster replication. In practice this requires:

  • A shared-nothing internal runtime (Redpanda has Seastar; Kafka's JVM model is thread-pool-based and doesn't naturally extend this way).
  • A per-broker addressability from outside the cluster (replication tasks on the shadow need to address source brokers directly, not via a proxy).
  • An in-broker mechanism to write records with externally- determined offsets (the broker- internal property that unlocks offset preservation).

Apache Kafka itself has neither (1) the shared-nothing runtime nor (3) a broker-internal write-with-externally-determined-offsets path, which is why MM2 is the canonical Kafka-side answer and Shadow Linking is Redpanda-side only.

Seen in

Last updated · 550 distilled / 1,221 read