Skip to content

CONCEPT Cited by 1 source

Broker-internal cross-cluster replication

Definition

Broker-internal cross-cluster replication is the architectural property where the broker itself implements cross-cluster replication — using the Kafka wire protocol to link to another cluster and stream records directly into its log layer — rather than running replication as an external consumer+producer pipeline (Kafka Connect + connectors).

The distinction is about where replication code lives and what API surface it uses, not just whether replication happens. Both broker-internal and connector-based replication can be async, can be cross-region, can carry topic configs. But they differ in three load-bearing ways.

Canonical wiki source

Introduced in the Redpanda 25.3 launch post as the architectural distinction of Shadowing:

"Shadowing is built into the Redpanda broker itself and uses the standard Kafka API to link clusters. No MirrorMaker 2 or Redpanda Migrator connectors are used under the hood."

Contrast: connector-based replication

MirrorMaker2 and the prior Redpanda Migrator are connector-based: they run as Kafka Connect source + destination connectors that consume from the source cluster via the public consumer API and produce to the destination cluster via the public producer API. From the destination's perspective, MM2 is indistinguishable from any other producer.

This architecture has three load-bearing costs:

  1. Operational surface. A Kafka Connect cluster is itself a distributed system — tasks, connectors, internal topics, rebalancing, task restarts. It sits outside the Kafka broker and has to be operated, upgraded, and monitored separately.
  2. Offset re-assignment. Records produced via the public API get whatever offset the destination leader assigns them. Source offsets are not preserved; consumers need an external translation map at failover. See concepts/offset-preserving-replication.
  3. Ceiling on throughput. Public-API consume + public-API produce is a round-trip through Kafka's own client code; broker-internal replication can skip parts of that code path.

The structural trade-off

Broker-internal replication moves replication from the ecosystem layer (Kafka Connect) into the broker layer. The wins:

  • No extra cluster to operate. DR is a broker feature, not a Connect deployment.
  • Offset preservation is possible. The broker's log layer can write records at source-assigned offsets.
  • Tighter coupling with broker lifecycle primitives. Topic configs, ACLs, schema-registry metadata, consumer-group offsets can all be replicated as part of the same feature, under the same config surface, on the same upgrade cadence.

The losses:

  • Tighter vendor coupling. Broker-internal replication only works between two clusters of the same vendor's broker. MM2 works between any Kafka-API-compatible broker pair (Kafka, Redpanda, WarpStream, Confluent Kora, Apache Pulsar via Kafka shim) — it's the ecosystem-interoperability shape.
  • Wire-protocol ambiguity. Broker-internal replication may use vendor extensions over the Kafka API rather than the pure public API; upgrade coordination across source and destination becomes a broker concern.
  • Feature fork. Vendors with broker-internal replication (Redpanda Shadowing) and vendors with connector-based replication (Confluent) diverge in how DR is operated in the field — not a pure Kafka-API-ecosystem experience.

Position on the Kafka-ecosystem axis

Broker-internal cross-cluster replication is an example of the broader "move from connector to broker" trend in the Kafka API ecosystem — the same trend that moved:

  • Iceberg topics from external Kafka-Connect-based sinks into the broker.
  • Connector-mediated CDC (Debezium on Kafka Connect) into first-class Redpanda Connect inputs (see Redpanda Connect).
  • Connector-mediated schema handling into native schema-registry integration.

The trade-off pattern is consistent: operational simplification + feature coupling at the cost of ecosystem neutrality.

Seen in

Last updated · 470 distilled / 1,213 read