Skip to content

REDPANDA 2026-08-25

Read original ↗

Push-button migration from Confluent to Redpanda with Shadowing

Summary

Redpanda 26.2 extends its in-broker Shadowing replication feature — previously a Redpanda-to-Redpanda disaster-recovery mechanism — to migrate off Confluent Cloud, Confluent Platform, or any Apache-Kafka-compatible cluster. The headline addition is API-mode Schema Registry replication: instead of requiring a Redpanda source (which replicated schemas byte-for-byte by shadowing the internal _schemas topic), the shadow cluster now polls any registry speaking the standard Schema Registry REST API and imports all subjects, versions, and compatibility settings into Redpanda's built-in registry. One shadow link carries the whole migration on a single connection — topic data on the Kafka side, schemas on the registry side, plus consumer-group offsets and ACLs — enabling a self-paced, side-by-side migration where applications cut over one at a time on their own schedule rather than in a big-bang cutover weekend. The post frames Schema Registry (not topic data) as the component that historically stalls Confluent migrations, because a botched schema-ID reconciliation breaks every consumer at once.

Key takeaways

  1. Schema Registry is the migration bottleneck, not topics. "Topics get all the attention in a migration plan … but the Schema Registry is usually what stalls the project." It sits between every producer and consumer, and getting it wrong "can break all your applications at once rather than just those using a particular topic." (Source: sources/2026-08-25-redpanda-push-button-migration-from-confluent-to-redpanda-with-shadowing)

  2. API-mode Schema Registry replication is the new 26.2 primitive. Earlier releases replicated schemas Redpanda-to-Redpanda byte-for-byte by shadowing the internal _schemas topic. In 26.2 the shadow cluster polls any registry over the standard Schema Registry REST API and imports subjects, versions, and compatibility settings directly into Redpanda's built-in registry. See concepts/schema-registry.

  3. Shadowing is pull-based and broker-internal. "Each broker in the shadow cluster runs replication tasks that fetch data from the source via the standard Kafka and Schema Registry APIs." No MirrorMaker, no Redpanda Connect/Migrator deployment to stand up and babysit — the mechanism lives in the broker (contrast the connector-based Redpanda Migrator, which "was still another deployment to stand up, secure, and babysit for the duration of the migration.").

  4. Two sync cycles keep registries in step. A tail sync runs every 10s by default (tail_interval: 10s) picking up incremental changes; a full sync scans all selected subjects every 5 minutes (full_sync_interval: 5m) to catch anything a tail missed. A configurable max_source_requests_per_second (default 30) rate-limits requests against the source — "which matters more than you'd think when the source is a Confluent Cloud registry you're metered on."

  5. Schema IDs and subject/version names are preserved. "Producers and consumers that resolve schemas by ID keep working after cutover. This is the part hand-rolled migrations get wrong." The wire format bakes a schema ID into every record; if IDs don't line up, consumers deserialize garbage. See schema-id-preservation.

  6. Schema references import in dependency order. Subjects that point at other subjects resolve correctly — complex hierarchical Avro, Protobuf, and JSON Schema all replicate. See dependency-ordered-schema-import.

  7. Schemas are validated on import, with a policy knob. Redpanda validates each schema against its own registry before importing. unsupported_schema_feature_policy: FAIL (the default) skips a schema with Confluent-specific extensions (rule sets, metadata tags, compatibility groups) and reports the error; REMOVE strips the unsupported fields and imports the rest. "Either way, you find out during a sync you're watching, not during a cutover you can't stop."

  8. The destination is write-blocked while the link is active. Redpanda rejects client writes to the schema contexts a live shadow link owns, so the two registries can't drift apart mid-migration. Contexts outside the link's filter stay writable. See write-blocked-replica-to-prevent-drift.

  9. Self-paced, per-topic/per-application cutover. Shadow Links can be [failed over one topic at a time], so "the migration becomes a sequence of small, boring cutovers end users never notice." Consumer-group offsets are already replicated (consumers resume where they left off, not from the partition start), and ACLs are already in place. See self-paced-per-application-cutover.

  10. Contexts handle sprawling multi-tenant Confluent estates. Redpanda's Schema Registry supports contexts — independent subject namespaces within one registry, each with its own schema-ID counter, mode, and compatibility settings, compatible with Confluent's Contexts API. A source_filter selects specific contexts/subjects (qualified syntax :.staging:orders-value); identity destination mapping preserves source context names, while exact mapping remaps/renames contexts to consolidate several source registries into one Redpanda cluster without subject-name collisions. See concepts/schema-registry.

  11. Redpanda-as-safety-net is a legitimate end state. Risk-averse teams point a Redpanda shadow at Confluent/Kafka and leave it there — a warm standby "in a different failure domain, with a different support relationship, usually at much lower cost than a second Confluent cluster." Some run this for a quarter+ before cutting over anything. See warm-standby-cross-platform-replica.

Operational numbers & config

  • tail_interval: 10s — incremental schema sync cadence (default).
  • full_sync_interval: 5m — full source-scan cadence catching missed changes.
  • max_source_requests_per_second: 30 — default source-registry rate limit (matters against metered Confluent Cloud registries).
  • unsupported_schema_feature_policy: FAIL (default) | REMOVE — behavior on Confluent-specific schema extensions (rule sets, metadata tags, compatibility groups) with no Redpanda equivalent.
  • source_filter — selects contexts/subjects, qualified syntax :.staging:orders-value.
  • Destination context mapping: identity (preserve names) | exact (rename/remap to consolidate registries).
  • Setup surfaces: single config file, UI wizard, Kubernetes CRD, or Terraform module.
  • Availability: Redpanda Self-Managed 26.2 and Redpanda Cloud BYOC / Dedicated. Shadowing + API-mode schema replication are enterprise features (require an Enterprise license or Cloud/BYOC trial).
  • Hands-on lab: a Docker Compose lab runs a real Confluent Platform stack (Confluent Kafka in KRaft mode + Confluent Schema Registry) as source and replicates to a Redpanda shadow in ~20 minutes; deliberately exercises schema references (shipping-value → address-value), JSON Schema + Protobuf + Avro subjects, BACKWARD and FULL_TRANSITIVE per-subject compatibility modes, and verifies records decode identically against both registries after a single-config-line switch.

Caveats

  • Marketing framing: the post opens on the March 2026 IBM acquisition of Confluent as the motivation to reconsider vendors — a competitive-positioning angle, not an architectural one. The architecture content (pull-based broker tasks, API-mode registry replication, contexts, write-blocking) is the in-scope substance.
  • Shadowing/API-mode replication are enterprise-licensed features, not in the open-source tier.
  • Confluent-specific registry extensions (rule sets, metadata tags, compatibility groups) have no Redpanda equivalent — they are either skipped (FAIL) or stripped (REMOVE), so a migration relying on those features is not fully faithful.

Source

  • systems/redpanda-shadowing — the in-broker replication feature this article extends to Confluent/Kafka sources.
  • systems/redpanda-migrator — the connector-based migration path this supersedes for schema-carrying migrations.
  • concepts/schema-registry — the new 26.2 primitive.
  • concepts/schema-registry — independent subject namespaces for multi-tenant consolidation.
  • schema-id-preservation — why ID stability is load-bearing.
  • self-paced-per-application-cutover — the migration cadence.
  • warm-standby-cross-platform-replica — Redpanda-as-safety-net.
  • write-blocked-replica-to-prevent-drift — anti-drift mechanism.
  • dependency-ordered-schema-import — reference resolution order.
  • zero-downtime-migration — the migration class.
  • concepts/schema-evolution — the compatibility-obligation backdrop.
  • companies/redpanda — the company shipping the feature.
Last updated · 766 distilled / 2,225 read