Skip to content

REDPANDA 2025-11-06

Read original ↗

Redpanda — Redpanda 25.3 delivers near-instant disaster recovery and more

Summary

2025-11-06 Redpanda product-launch post previewing the four headline features of the Redpanda 25.3 release: Shadowing (near-instant multi-region disaster recovery), Cloud Topics (per-topic cost-tiering to object storage within a single cluster), Iceberg Topics + BigLake metastore integration, and Microsoft SQL Server CDC input for Redpanda Connect. Announcement voice throughout ("simple to run, efficient at scale, interoperable with your current stack, and safe for the enterprise"), but three of the four features carry load-bearing architectural content — a new broker-internal cross-cluster replication mechanism, a new per-topic storage-tiering shape within a single cluster, and a new CDC engine-family addition — that extends the wiki's Redpanda coverage on the DR, cost-tiering, lakehouse-catalog, and CDC-ecosystem axes. Borderline-include on vocabulary- canonicalisation grounds: four new vocabulary primitives missing from prior wiki coverage (offset-preserving replication, broker-internal cross-cluster replication, cross-AZ replication bandwidth cost, latency-critical vs latency- tolerant workload classification) plus two new features-as- systems (Shadowing, Cloud Topics) plus one new CDC engine (SQL Server).

Key takeaways

  1. Shadowing is Redpanda's new DR mechanism and is architecturally distinct from both MirrorMaker2 and the prior Redpanda Migrator. Verbatim: "Shadowing is built into the Redpanda broker itself and uses the standard Kafka API to link clusters. No MirrorMaker 2 or Redpanda Migrator connectors are used under the hood." The shape is a byte-for-byte, offset-preserving hot-standby clone in a different region — "a fully functional, hot-standby clone of your entire Redpanda cluster — topics, configs, consumer group offsets, ACLs, schemas — the works!" New wiki concept: concepts/broker-internal-cross-cluster-replication. (Source: sources/2025-11-06-redpanda-253-delivers-near-instant-disaster-recovery-and-more)

  2. Shadowing's RPO/RTO is seconds, not the replication-lag + connector-restart shape of MirrorMaker2. Verbatim: "We're talking Recovery Point Objectives (RPOs) measured in a few seconds and similar Recovery Time Objectives (RTOs), limited only by timeout settings for producers and consumers." The load-bearing client-side benefit — offset preservation — means consumers can fail over to the shadow cluster at the same offsets they held on the source, without offset-translation bookkeeping. Contrast MirrorMaker2 which maintains per-consumer-group offset translations between source and destination because the two clusters have independent offset numbering. (Source: sources/2025-11-06-redpanda-253-delivers-near-instant-disaster-recovery-and-more)

  3. Cloud Topics does per-topic storage tiering within a single cluster, with "each batch of messages … passed straight through and written to cost-effective object storage (S3/ADLS/GCS) while topic metadata is managed in-broker — replicated via Raft for high availability". This is architecturally distinct from both whole-cluster tiered storage (where primary data is on NVMe and cold data is on object store per-broker) and from WarpStream's everything-on-S3 shape (which sacrifices hot-path latency). The key operational claim: "virtually eliminates the cross-AZ network traffic associated with data replication" — replication cost shifts from cross-AZ bandwidth to object-storage PUT/GET. (Source: sources/2025-11-06-redpanda-253-delivers-near-instant-disaster-recovery-and-more)

  4. Cloud Topics is explicitly positioned against Confluent's two-cluster shape. Verbatim: "Contrast this with Confluent, where you may need a mix of Kora-powered Confluent Cloud clusters (standard/dedicated or Freight) and the separate Confluent WarpStream engine (BYOC) to satisfy different requirements." Redpanda's differentiator framing: one cluster, multiple storage tiers, single control plane — customers pick latency-critical topics (NVMe-backed, write-caching enabled, low-latency) vs latency-tolerant topics (object-storage-backed, cost-optimised) per-topic, not per-cluster. New wiki concept: concepts/latency-critical-vs-latency-tolerant-workload. New wiki pattern: patterns/per-topic-storage-tier-within-one-cluster. (Source: sources/2025-11-06-redpanda-253-delivers-near-instant-disaster-recovery-and-more)

  5. BigLake metastore integration adds GCP to the existing REST catalog support for Iceberg Topics. Prior wiki coverage (from the 2025-04-07 GA post) named REST catalog sync via OIDC+TLS against Databricks Unity Catalog, Snowflake Open Catalog (based on Apache Polaris), and AWS Glue; 25.3 adds Google BigLake metastore (with Dataplex as the governance layer), so Iceberg tables written by Redpanda are discoverable in BigQuery without external ETL. (Source: sources/2025-11-06-redpanda-253-delivers-near-instant-disaster-recovery-and-more)

  6. Microsoft SQL Server CDC input extends the Redpanda Connect CDC engine family from four to five. The prior family (canonicalised in the 2025-03-18 connectors post) was postgres_cdc + mysql_cdc + mongodb_cdc + gcp_spanner_cdc; 25.3 adds microsoft_sql_server_cdc via MSSQL's native change tables mechanism — "non-invasively captures every single insert, update, and delete from your SQL Server tables in real time". Available in Redpanda Connect 4.67.5 (enterprise connector) and Redpanda Cloud. Fits the prior wiki framing of CDC driver ecosystem — one connector per source-database engine, riding on each engine's native change log. (Source: sources/2025-11-06-redpanda-253-delivers-near-instant-disaster-recovery-and-more)

  7. MSSQL CDC benchmark: ~40 MB/s average ingest, 3-minute initial snapshot on a 5-million-row table, versus an unnamed "alternative hosted Kafka + CDC service" at ~14.5 MB/s and ~8-minute snapshot. Test setup disclosed: "One table (5 million rows); SQL Server co-located with Azure instance running Redpanda Connect (instance specs: 4 vCPUs bound to 1 logical core, 16 GB memory); default settings used; identical schema/workload across runs." Caveats acknowledged inline: "Your results will vary by table size, change rate, and environment." Vendor-benchmark voice — alternative vendor unnamed, workload narrow (single table, initial snapshot dominated). (Source: sources/2025-11-06-redpanda-253-delivers-near-instant-disaster-recovery-and-more)

  8. Shadowing + Cloud Topics together reframe the RPO-vs-cost-vs-latency trade-off from a cluster-level choice to a topic-level choice. Prior wiki framing (from the 2025-02-11 stretch-clusters post) was: stretch cluster (RPO=0, cross-region RTT per write) vs MM2 async (non-zero RPO, local write latency) — a per-cluster decision. Post-25.3: Shadowing offers seconds-RPO without the connector overhead of MM2; Cloud Topics offers per-topic latency-tolerance. A customer can now run one cluster that hosts mission-critical low-latency topics on NVMe with a Shadowing clone in a second region and latency-tolerant compliance/observability topics on object storage — "one system for all streams". (Source: sources/2025-11-06-redpanda-253-delivers-near-instant-disaster-recovery-and-more)

Systems

Concepts

  • concepts/offset-preserving-replication — new. The property that a replicated cluster holds the same offsets as the source, so consumers can fail over without offset translation. The structural property that separates Shadowing from MirrorMaker2-style connector-mediated replication.
  • concepts/broker-internal-cross-cluster-replication — new. The architecturally distinctive property of Shadowing: replication implemented inside the broker rather than via an external Kafka Connect cluster.
  • concepts/cross-az-replication-bandwidth-cost — new. The per-byte cloud-provider charge for replicating data across availability zones within a region. The cost Cloud Topics eliminates by routing data to object storage instead.
  • concepts/latency-critical-vs-latency-tolerant-workload — new. The workload-class distinction Redpanda uses to motivate per-topic tiering: latency-critical (payments, trading, cybersecurity) vs latency-tolerant (observability, model training, compliance reporting).
  • concepts/rpo-rto — extended. Shadowing is a new seconds-RPO / seconds-RTO shape in a different architectural position from stretch cluster and MM2.
  • concepts/mirrormaker2-async-replication — extended. 25.3-era comparison: Shadowing is the new preferred DR shape; MM2 is explicitly called out as not the mechanism under Shadowing.
  • concepts/change-data-capture — extended. MSSQL is the fifth source-database engine with a first-class Redpanda Connect connector.
  • concepts/iceberg-catalog-rest-sync — extended. BigLake metastore added to the canonical REST-catalog list (Unity Catalog / Snowflake Open Catalog / AWS Glue / BigLake).

Patterns

Operational numbers

  • Shadowing RPO/RTO: "measured in a few seconds … limited only by timeout settings for producers and consumers".
  • Cloud Topics storage substrate: S3 / ADLS / GCS; metadata via Raft in-broker.
  • MSSQL CDC benchmark: ~40 MB/s average ingest, 3:15 initial snapshot on a 5M-row table.
  • MSSQL CDC alternative (unnamed): ~14.5 MB/s, 8:04 initial snapshot.
  • MSSQL CDC benchmark hardware: 4 vCPUs (1 logical core), 16 GB memory Azure instance; SQL Server co-located.
  • Release: Redpanda 25.3 (coming soon at publication).
  • Cloud Topics scope: beta in 25.3; "Contact us to enable it".
  • MSSQL CDC availability: Redpanda Connect 4.67.5; Redpanda Cloud. Enterprise-gated.

Caveats

  • Launch / announcement voice throughout. Tagline is "Redpanda 25.3 delivers near-instant disaster recovery" — product-post framing, not post-mortem or production case study.
  • Shadowing mechanism under-specified. Asynchronous replication shape is named but the wire protocol, replication frame format, conflict resolution at failover, and per-partition ordering guarantees are not disclosed. No throughput / latency numbers for the replication stream itself.
  • Shadowing DR-drill mechanics elided. "Run a DR drill" is named in the summary recap but no procedure, no failover primitives, no reverse-replication story for failback.
  • Cloud Topics durability vs Cloud Topics availability unbundled only loosely. "without sacrificing durability" is asserted. Object-storage durability is the load-bearing premise but the availability implications of per-topic cross-AZ-elimination aren't walked — a single-AZ object-store outage in the cluster's region would still impact the Cloud-Topic write path.
  • Cloud Topics performance profile undisclosed. "latency- tolerant" workloads are named (observability, model training, compliance) but no p50/p99 write/read latencies, no throughput ceilings.
  • Cross-AZ-cost claim unquantified. "Dramatically lower TCO" — no dollar figures, no before/after on a reference workload.
  • MSSQL CDC benchmark alternative unnamed. "an alternative hosted Kafka + CDC service" — likely Debezium under Confluent's Managed Connectors or Redpanda's closest direct competitor, but not confirmed. Vendor-benchmark standard caveat: opponent unnamed, workload narrow.
  • MSSQL CDC connector scope not enumerated. Change-table consumption is named; GTID / AG / mirroring / cross- instance-shipping topologies not stated (cf. the 2025-03-18 post where MySQL CDC was explicitly limited to no-GTID / no-Group-Replication / no-multi-source).
  • BigLake integration mechanism named but undisclosed. "Can automatically register streaming tables to the Google BigLake metastore" — the registration protocol, credential handling, and catalog-sync cadence are not walked.
  • No comparison to file-based catalog shape from the 2025-05-13 BYOC-beta post: BigLake is REST-catalog; the prior BigQuery worked example used the file-based metadata-pointer shape. Both shapes now exist in parallel but the trade-off isn't discussed.
  • Confluent foil unbalanced. Cloud Topics is compared against a specific Confluent product split (Kora + WarpStream); the comparison doesn't disclose whether Confluent's own Tiered Storage feature in Kora already addresses the same cost axis.
  • 25.3 release date not given — "coming soon". The beta scope of Cloud Topics is explicit; Shadowing's beta/GA status is ambiguous.
  • Unsigned. Redpanda default attribution; no author byline.

Source

Last updated · 470 distilled / 1,213 read